Lexical Scanner Implementation (4)
- 2018-10-24
- Liu, An-Chi 劉安齊
Gitbook: https://tigercosmos.github.io/lets-build-dbms/
I write a scanner to get tokens. The code is at lexer.rs
.
The algorithm is simple. The scanner will read char by char. There are two cursors, cursor_l
and cursor_r
. When the income char is one of separators or delimiters, the scanner will check the last word selected by two cursors. If the word is a keyword, add the keyword as the token. Otherwise, add the identifier with its name.
There are some small parts need to fix, but the function is almost done.
If we have a message:
1 | select customername, contactname, address from customers where address is null; |
then the scanner will get:
1 | [ |
is null
should be recognized as a token, so I will fix later.
as you can see, we get the tokens and we can use these token to do the next step.
!FILENAME sql/lexer.rs
1 | use sql::symbol; |