![8
→ t_RANGLES = r’>’
→ def t_RANGLES(token):
r’>’
return token
→ def t_NUMBER(token):
r’[0-9]+’
token.value = int(token.value)
return token
2. Define Type of Token](https://image.slidesharecdn.com/opencourseprogramminglanguages20150128-160712052847/85/codin9cafe-2015-01-28-Open-course-programming-languages-Ch-Jang-8-320.jpg)
![13
→ def t_STRING(token)
r’”[^”]*”’
return token
→ def t_WORD(token)
r’[^ <>n]+’
return token
Rule Order
?
“Hello”
Ordering token definitions is of prime importance.
First One Wins!](https://image.slidesharecdn.com/opencourseprogramminglanguages20150128-160712052847/85/codin9cafe-2015-01-28-Open-course-programming-languages-Ch-Jang-13-320.jpg)
codin9cafe[2015.01.28]Open course(programming languages) - 장철호(Ch Jang)
- 1. Programming Languages Building a Web Brower Instructor : Westley Weimer 1
- 2. 2
- 3. 3 Lexical Analysis the process of converting a sequence of characters into a sequence of tokens Lexical Analysis String (I like you, …) List of Tokens
- 4. 4 Token the smallest unit of a programming language that has a meaning ＂I <b> like＂ word start of tag word end of tag word I < b > like
- 5. 5 ＂I <b>like</b> you＂ I < b > Like </ b > you WORD LANGLE WORD RANGLE WORD LANGLESLASH WORD RANGLE WORD word start of tag word end of tag word start of closing tag word end of closing tag word
- 6. 6 Python Lex-Yacc A computer program that generates lexical analyzers Lexical Analysis String Token LexInput Definition section %% Rules section %% C code section
- 7. 7 tokens = ( ‘LANGLE’, # < ‘LANGLESLASH’, # </ ‘RANGLE’, # > ‘RANGLESLASH’, # /> ‘EQUAL’, # = ‘STRING’, # “love” ‘WORD’, # like ) 1. Define Name of Token
- 8. 8 → t_RANGLES = r’>’ → def t_RANGLES(token): r’>’ return token → def t_NUMBER(token): r’[0-9]+’ token.value = int(token.value) return token 2. Define Type of Token
- 9. 9 → htmllexer = lex.lex() htmllexer.input(htmlcode) while True: tok = htmllexer.token() if not tok: break print tok 3. Building and Using the Lexer
- 10. 10 Python Code
- 11. 11 Output Input(Htmlcode) = This is <b>my</b> webpage! → LexToken(WORD, ‘This’, 1, 0) LexToken(WORD, ‘is’, 1, 5) LexToken(LANGLE, ‘<’, 1, 8) LexToken(WORD, ‘b’, 1, 9) LexToken(RANGLE, ‘>’, 1, 10) LexToken(WORD, ‘my’, 1, 11) LexToken(LANGLESLASH, ‘</’, 1, 13) LexToken(WORD, ‘b’, 1, 15) LexToken(RANGLE, ‘>’, 1, 16) LexToken(WORD, ‘webpage!’, 1, 18)
- 12. 12 - Rule Order - White Space - Tracking Line Number - Comment Notice
- 13. 13 → def t_STRING(token) r’”[^”]*”’ return token → def t_WORD(token) r’[^ <>n]+’ return token Rule Order ? “Hello” Ordering token definitions is of prime importance. First One Wins!
- 14. 14 → def t_WHITESPACES(token) r’ ’ pass → t_ignore = ‘ ‘ # ‘ tvr’ White Space Tokens do not refer to white space
- 15. 15 → def t_newline(token) r’n’ token.lexer.lineno += 1 pass Tracking Line Number Lex knows nothing about line numbers
- 16. 16 → states = ( (‘htmlcomment’, ‘exclusive’), ) → def t_htmlcomment(t) r’<!--’ t.lexer.begin(‘htmlcomment’) Comment Not find words or strings or numbers or tags → def t_htmlcomment_end(t) r’-->’ t.lexer.lineno += t.value.count(‘n’) t.lexer.begin(‘INITIAL’) → def t_htmlcomment_error(t) t.lexer.skip(1)
- 17. 17