Recognition-of-tokens
28 likes39,360 views
The document discusses the process of recognizing tokens in a programming language, detailing how to specify tokens and write a program to recognize them. It introduces transition diagrams, which describe states and transitions in a deterministic manner, showing how each token, whitespace, and other inputs are managed. Additionally, it explains how to handle and return token values through a structured approach.
Recognition-of-tokens
- 1. Recognition of Tokens • We now know how to specify the tokens for our language. But how do we write a program to recognize them? if -> if then -> then else -> else relop -> < | <= | = | <> | > | >= id -> letter ( letter | digit )* num -> digit ( . digit )? ( E (+|-)? digit )?
- 2. Token recognition • We also want to strip whitespace, so we need definitions delim -> blank | tab | newline ws -> delim+
- 3. Attribute values Regular Expression Token Attribute value ws - - if if - then then - else else - id id ptr to sym table entry num num ptr to sym table entry < relop LT <= relop LE = relop EQ <> relop NE > relop GT >= relop GE
- 4. Transition diagrams • Transition diagrams are also called finite automata. • We have a collection of STATES drawn as nodes in a graph. • TRANSITIONS between states are represented by directed edges in the graph. • Each transition leaving a state s is labeled with a set of input characters that can occur after state s.
- 5. Transition diagrams (Conti…) • For now, the transitions must be DETERMINISTIC. • Each transition diagram has a single START state and a set of TERMINAL STATES. • The label OTHER on an edge indicates all possible inputs not handled by the other transitions. • Usually, when we recognize OTHER, we need to put it back in the source stream since it is part of the next token. This action is denoted with a * next to the corresponding state.
- 7. 0 6 8 7 Start > other = For eg: >= *
- 8. 0 6 8 7Star t > other = For eg: >= *
- 9. 0 6 8 7Star t > other = For eg: >= *
- 10. 0 6 8 7Star t > other = For eg: >= *
- 11. 0 6 8 7Star t > other = For eg: >= *
- 12. 0 6 8 7Star t > other = For eg: >= *
- 13. 0 6 8 7Star t > other = For eg: >= *
- 14. 0 6 8 7 Start > other = For eg: abc > pqr *
- 15. 0 6 8 7Star t > other = For eg: abc > pqr *
- 16. 0 6 8 7Star t > other = For eg: abc > pqr *
- 17. 0 6 8 7Star t > other = For eg: abc > pqr *
- 18. 0 6 8 7Star t > other = For eg: abc > pqr *
- 19. 0 6 8 7Star t > othe r = For eg: abc > pqr *
- 20. 0 6 8 7Star t > othe r = For eg: abc > pqr *
- 21. 0 6 8 7Star t > othe r = For eg: abc > pqr *
- 22. 0 6 8 7Star t > othe r = For eg: abc > pqr *
- 23. 0 21 6 3 4 5 7 8 start < = > other return( relop, LE) return( relop, NE) return( relop, LT) return( relop, EQ) return( relop, GE) return( relop, GT) other = * *= >
- 24. 0 21 6 3 4 5 7 8 start < = > other return( relop, LE) return( relop, NE) return( relop, LT) return( relop, EQ) return( relop, GE) return( relop, GT) other = * *= >
- 25. 9 10 11 letter letter or digit other *return(gettoken(), install_id()) start 25 26 27 digit digit other * start