![Extensions
One or more instances: (r)+
Zero of one instances: r?
Character classes: [abc]
Example:
letter_ -> [A-Za-z_]
digit -> [0-9]
id -> letter_(letter|digit)*](https://image.slidesharecdn.com/02-240830042900-aecdf2d9/85/02-Chapter-3-Lexical-Analysis-NLP-ppt-15-320.jpg)
![Recognition of tokens (cont.)
The next step is to formalize the patterns:
digit -> [0-9]
Digits -> digit+
number -> digit(.digits)? (E[+-]? Digit)?
letter -> [A-Za-z_]
id -> letter (letter|digit)*
If -> if
Then -> then
Else -> else
Relop -> < | > | <= | >= | = | <>
We also need to handle whitespaces:
ws -> (blank | tab | newline)+](https://image.slidesharecdn.com/02-240830042900-aecdf2d9/85/02-Chapter-3-Lexical-Analysis-NLP-ppt-17-320.jpg)
![Example
%{
/* definitions of manifest constants
LT, LE, EQ, NE, GT, GE,
IF, THEN, ELSE, ID, NUMBER, RELOP */
%}
/* regular definitions
delim [ tn]
ws {delim}+
letter [A-Za-z]
digit [0-9]
id {letter}({letter}|{digit})*
number {digit}+(.{digit}+)?(E[+-]?{digit}+)?
%%
{ws} {/* no action and no return */}
if {return(IF);}
then {return(THEN);}
else {return(ELSE);}
{id} {yylval = (int) installID(); return(ID); }
{number} {yylval = (int) installNum(); return(NUMBER);}
…
Int installID() {/* funtion to install
the lexeme, whose first
character is pointed to by
yytext, and whose length is
yyleng, into the symbol table
and return a pointer thereto */
}
Int installNum() { /* similar to
installID, but puts numerical
constants into a separate table
*/
}](https://image.slidesharecdn.com/02-240830042900-aecdf2d9/85/02-Chapter-3-Lexical-Analysis-NLP-ppt-25-320.jpg)
![48
Implementation
A DFA can be implemented by a 2D table T
One dimension is “states”
Other dimension is “input symbols”
For every transition Si a
Sk define T[i,a] = k
DFA “execution”
If in state Si and input a, read T[i,a] = k and skip to
state Sk
Very efficient](https://image.slidesharecdn.com/02-240830042900-aecdf2d9/85/02-Chapter-3-Lexical-Analysis-NLP-ppt-48-320.jpg)
02. Chapter 3 - Lexical Analysis NLP.ppt
- 2. Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata Design of lexical analyzer generator
- 3. The role of lexical analyzer Lexical Analyzer Parser Source program token getNextToken Symbol table To semantic analysis
- 4. Why to separate Lexical analysis and parsing 1. Simplicity of design 2. Improving compiler efficiency 3. Enhancing compiler portability
- 5. Tokens, Patterns and Lexemes A token is a pair a token name and an optional token value A pattern is a description of the form that the lexemes of a token may take A lexeme is a sequence of characters in the source program that matches the pattern for a token
- 6. Example Token Informal description Sample lexemes if else comparison id number literal Characters i, f Characters e, l, s, e < or > or <= or >= or == or != Letter followed by letter and digits Any numeric constant Anything but “ sorrounded by “ if else <=, != pi, score, D2 3.14159, 0, 6.02e23 “core dumped” printf(“total = %dn”, score);
- 7. Attributes for tokens E = M * C ** 2 <id, pointer to symbol table entry for E> <assign-op> <id, pointer to symbol table entry for M> <mult-op> <id, pointer to symbol table entry for C> <exp-op> <number, integer value 2>
- 8. Lexical errors Some errors are out of power of lexical analyzer to recognize: fi (a == f(x)) … However it may be able to recognize errors like: d = 2r Such errors are recognized when no pattern for tokens matches a character sequence
- 9. Error recovery Panic mode: successive characters are ignored until we reach to a well formed token Delete one character from the remaining input Insert a missing character into the remaining input Replace a character by another character Transpose two adjacent characters
- 10. Input buffering Sometimes lexical analyzer needs to look ahead some symbols to decide about the token to return In C language: we need to look after -, = or < to decide what token to return In Fortran: DO 5 I = 1.25 We need to introduce a two buffer scheme to handle large look-aheads safely E = M * C * * 2 eof
- 11. Sentinels Switch (*forward++) { case eof: if (forward is at end of first buffer) { reload second buffer; forward = beginning of second buffer; } else if {forward is at end of second buffer) { reload first buffer; forward = beginning of first buffer; } else /* eof within a buffer marks the end of input */ terminate lexical analysis; break; cases for the other characters; } E = M eof * C * * 2 eof eof
- 12. Specification of tokens In theory of compilation regular expressions are used to formalize the specification of tokens Regular expressions are means for specifying regular languages Example: Letter_(letter_ | digit)* Each regular expression is a pattern specifying the form of strings
- 13. Regular expressions Ɛ is a regular expression, L(Ɛ) = {Ɛ} If a is a symbol in ∑then a is a regular expression, L(a) = {a} (r) | (s) is a regular expression denoting the language L(r) ∪ L(s) (r)(s) is a regular expression denoting the language L(r)L(s) (r)* is a regular expression denoting (L9r))* (r) is a regular expression denting L(r)
- 14. Regular definitions d1 -> r1 d2 -> r2 … dn -> rn Example: letter_ -> A | B | … | Z | a | b | … | Z | _ digit -> 0 | 1 | … | 9 id -> letter_ (letter_ | digit)*
- 15. Extensions One or more instances: (r)+ Zero of one instances: r? Character classes: [abc] Example: letter_ -> [A-Za-z_] digit -> [0-9] id -> letter_(letter|digit)*
- 16. Recognition of tokens Starting point is the language grammar to understand the tokens: stmt -> if expr then stmt | if expr then stmt else stmt | Ɛ expr -> term relop term | term term -> id | number
- 17. Recognition of tokens (cont.) The next step is to formalize the patterns: digit -> [0-9] Digits -> digit+ number -> digit(.digits)? (E[+-]? Digit)? letter -> [A-Za-z_] id -> letter (letter|digit)* If -> if Then -> then Else -> else Relop -> < | > | <= | >= | = | <> We also need to handle whitespaces: ws -> (blank | tab | newline)+
- 18. Transition diagrams Transition diagram for relop
- 19. Transition diagrams (cont.) Transition diagram for reserved words and identifiers
- 20. Transition diagrams (cont.) Transition diagram for unsigned numbers
- 21. Transition diagrams (cont.) Transition diagram for whitespace
- 22. Architecture of a transition- diagram-based lexical analyzer TOKEN getRelop() { TOKEN retToken = new (RELOP) while (1) { /* repeat character processing until a return or failure occurs */ switch(state) { case 0: c= nextchar(); if (c == ‘<‘) state = 1; else if (c == ‘=‘) state = 5; else if (c == ‘>’) state = 6; else fail(); /* lexeme is not a relop */ break; case 1: … … case 8: retract(); retToken.attribute = GT; return(retToken); }
- 23. Lexical Analyzer Generator - Lex Lexical Compiler Lex Source program lex.l lex.yy.c C compiler lex.yy.c a.out a.out Input stream Sequence of tokens
- 24. Structure of Lex programs declarations %% translation rules %% auxiliary functions Pattern {Action}
- 25. Example %{ /* definitions of manifest constants LT, LE, EQ, NE, GT, GE, IF, THEN, ELSE, ID, NUMBER, RELOP */ %} /* regular definitions delim [ tn] ws {delim}+ letter [A-Za-z] digit [0-9] id {letter}({letter}|{digit})* number {digit}+(.{digit}+)?(E[+-]?{digit}+)? %% {ws} {/* no action and no return */} if {return(IF);} then {return(THEN);} else {return(ELSE);} {id} {yylval = (int) installID(); return(ID); } {number} {yylval = (int) installNum(); return(NUMBER);} … Int installID() {/* funtion to install the lexeme, whose first character is pointed to by yytext, and whose length is yyleng, into the symbol table and return a pointer thereto */ } Int installNum() { /* similar to installID, but puts numerical constants into a separate table */ }
- 26. 26 Finite Automata Regular expressions = specification Finite automata = implementation A finite automaton consists of An input alphabet A set of states S A start state n A set of accepting states F S A set of transitions state input state
- 27. 27 Finite Automata Transition s1 a s2 Is read In state s1 on input “a” go to state s2 If end of input If in accepting state => accept, othewise => reject If no transition possible => reject
- 28. 28 Finite Automata State Graphs A state • The start state • An accepting state • A transition a
- 29. 29 A Simple Example A finite automaton that accepts only “1” A finite automaton accepts a string if we can follow transitions labeled with the characters in the string from the start to some accepting state 1
- 30. 30 Another Simple Example A finite automaton accepting any number of 1’s followed by a single 0 Alphabet: {0,1} Check that “1110” is accepted but “110…” is not 0 1
- 31. 31 And Another Example Alphabet {0,1} What language does this recognize? 0 1 0 1 0 1
- 32. 32 And Another Example Alphabet still { 0, 1 } The operation of the automaton is not completely defined by the input On input “11” the automaton could be in either state 1 1
- 33. 33 Epsilon Moves Another kind of transition: -moves • Machine can move from state A to state B without reading input A B
- 34. 34 Deterministic and Nondeterministic Automata Deterministic Finite Automata (DFA) One transition per input per state No -moves Nondeterministic Finite Automata (NFA) Can have multiple transitions for one input in a given state Can have -moves Finite automata have finite memory Need only to encode the current state
- 35. 35 Execution of Finite Automata A DFA can take only one path through the state graph Completely determined by input NFAs can choose Whether to make -moves Which of multiple transitions for a single input to take
- 36. 36 Acceptance of NFAs An NFA can get into multiple states • Input: 0 1 1 0 1 0 1 • Rule: NFA accepts if it can get in a final state
- 37. 37 NFA vs. DFA (1) NFAs and DFAs recognize the same set of languages (regular languages) DFAs are easier to implement There are no choices to consider
- 38. 38 NFA vs. DFA (2) For a given language the NFA can be simpler than the DFA 0 1 0 0 0 1 0 1 0 1 NFA DFA • DFA can be exponentially larger than NFA
- 39. 39 Regular Expressions to Finite Automata High-level sketch Regular expressions NFA DFA Lexical Specification Table-driven Implementation of DFA
- 40. 40 Regular Expressions to NFA (1) For each kind of rexp, define an NFA Notation: NFA for rexp A A • For • For input a a
- 41. 41 Regular Expressions to NFA (2) For AB A B • For A | B A B
- 42. 42 Regular Expressions to NFA (3) For A* A
- 43. 43 Example of RegExp -> NFA conversion Consider the regular expression (1 | 0)*1 The NFA is 1 C E 0 D F B G A H 1 I J
- 45. 45 NFA to DFA. The Trick Simulate the NFA Each state of resulting DFA = a non-empty subset of states of the NFA Start state = the set of NFA states reachable through -moves from NFA start state Add a transition S a S’ to DFA iff S’ is the set of NFA states reachable from the states in S after seeing the input a considering -moves as well
- 46. 46 NFA -> DFA Example 1 0 1 A B C D E F G H I J ABCDHI FGABCDHI EJGABCDHI 0 1 0 1 0 1
- 47. 47 NFA to DFA. Remark An NFA may be in many states at any time How many different states ? If there are N states, the NFA must be in some subset of those N states How many non-empty subsets are there? 2N - 1 = finitely many, but exponentially many
- 48. 48 Implementation A DFA can be implemented by a 2D table T One dimension is “states” Other dimension is “input symbols” For every transition Si a Sk define T[i,a] = k DFA “execution” If in state Si and input a, read T[i,a] = k and skip to state Sk Very efficient
- 49. 49 Table Implementation of a DFA S T U 0 1 0 1 0 1 0 1 S T U T T U U T U
- 50. 50 Implementation (Cont.) NFA -> DFA conversion is at the heart of tools such as flex or jflex But, DFAs can be huge In practice, flex-like tools trade off speed for space in the choice of NFA and DFA representations
- 51. Readings Chapter 3 of the book