Lexical Analysis - Compiler Design
Download as PPTX, PDF1 like3,288 views
Lexical analysis is the first phase of compilation. It reads source code characters and divides them into tokens by recognizing patterns using finite automata. It separates tokens, inserts them into a symbol table, and eliminates unnecessary characters. Tokens are passed to the parser along with line numbers for error handling. An input buffer is used to improve efficiency by reading source code in blocks into memory rather than character-by-character from secondary storage. Lexical analysis groups character sequences into lexemes, which are then classified as tokens based on patterns.
Lexical Analysis - Compiler Design
- 1. Akhil Kaushik Asstt. Prof., CE Deptt., TIT Bhiwani Lexical Analysis
- 2. Lexical Analysis • 1st phase of compiler. • Part of front end of compiler. • It is usually implemented as subroutine or co- routine of parser.
- 3. Lexical Analysis • Reads the source program as a file of characters and divide them up into tokens. • Tokens are recognized using Finite Automata or Transition Diagrams.
- 4. Lexical Analysis • The vital functions of Lexical Analyzer are:- – Separate tokens from program – Insert tokens into symbol table – Eliminates whitespaces, comments, newline characters, etc. from string. – Give tokens to parser (pass their integer no./code to parser) – Keep track of line numbers (for error handling). – Pre-processing of Macros.
- 5. Lexical Analysis • It reads characters from the input, groups them into lexeme and • Then, passes the tokens formed by the lexemes, together with their attribute value, to the parser. • In some situations ,it has to read some characters ahead before it can decide on the token to be returned to the parser.
- 6. Lexical Analysis • Ex: lexical analyzer for C must read ahead after it sees the character ‘>’. • If the next character is = ,then the character sequence >= is the lexeme forming the token for the “Greater than or equal to ” operator. • Other wise > is the lexeme forming “Greater than ” operator ,and the lexical analyzer has read one character too many.
- 7. Lexical Analysis • The extra character has to be pushed back on to the input, because it can be the beginning of the next lexeme in the input. • The lexical analyzer and parser form a producer- Consumer pair. • Produced tokens can be held in a token buffer until they are consumed.
- 8. Input Buffering • The interaction b/w the two is constrained only by the size of the buffer, because:- – lexical analyzer can not proceed when the buffer is full. – parser can not proceed when the buffer is empty. • Commonly,the buffer holds just one token. • Lexical Analyzer can be a procedure called by a parser, returning tokens on demand.
- 9. Input Buffering • The implementation of reading and pushing back character is usually done by setting up an input buffer. • A block of character is read into the buffer at a time. • A pointer keeps track of the portion of the input that has been analysed. • Pushing back a character is implemented by moving the pointer.
- 10. Input Buffering • Source program is read from HDD. • LA has to access secondary memory everytime it needs to identify tokens. • It may be time consuming, hence an input buffer is used. • LA scans input string from left to right one char. at a time to identify tokens.
- 11. Input Buffering • It uses 2 pointers: – Begin Pointer (bptr) – points to begin of string. – Look-ahead Pointer (lptr) – moves ahead to search for end of token. • Ex: int a, b;
- 12. Input Buffering
- 13. Input Buffering
- 14. Tokens, Lexemes & Patterns • Connected with lexical analysis are three important terms with similar meaning. • Lexeme • Token • Patterns
- 15. Tokens, Lexemes & Patterns • A token is a pair consisting of a token name and an optional attribute value. • Token names: – Keywords, – operators, – identifiers, – constants, – literal strings, – punctuation symbols(such as commas, semi-colons)
- 16. Tokens, Lexemes & Patterns A lexeme is a sequence of characters in the source program that matches the pattern for a token and is identified by the lexical analyzer as an instance of that token. Ex: Relation Operators {<,<=,>,>=,==,<>}
- 17. Tokens, Lexemes & Patterns • A pattern is a description of the form that the lexemes of token may take. • It gives an informal or formal description of a token. Ex: identifier • Gives a precise description/ specification of tokens. • Used to automatically generate a lexical analyzer
- 18. Tokens, Lexemes & Patterns
- 19. Tokens
- 20. Non-Tokens
- 21. Akhil Kaushik akhilkaushik05@gmail.com 9416910303 CONTACT ME AT: Akhil Kaushik akhilkaushik05@gmail.com 9416910303 THANK YOU !!!