Lecture 10 compiler i
0 likes463 views
The document discusses the syntax analysis phase of compilers. It introduces tokenizing, which breaks source code into individual tokens, and parsing, which checks that the tokens conform to the language's grammar. It provides examples of Jack code being tokenized and parsed. The parsing process involves matching tokens to rules in a grammar using a recursive descent parser. The output of syntax analysis is typically a parse tree that represents the structural relationship between tokens. The document outlines proposed implementations for a Jack tokenizer and parser that would perform syntax analysis on Jack code and output an XML representation of the parse tree.
Lecture 10 compiler i
- 1. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 1 www.nand2tetris.org Building a Modern Computer From First Principles Compiler I: Syntax Analysis
- 2. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 2 Course map Assembler Chapter 6 H.L. Language & Operating Sys. abstract interface Compiler Chapters 10 - 11 VM Translator Chapters 7 - 8 Computer Architecture Chapters 4 - 5 Gate Logic Chapters 1 - 3 Electrical Engineering Physics Virtual Machine abstract interface Software hierarchy Assembly Language abstract interface Hardware hierarchy Machine Language abstract interface Hardware Platform abstract interface Chips & Logic Gates abstract interface Human Thought Abstract design Chapters 9, 12
- 3. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 3 Motivation: Why study about compilers? Because Compilers … Are an essential part of applied computer science Are very relevant to computational linguistics Are implemented using classical programming techniques Employ important software engineering principles Train you in developing software for transforming one structure to another (programs, files, transactions, …) Train you to think in terms of ”description languages”.
- 4. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 4 The big picture . . . RISC machine other digital platforms, each equipped with its VM implementation RISC machine language Hack computer Hack machine language CISC machine language CISC machine . . . written in a high-level language Any computer . . . HW lectures (Projects 1-6) Intermediate code VM implementation over CISC platforms VM imp. over RISC platforms VM imp. over the Hack platform VM emulator VM lectures (Projects 7-8) Some Other language Jack language Some compiler Some Other compiler Jack compiler . . .Some language . . . Compiler lectures (Projects 10,11) Modern compilers are two-tiered: Front-end: from high-level language to some intermediate language Back-end: from the intermediate language to binary code.
- 5. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 5 Compiler architecture (front end) . . . Intermediate code RISC machine language Hack machine language CISC machine language . . . written in a high-level language . . . VM implementation over CISC platforms VM imp. over RISC platforms VM imp. over the Hack platform VM emulator Some Other language Jack language Some compiler Some Other compiler Jack compiler . . .Some language . . . Syntax analysis: understanding the semantics implied by the source code Code generation: reconstructing the semantics using the syntax of the target code. Tokenizing: creating a stream of “atoms” Parsing: matching the atom stream with the language grammar XML output = one way to demonstrate that the syntax analyzer works (Chapter 11)Jack Program Toke- nizer Parser Code Gene -ration Syntax Analyzer Jack Compiler VM code XML code (Chapter 10) (source) (target)
- 6. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 6 Tokenizing / Lexical analysis Remove white space Construct a token list (language atoms) Things to worry about: Language specific rules: e.g. how to treat “++” Language-specific classifications: keyword, symbol, identifier, integerCconstant, stringConstant,... While we are at it, we can have the tokenizer record not only the token, but also its lexical classification (as defined by the source language grammar).
- 7. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 7 Jack Tokenizer if (x < 153) {let city = ”Paris”;}if (x < 153) {let city = ”Paris”;} Source code <tokens> <keyword> if </keyword> <symbol> ( </symbol> <identifier> x </identifier> <symbol> < </symbol> <integerConstant> 153 </integerConstant> <symbol> ) </symbol> <symbol> { </symbol> <keyword> let </keyword> <identifier> city </identifier> <symbol> = </symbol> <stringConstant> Paris </stringConstant> <symbol> ; </symbol> <symbol> } </symbol> </tokens> <tokens> <keyword> if </keyword> <symbol> ( </symbol> <identifier> x </identifier> <symbol> < </symbol> <integerConstant> 153 </integerConstant> <symbol> ) </symbol> <symbol> { </symbol> <keyword> let </keyword> <identifier> city </identifier> <symbol> = </symbol> <stringConstant> Paris </stringConstant> <symbol> ; </symbol> <symbol> } </symbol> </tokens> Tokenizer’s output Tokenizer
- 8. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 8 Parsing The tokenizer discussed thus far is part of a larger program called parser Each language is characterized by a grammar. The parser is implemented to recognize this grammar in given texts The parsing process: A text is given and tokenized The parser determines weather or not the text can be generated from the grammar In the process, the parser performs a complete structural analysis of the text The text can be in an expression in a : Natural language (English, …) Programming language (Jack, …).
- 9. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 9 Parsing examples (5+3)*2 – sqrt(9*4) she discussed sex with her doctor - 5 sqrt + * 3 2 9 4 * Jack English discussed she sex with her doctor parse 1 discussed she with her doctor parse 2 sex
- 10. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 10 More examples of challenging parsing We gave the monkeys the bananas because they were hungry We gave the monkeys the bananas because they were over-ripe I never said she stole my money I never said she stole my money I never said she stole my money I never said she stole my money I never said she stole my money I never said she stole my money I never said she stole my money I never said she stole my money Time flies like an arrow
- 11. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 11 Simple (terminal) forms / complex (non-terminal) forms Grammar = set of rules on how to construct complex forms from simpler forms Highly recursive. A typical grammar of a typical C-like language while (expression) { if (expression) statement; while (expression) { statement; if (expression) statement; } while (expression) { statement; statement; } } if (expression) { statement; while (expression) statement; statement; } if (expression) if (expression) statement; } while (expression) { if (expression) statement; while (expression) { statement; if (expression) statement; } while (expression) { statement; statement; } } if (expression) { statement; while (expression) statement; statement; } if (expression) if (expression) statement; } Code sample program: statement; statement: whileStatement | ifStatement | // other statement possibilities ... | '{' statementSequence '}' whileStatement: 'while' '(' expression ')' statement ifStatement: simpleIf | ifElse simpleIf: 'if' '(' expression ')' statement ifElse: 'if' '(' expression ')' statement 'else' statement statementSequence: '' // null, i.e. the empty sequence | statement ';' statementSequence expression: // definition of an expression comes here // more definitions follow program: statement; statement: whileStatement | ifStatement | // other statement possibilities ... | '{' statementSequence '}' whileStatement: 'while' '(' expression ')' statement ifStatement: simpleIf | ifElse simpleIf: 'if' '(' expression ')' statement ifElse: 'if' '(' expression ')' statement 'else' statement statementSequence: '' // null, i.e. the empty sequence | statement ';' statementSequence expression: // definition of an expression comes here // more definitions follow Grammar
- 12. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 12 Parse tree while . . .( )count <= 100 { count ++ statement whileStatement expression statementSequence statement ; statement statementSequence Input Text: while (count<=100) { /** demonstration */ count++; // ... Tokenized: while ( count <= 100 ) { count ++ ; ... program: statement; statement: whileStatement | ifStatement | // other statement possibilities ... | '{' statementSequence '}' whileStatement: 'while' '(' expression ')' statement ... program: statement; statement: whileStatement | ifStatement | // other statement possibilities ... | '{' statementSequence '}' whileStatement: 'while' '(' expression ')' statement ...
- 13. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 13 Recursive descent parsing Parser implementation: a set of parsing methods, one for each rule: parseStatement() parseWhileStatement() parseIfStatement() parseStatementSequence() parseExpression(). Highly recursive LL(0) grammars: the first token determines in which rule we are In other grammars you have to look ahead 1 or more tokens Jack is almost LL(0). while (expression) { statement; statement; while (expression) { while (expression) statement; statement; } } while (expression) { statement; statement; while (expression) { while (expression) statement; statement; } } code sample
- 14. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 14 A linguist view on parsing Parsing: One of the mental processes involved in sentence comprehension, in which the listener determines the syntactic categories of the words, joins them up in a tree, and identifies the subject, object, and predicate, a prerequisite to determining who did what to whom from the information in the sentence. (Steven Pinker, The Language Instinct)
- 15. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 15 The Jack grammar ’x’: x appears verbatim x: x is a language construct x?: x appears 0 or 1 times x*: x appears 0 or more times x|y: either x or y appears (x,y): x appears, then y. ’x’: x appears verbatim x: x is a language construct x?: x appears 0 or 1 times x*: x appears 0 or more times x|y: either x or y appears (x,y): x appears, then y.
- 16. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 16 The Jack grammar (cont.) ’x’: x appears verbatim x: x is a language construct x?: x appears 0 or 1 times x*: x appears 0 or more times x|y: either x or y appears (x,y): x appears, then y. ’x’: x appears verbatim x: x is a language construct x?: x appears 0 or 1 times x*: x appears 0 or more times x|y: either x or y appears (x,y): x appears, then y.
- 17. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 17 Jack syntax analyzer in action Class Bar { method Fraction foo(int y) { var int temp; // a variable let temp = (xxx+12)*-63; ... ... Class Bar { method Fraction foo(int y) { var int temp; // a variable let temp = (xxx+12)*-63; ... ... Syntax analyzer Using the language grammar, a programmer can write a syntax analyzer program (parser) The syntax analyzer takes a source text file and attempts to match it on the language grammar If successful, it can generate a parse tree in some structured format, e.g. XML. <varDec> <keyword> var </keyword> <keyword> int </keyword> <identifier> temp </identifier> <symbol> ; </symbol> </varDec> <statements> <letStatement> <keyword> let </keyword> <identifier> temp </identifier> <symbol> = </symbol> <expression> <term> <symbol> ( </symbol> <expression> <term> <identifier> xxx </identifier> </term> <symbol> + </symbol> <term> <int.Const.> 12 </int.Const.> </term> </expression> ... <varDec> <keyword> var </keyword> <keyword> int </keyword> <identifier> temp </identifier> <symbol> ; </symbol> </varDec> <statements> <letStatement> <keyword> let </keyword> <identifier> temp </identifier> <symbol> = </symbol> <expression> <term> <symbol> ( </symbol> <expression> <term> <identifier> xxx </identifier> </term> <symbol> + </symbol> <term> <int.Const.> 12 </int.Const.> </term> </expression> ... Syntax analyzer The syntax analyzer’s algorithm shown in this slide: If xxx is non-terminal, output: <xxx> Recursive code for the body of xxx </xxx> If xxx is terminal (keyword, symbol, constant, or identifier) , output: <xxx> xxx value </xxx>
- 18. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 18 JackTokenizer: a tokenizer for the Jack language (proposed implementation)
- 19. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 19 JackTokenizer (cont.)
- 20. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 20 CompilationEngine: a recursive top-down parser for Jack The CompilationEngine effects the actual compilation output. It gets its input from a JackTokenizer and emits its parsed structure into an output file/stream. The output is generated by a series of compilexxx() routines, one for every syntactic element xxx of the Jack grammar. The contract between these routines is that each compilexxx() routine should read the syntactic construct xxx from the input, advance() the tokenizer exactly beyond xxx, and output the parsing of xxx. Thus, compilexxx()may only be called if indeed xxx is the next syntactic element of the input. In the first version of the compiler, which we now build, this module emits a structured printout of the code, wrapped in XML tags (defined in the specs of project 10). In the final version of the compiler, this module generates executable VM code (defined in the specs of project 11). In both cases, the parsing logic and module API are exactly the same.
- 21. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 21 CompilationEngine (cont.)
- 22. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 22 CompilationEngine (cont.)
- 23. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 23 CompilationEngine (cont.)
- 24. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 24 Summary and next step (Chapter 11)Jack Program Toke- nizer Parser Code Gene -ration Syntax Analyzer Jack Compiler VM code XML code (Chapter 10) Syntax analysis: understanding syntax Code generation: constructing semantics The code generation challenge: Extend the syntax analyzer into a full-blown compiler that, instead of generating passive XML code, generates executable VM code Two challenges: (a) handling data, and (b) handling commands.
- 25. Elements of Computing Systems, Nisan & Schocken, MIT Press, www.nand2tetris.org , Chapter 10: Compiler I: Syntax Analysis slide 25 Perspective The parse tree can be constructed on the fly Syntax analyzers can be built using: Lex tool for tokenizing Yacc tool for parsing Do everything from scratch (our approach ...) The Jack language is intentionally simple: Statement prefixes: let, do, ... No operator priority No error checking Basic data types, etc. Richer languages require more powerful compilers The Jack compiler: designed to illustrate the key ideas that underlie modern compilers, leaving advanced features to more advanced courses Industrial-strength compilers: Have good error diagnostics Generate tight and efficient code Support parallel (multi-core) processors.