Lex and Yacc ppt

Lex Yacc tutorial
Kun-Yuan Hsieh
kyshieh@pllab.cs.nthu.edu.tw
Programming Language Lab., NTHU
PLLab, NTHU,Cs2403 Programming
Languages
1

Languages
2
Overview
take a glance at Lex!

Compilation Sequence
Languages
3

Languages
4
What is Lex?
• The main job of a lexical analyzer
(scanner) is to break up an input stream
into more usable elements (tokens)
a = b + c * d;
ID ASSIGN ID PLUS ID MULT ID SEMI
• Lex is an utility to help you rapidly
generate your scanners

Lex – Lexical Analyzer
• Lexical analyzers tokenize input streams
• Tokens are the terminals of a language
– English
Languages
5
• words, punctuation marks, …
– Programming language
• Identifiers, operators, keywords, …
• Regular expressions define
terminals/tokens

Lex Source Program
Languages
6
• Lex source is a table of
– regular expressions and
– corresponding program fragments
digit [0-9]
letter [a-zA-Z]
%%
{letter}({letter}|{digit})* printf(“id: %sn”, yytext);
n printf(“new linen”);
%%
main() {
yylex();
}

Lex Source to C Program
• The table is translated to a C program
(lex.yy.c) which
– reads an input stream
– partitioning the input into strings which
match the given expressions and
– copying it to an output stream if necessary
Languages
7

An Overview of Lex
Languages
8
Lex
C compiler
a.out
Lex source
program
lex.yy.c
input
lex.yy.c
a.out
tokens

(required)
(optional)
{definitions}
%%
{transition rules}
%%
{user subroutines}
Languages
9
Lex Source
• Lex source is separated into three sections by %
% delimiters
• The general format of Lex source is
• The %%
absolute minimum Lex program is thus

Languages
10
Lex v.s. Yacc
• Lex
– Lex generates C code for a lexical analyzer, or
scanner
– Lex uses patterns that match strings in the input
and converts the strings to tokens
• Yacc
– Yacc generates C code for syntax analyzer, or
parser.
– Yacc uses grammar rules that allow it to analyze
tokens from Lex and create a syntax tree.

Lex source
(Lexical Rules)
Yacc source
(Grammar Rules)
call
Input Parsed
Languages
11
Lex with Yacc
Lex Yacc
yylex() yyparse()
Input
lex.yy.c y.tab.c
return token

Regular Expressions
Languages
12

Lex Regular Expressions
(Extended Regular Expressions)
• A regular expression matches a set of strings
• Regular expression
Languages
13
– Operators
– Character classes
– Arbitrary character
– Optional expressions
– Alternation and grouping
– Context sensitivity
– Repetitions and definitions

Languages
14
Operators
“ [ ] ^ - ? . * + | ( ) $ / { } % < >
• If they are to be used as text characters, an
escape should be used
$ = “$”
= “”
• Every character but blank, tab (t), newline (n)
and the list above is always a text character

Character Classes []
• [abc] matches a single character, which may
be a, b, or c
• Every operator meaning is ignored except -
and ^
• e.g.
[ab] => a or b
[a-z] => a or b or c or … or z
[-+0-9] => all the digits and the two signs
[^a-zA-Z] => any character which is not a
Languages
15
letter

Arbitrary Character .
• To match almost character, the
operator character . is the class of all
characters except newline
• [40-176] matches all printable
characters in the ASCII character set,
from octal 40 (blank) to octal 176
(tilde~)
Languages
16

Optional & Repeated
Languages
17
Expressions
• a? => zero or one instance of a
• a* => zero or more instances of a
• a+ => one or more instances of a
• E.g.
ab?c => ac or abc
[a-z]+ => all strings of lower case letters
[a-zA-Z][a-zA-Z0-9]* => all
alphanumeric strings with a leading
alphabetic character

Precedence of Operators
• Level of precedence
– Kleene closure (*), ?, +
– concatenation
– alternation (|)
• All operators are left associative.
• Ex: a*b|cd* = ((a*)b)|(c(d*))
Languages
18

Pattern Matching Primitives
Metacharacter Matches
. any character except newline
n newline
* zero or more copies of the preceding expression
+ one or more copies of the preceding expression
? zero or one copy of the preceding expression
^ beginning of line / complement
$ end of line
a|b a or b
(ab)+ one or more copies of ab (grouping)
[ab] a or b
a{3} 3 instances of a
“a+b” literal “a+b” (C escapes still work)
Languages
19

Recall: Lex Source
a = b + c;
a operator: ASSIGNMENT b + c;
Languages
20
• Lex source is a table of
– regular expressions and
– corresponding program fragments (actions)
…
%%
<regexp> <action>
<regexp> <action>
…
%%
%%
“=“ printf(“operator: ASSIGNMENT”);

Transition Rules
• regexp <one or more blanks> action (C code);
• regexp <one or more blanks> { actions (C code) }
• A null statement ; will ignore the input (no
actions)
[ tn] ;
– Causes the three spacing characters to be ignored a = b + c;
Languages
21
d = b * c;
↓ ↓
a=b+c;d=b*c;

Transition Rules (cont’d)
• Four special options for actions:
|, ECHO;, BEGIN, and REJECT;
• | indicates that the action for this rule is from
the action for the next rule
Languages
22
– [ tn] ;
– “ “ |
“t” |
“n” ;
• The unmatched token is using a default
action that ECHO from the input to the output

Transition Rules (cont’d)
Languages
23
• REJECT
– Go do the next alternative
…
%%
pink {npink++; REJECT;}
ink {nink++; REJECT;}
pin {npin++; REJECT;}
. |
n ;
%%
…

Lex Predefined Variables
Languages
24
• yytext -- a string containing the lexeme
• yyleng -- the length of the lexeme
• yyin -- the input stream pointer
– the default input of default main() is stdin
• yyout -- the output stream pointer
– the default output of default main() is stdout.
• cs20: %./a.out < inputfile > outfile
• E.g.
[a-z]+ printf(“%s”, yytext);
[a-z]+ ECHO;
[a-zA-Z]+ {words++; chars += yyleng;}

Lex Library Routines
Languages
25
• yylex()
– The default main() contains a call of yylex()
• yymore()
– return the next token
• yyless(n)
– retain the first n characters in yytext
• yywarp()
– is called whenever Lex reaches an end-of-file
– The default yywarp() always returns 1

Review of Lex Predefined
Languages
26
Variables
Name Function
char *yytext pointer to matched string
int yyleng length of matched string
FILE *yyin input stream pointer
FILE *yyout output stream pointer
int yylex(void) call to invoke lexer, returns token
char* yymore(void) return the next token
int yyless(int n) retain the first n characters in yytext
int yywrap(void) wrapup, return 1 if done, 0 if not done
ECHO write matched string
REJECT go to the next alternative rule
INITAL initial start condition
BEGIN condition switch start condition

User Subroutines Section
• You can use your Lex routines in the same
ways you use routines in other programming
languages.
Languages
27
%{
void foo();
%}
letter [a-zA-Z]
%%
{letter}+ foo();
%%
…
void foo() {
…
}

User Subroutines Section
Languages
28
(cont’d)
• The section where main() is placed
%{
int counter = 0;
%}
letter [a-zA-Z]
%%
{letter}+ {printf(“a wordn”); counter++;}
%%
main() {
yylex();
printf(“There are total %d wordsn”, counter);
}

Languages
29
Usage
• To run Lex on a source file, type
lex scanner.l
• It produces a file named lex.yy.c which is
a C program for the lexical analyzer.
• To compile lex.yy.c, type
cc lex.yy.c –ll
• To run the lexical analyzer program, type
./a.out < inputfile

Languages
30
Versions of Lex
• AT&T -- lex
http://www.combo.org/lex_yacc_page/lex.html
• GNU -- flex
http://www.gnu.org/manual/flex-2.5.4/flex.html
• a Win32 version of flex :
http://www.monmouth.com/~wstreett/lex-yacc/lex-yacc.html
or Cygwin :
http://sources.redhat.com/cygwin/
• Lex on different machines is not created equal.

Yacc - Yet Another Compiler-
Languages
31
Compiler

Languages
32
Introduction
• What is YACC ?
– Tool which will produce a parser for a
given grammar.
– YACC (Yet Another Compiler Compiler)
is a program designed to compile a
LALR(1) grammar and to produce the
source code of the syntactic analyzer of
the language produced by this grammar.

How YACC Works
Languages
33
a.out
File containing desired
grammar in yacc format
yyaacccc pprrooggrraamm
C source program created by yacc
CC ccoommppiilleerr
Executable program that will parse
grammar given in gram.y
gram.y
yacc
y.tab.c
cc
or gcc

How YACC Works
y.output
y.tab.c a.out
Languages
34
yacc
(1) Parser generation time
YACC source (*.y)
y.tab.h
y.tab.c
C compiler/linker
(2) Compile time
a.out
(3) Run time
Token stream
Abstract
Syntax
Tree

An YACC File Example
Languages
35
%{
#include <stdio.h>
%}
%token NAME NUMBER
%%
statement: NAME '=' expression
| expression { printf("= %dn", $1); }
;
expression: expression '+' NUMBER { $$ = $1 + $3; }
| expression '-' NUMBER { $$ = $1 - $3; }
| NUMBER { $$ = $1; }
;
%%
int yyerror(char *s)
{
fprintf(stderr, "%sn", s);
return 0;
}
int main(void)
{
yyparse();
return 0;
}

Languages
36
Works with Lex
How to
work ?

Languages
37
Works with Lex
call yylex() [0-9]+
next token is NUM
NUM ‘+’ NUM

YACC File Format
Languages
38
%{
C declarations
%}
yacc declarations
%%
Grammar rules
%%
Additional C code
– Comments enclosed in /* ... */ may appear in
any of the sections.

Definitions Section
Languages
39
%{
#include <stdio.h>
#include <stdlib.h>
%}
%token ID NUM
%start expr
It is a terminal
由 expr 開始
parse

Languages
40
Start Symbol
• The first non-terminal specified in the
grammar specification section.
• To overwrite it with %start declaraction.
%start non-terminal

Languages
41
Rules Section
• This section defines grammar
• Example
expr : expr '+' term | term;
term : term '*' factor | factor;
factor : '(' expr ')' | ID | NUM;

Languages
42
Rules Section
• Normally written like this
• Example:
expr : expr '+' term
| term
;
term : term '*' factor
| factor
;
factor : '(' expr ')'
| ID
| NUM
;

The Position of Rules
expr : expr '+' term { $$ = $1 + $3; }
| term { $$ = $1; }
;
term : term '*' factor { $$ = $1 * $3; }
| factor { $$ = $1; }
;
factor : '(' expr ')' { $$ = $2; }
Languages
43
| ID
| NUM
;

expr : eexxpprr '+' term { $$ = $1 + $3; }
| tteerrmm { $$ = $1; }
;
term : tteerrmm '*' factor { $$ = $1 * $3; }
| ffaaccttoorr { $$ = $1; }
;
factor : '((' expr ')' { $$ = $2; }
Languages
44
| ID
| NUM
;
$$11

expr : expr '++' term { $$ = $1 + $3; }
| term { $$ = $1; }
;
term : term '**' factor { $$ = $1 * $3; }
| factor { $$ = $1; }
;
factor : '(' eexxpprr ')' { $$ = $2; }
Languages
45
| ID
| NUM
; $$22

expr : expr '+' tteerrmm { $$ = $1 + $3; }
| term { $$ = $1; }
;
term : term '*' ffaaccttoorr { $$ = $1 * $3; }
| factor { $$ = $1; }
;
factor : '(' expr '))' { $$ = $2; }
| ID
| NUM
; $$33 DDeeffaauulltt:: $$$$ == $$11;;
Languages
46

Communication between LEX and
Languages
47
YACC
call yylex() [0-9]+
next token is NUM
NUM ‘+’ NUM
LEX and YACC需要一套方法確認token的身份

Communication between LEX
Languages
48
and YACC
yacc -d gram.y
Will produce:
y.tab.h
• Use enumeration ( 列舉 ) /
define
• 由一方產生，另一方
include
• YACC 產生 y.tab.h
• LEX include y.tab.h

Communication between LEX
scanner.l
Languages
49
and YACC
%{
#include <stdio.h>
#include "y.tab.h"
%}
id [_a-zA-Z][_a-zA-Z0-9]*
%%
int { return INT; }
char { return CHAR; }
float { return FLOAT; }
{id} { return ID;}
%{
#include <stdio.h>
#include <stdlib.h>
%}
%token CHAR, FLOAT, ID, INT
%%
yacc -d xxx.y
Produced
y.tab.h:
# define CHAR 258
# define FLOAT 259
# define ID 260
# define INT 261
parser.y

Phrase -> cart_animal AND CART
| work_animal AND PLOW…
Languages
50
YACC
• Rules may be recursive
• Rules may be ambiguous*
• Uses bottom up Shift/Reduce parsing
– Get a token
– Push onto stack
– Can it reduced (How do we know?)
• If yes: Reduce using a rule
• If no: Get another token
• Yacc cannot look ahead more than one token

Languages
51
Yacc Example
• Taken from Lex & Yacc
• Simple calculator
a = 4 + 6
a
a=10
b = 7
c = a + b
c
c = 17
$

Languages
53
statement_list: statement 'n'
| statement_list statement 'n'
;
statement: NAME '=' expression { $1->value = $3; }
| expression { printf("= %gn", $1); }
;
expression: expression '+' term { $$ = $1 + $3; }
| expression '-' term { $$ = $1 - $3; }
| term
;
parser.y
Parser (cont’d)

Parser (cont’d)
Languages
54
term: term '*' factor { $$ = $1 * $3; }
| term '/' factor { if ($3 == 0.0)
yyerror("divide by zero");
else
$$ = $1 / $3;
}
| factor
;
factor: '(' expression ')' { $$ = $2; }
| '-' factor { $$ = -$2; }
| NUMBER { $$ = $1; }
| NAME { $$ = $1->value; }
;
%% parser.y

Scanner
%{
#include "y.tab.h"
#include "parser.h"
#include <math.h>
%}
%%
([0-9]+|([0-9]*.[0-9]+)([eE][-+]?[0-9]+)?) {
Languages
55
yylval.dval = atof(yytext);
return NUMBER;
}
[ t] ; /* ignore white space */
scanner.l

Scanner (cont’d)
[A-Za-z][A-Za-z0-9]* { /* return symbol pointer */
yylval.symp = symlook(yytext);
return NAME;
Languages
56
}
"$" { return 0; /* end of input */ }
n|”=“|”+”|”-”|”*”|”/” return yytext[0];
%%
scanner.l

YACC Command
Languages
57
• Yacc (AT&T)
– yacc –d xxx.y
• Bison (GNU)
– bison –d –y xxx.y
產生y.tab.c, 與yacc相同
不然會產生xxx.tab.c

(1) 1 – 2 - 3
(2) 1 – 2 * 3
Languages
58
Precedence /
Association
1. 1-2-3 = (1-2)-3? or 1-(2-3)?
Define ‘-’ operator is left-association.
2. 1-2*3 = 1-(2*3)
Define “*” operator is precedent to “-”
operator

Languages
59
Precedence /
Association
%right ‘=‘
%left '<' '>' NE LE GE
%left '+' '-‘
%left '*' '/'
highest precedence

%left '+' '-'
%left '*' '/'
%noassoc UMINUS
Languages
60
Precedence /
Association
expr : expr ‘+’ expr { $$ = $1 + $3; }
| expr ‘-’ expr { $$ = $1 - $3; }
| expr ‘*’ expr { $$ = $1 * $3; }
| expr ‘/’ expr
{
if($3==0)
yyerror(“divide 0”);
else
$$ = $1 / $3;
}
| ‘-’ expr %prec UMINUS {$$ = -$2; }

Shift/Reduce Conflicts
• shift/reduce conflict
– occurs when a grammar is written in
such a way that a decision between
shifting and reducing can not be made.
– ex: IF-ELSE ambigious.
• To resolve this conflict, yacc will
choose to shift.
Languages
61

YACC Declaration
Languages
62
Summary `%start'
Specify the grammar's start symbol
`%union'
Declare the collection of data types that semantic values may
have
`%token'
Declare a terminal symbol (token type name) with no
precedence or
associativity specified
`%type'
Declare the type of semantic values for a nonterminal symbol

YACC Declaration
Languages
63
`%right' Summary
Declare a terminal symbol (token type name) that is
right-associative
`%left'
Declare a terminal symbol (token type name) that is left-associative
`%nonassoc'
Declare a terminal symbol (token type name) that is
nonassociative
(using it in a way that would be associative is a syntax error,
ex: x op. y op. z is syntax error)

Reference Books
Languages
64
• lex & yacc, 2nd Edition
– by John R.Levine, Tony Mason & Doug
Brown
– O’Reilly
– ISBN: 1-56592-000-7
• Mastering Regular Expressions
– by Jeffrey E.F. Friedl
– O’Reilly
– ISBN: 1-56592-257-3

Lex and Yacc ppt

More Related Content

What's hot (20)

Similar to Lex and Yacc ppt (20)

Recently uploaded (20)

Lex and Yacc ppt