Parsing Expression Grammars

Parsing Expression Grammars:
A RecognitionBased Syntactic Foundation
Bryan Ford
Massachusetts Institute of Technology
January 14, 2004

Designing a Language Syntax
1.Formalize syntax via
contextfree grammar
2.Write a YACC parser
specification
3.Hack on grammar
until “ nearLALR(1)”
4.Use generated parser
Textbook Method

Designing a Language Syntax
1.Formalize syntax via
contextfree grammar
2.Write a YACC parser
specification
3.Hack on grammar
until “ nearLALR(1)”
4.Use generated parser
1.Specify syntax
informally
2.Write a recursive
descent parser
Textbook Method Pragmatic Method


What exactly does a CFG describe?
Short answer:
a rule system to generate language strings
Example CFG:
S  aaS
S  
S
aaS
aa aaaaS
...
aaaa


Short answer:
Example CFG:
S  aaS
S  
S
aaS
aa aaaaS
...
aaaa
Start symbol


Short answer:
Example CFG:
S  aaS
S  
S
aaS
aa aaaaS
...
aaaa
Start symbol
Output strings

What exatly do we want to describe?
Proposed answer:
a rule system to recognize language strings
Parsing Expression Grammar (PEG)
models recursive descent parsing practice
Example PEG:
S  aaS / 
a a a a 
S
S
S
a a
a a

Proposed answer:
Example PEG:
S  aaS / 
a a a a 
S
S
S
a a
a a
Input
string

Proposed answer:
Example PEG:
S  aaS / 
a a a a 
S
S
S
a a
a a
Input
string
Derive
structure

TakeHome Points
Key benefits of PEGs:
● Simplicity, formalism, analyzability of CFGs
● Closer match to syntax practices
– More expressive than deterministic CFGs (LL/LR)
– More of the “ right kind” of expressiveness:
prioritized choice, greedy rules, syntactic predicates
– Unlimited lookahead, backtracking
● Lineartime parsing for any PEG

What kind of
recursive descent parsing?
Key assumptions:
● Parsing functions are stateless:
depend only on input string
● Parsing functions make decisions locally:
return at most one result (success/failure)

Parsing Expression Grammars
Consists of: (∑, N, R, eS)
– ∑: finite set of terminals (character set)
– N: finite set of nonterminals
– R: finite set of rules of the form “A  e”,
where A ∈ N, e is a parsing expression.
– eS: a parsing expression called the start expression.

Parsing Expressions
 the empty string
a terminal (a ∈ ∑)
A nonterminal (A ∈ N)
e1 e2 a sequence of parsing expressions
e1 / e2 prioritized choice between alternatives
e?, e*, e+ optional, zeroormore, oneormore
&e, !e syntactic predicates

How PEGs Express Languages
Given input string s, a parsing expression either:
– Matches and consumes a prefix s'
of s.
– Fails on s.
Example:
S  bad
S matches “ badder”
S matches “ baddest”
S fails on “ abad”
S fails on “ babe”

Prioritized Choice with Backtracking
S  A / B means:
“ To parse an S, first try to parse an A.
If A fails, then backtrack and try to parse a B.”
Example:
S  if C then S else S /
if C then S
S matches “ if C then S foo”
S matches “ if C then S1 else S2”
S fails on “ if C else S”

Prioritized Choice with Backtracking
S  A / B means:
“ To parse an S, first try to parse an A.
If A fails, then backtrack and try to parse a B.”
Example from the C++ standard:
“ An expressionstatement ... can be indistinguishable
from a declaration ... In those cases the statement is a
declaration.”
statement  declaration /
expressionstatement

Greedy Option and Repetition
A  e? equivalent to A  e / 
A  e* equivalent to A  e A / 
A  e+ equivalent to A  e e*
Example:
I  L+
L  a / b / c / ...
I matches “ foobar”
I matches “ foo(bar)”
I fails on “ 123”

Syntactic Predicates
Andpredicate: &e succeeds whenever e does,
but consumes no input [Parr '
94, '
95]
Notpredicate: !e succeeds whenever e fails
Example:
A  foo &(bar)
B  foo !(bar)
A matches “ foobar”
A fails on “ foobie”
B matches “ foobie”
B fails on “ foobar”

94, '
95]
Example:
C  B I* E
I  !E (C / T)
B  (*
E  *)
T  [any terminal]
C matches “ (*ab*)cd”
C matches “ (*a(*b*)c*)”
C fails on “ (*a(*b*)”

94, '
95]
Example:
C  B I* E
I  !E (C / T)
B  (*
E  *)
Begin marker

94, '
95]
Example:
C  B I* E
I  !E (C / T)
B  (*
E  *)
Internal elements

94, '
95]
Example:
C  B I* E
I  !E (C / T)
B  (*
E  *)
End marker

94, '
95]
Example:
C  B I* E
I  !E (C / T)
B  (*
E  *)
➔

94, '
95]
Example:
C  B I* E
I  !E (C / T)
B  (*
E  *)
Only if an end marker doesn'
t start here...
➔

94, '
95]
Example:
C  B I* E
I  !E (C / T)
B  (*
E  *)
Only if an end marker doesn'
t start here...
...consume a nested comment,
or else consume any single character.
➔

Unified Grammars
PEGs can express both lexical and hierarchical
syntax of realistic languages in one grammar
● Example (in paper):
Complete selfdescribing PEG in 2/3 column
● Example (on web):
Unified PEG for Java language

Lexical/Hierarchical Interplay
Unified grammars create new design opportunities
Example:
To get Unicode “ ∀”,
instead of “u2200”,
write “(0x2200)”
or “(8704)”
or “(FOR_ALL)”
E  S / ( E ) / ...
S  “ C* “
C  ( E ) /
!“ ! T

Example:
write “(0x2200)”
or “(8704)”
or “(FOR_ALL)”
E  S / ( E ) / ...
S  “ C* “
C  ( E ) /
!“ ! T
Generalpurpose expression syntax

Example:
write “(0x2200)”
or “(8704)”
or “(FOR_ALL)”
E  S / ( E ) / ...
S  “ C* “
C  ( E ) /
!“ ! T
String literals

Example:
write “(0x2200)”
or “(8704)”
or “(FOR_ALL)”
E  S / ( E ) / ...
S  “ C* “
C  ( E ) /
!“ ! T
Quotable characters

Formal Properties of PEGs
● Express all deterministic languages LR(k)
● Closed under union, intersection, complement
●
Some noncontext free languages, e.g., an
bn
cn
● Undecidable whether L(G) = ∅
● Predicate operators can be eliminated
– ...but the process is nontrivial!

Minimalist Forms
Predicatefree PEG
⇩
TS [Birman '
70/'
73]
TDPL [Aho '
72]
Any PEG
⇩
gTS [Birman '
70/'
73]
GTDPL [Aho '
72]
A  
A  a
A  f
A  BC / D
A  
A  a
A  f
A  B[C, D]
⇦⇨

Formal Contributions
● Generalize TDPL/GTDPL with more expressive
structured parsing expression syntax
● Negative syntactic predicate !e
● Predicate elimination transformation
– Intermediate stages depend on
generalized parsing expressions
● Proof of equivalence of TDPL and GTDPL

What can'
t PEGs express directly?
● Ambiguous languages
That's
what CFGs were designed for!
● Globally disambiguated languages?
– {a,b}n a {a,b}n ?
● State or semanticdependent syntax
– C, C++ typedef symbol tables
– Python, Haskell, ML layout

Generating Parsers from PEGs
Recursivedescent parsing
☞Simple & direct, but exponentialtime if not careful
Packrat parsing [Birman '
70/'
73, Ford '
02]
☞Lineartime, but can consume substantial storage
Classic LL/LR algorithms?
☞Grammar restrictions, but both time & spaceefficient

Conclusion
PEGs model common parsing practices
– Prioritized choice, greedy rules, syntactic predicates
PEGs naturally complement CFGs
– CFG: generative system, for ambiguous languages
– PEG: recognitionbased, for unambiguous languages
For more info:
http://pdos.lcs.mit.edu/~baford/packrat
(or G
Go
oo
og
gl
le
e for “ Packrat Parsing”)

Parsing Expression Grammars

More Related Content

Similar to Parsing Expression Grammars (20)

Recently uploaded (20)

Parsing Expression Grammars