SlideShare a Scribd company logo

Recursive descent parsing

Boy Baukema, profile picture
Boy Baukema

The document discusses recursive descent parsing and describes building a parser for print_r output in PHP using a parsing expression grammar and scannerless predictive recursive descent parsing. It outlines a 3 version approach: V1 parses an empty array, V2 parses an array of strings, and V3 parses nested arrays. The document emphasizes that writing parsers by hand can prevent issues seen when using regexes or other methods to parse structured data.

TechnologyEducation
1 of 53
Downloaded 37 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
Recursive Descent Parsing In practice with PHP
Plan for the next 40 mins 1. Walk through creating a Parsing Expression Grammar and scannerless predictive recursive descent parser for a subset of print_r output. 2. Talk about why anyone would want to do such a thing. Source Code: https://bit.ly/dpc14rdp
Disclaimer: I am not …
I am Boy Baukema Senior Software Engineer @ ibuildings.nl
print_r (PHP 4, PHP 5) print_r — Prints human-readable information about a variable
An example Array ( [Talk] => Array ( [Title] => Ansible: Orchestrate [Type] => 3
Just one problem… ! ! It’s unparsable.
No escaping > print_r(array("a"=>"n [b] => evil")); Array ( [a] => [b] => evil )
print_r * for anything non-trivial
–Martin Fowler “…it’s a technique that isn't as widely known as it should be. Many people are under the impression that using it is quite hard. I think that this fear often comes from the fact that Syntax- Directed Translation is usually described in the context of parsing a general-purpose language—which introduces a lot of complexities that you don't face with a DSL.”
V1 - An empty array Source Code: https://bit.ly/dpc14rdp
> print_r(array()); Array ( )
ARRAY <- ARRAY_START LF PAREN_OPEN LF PAREN_CLOSE LF
ARRAY_START <- ‘Array’ LF <- “n” PAREN_OPEN <- ‘(’ PAREN_CLOSE <- ‘)’
PrintRLang V1 RecursiveDescentParser - $content : string + __construct ( string $content ) + consume ( string $terminal ) + lookAhead ( string $terminal ) Source Code: https://bit.ly/dpc14rdp
PrintRLang V1 
 ArrayParser - $parser : RecursiveDescentParser + __construct(RecursiveDescentParser $parser) + parse(): array + arrayStart() + lf() + braceOpen() + braceClose() Source Code: https://bit.ly/dpc14rdp
! $parser = new PrintRLang ArrayParser( new PrintRLang RecursiveDescentParser( "Arrayn(n)n" ) ); $parser->parse();
public function parse() { $this->arrayStart(); $this->lf(); $this->braceOpen(); $this->lf() $this->braceClose(); $this->lf(); return array(); }
A r r a y n ( n ) n public function arrayStart() { $this->parser->consume('Array'); } n ( n ) n
n ( n ) n public function lf() { $this->parser->consume("n"); } ( n ) n
( n ) n public function braceOpen() { $this->parser->consume('('); } n ) n
n ) n public function lf() { $this->parser->consume("n"); } ) n
) n public function braceClose() { $this->parser->consume(')'); } n
n public function lf() { $this->parser->consume("n"); }
V2 - Array of strings Source Code: https://bit.ly/dpc14rdp
Array ( [Room] => E104 [Difficulty] => 2 [Type] => 1 )
ARRAY <- ARRAY_START LF PAREN_OPEN LF ARRAY_ASSIGN* PAREN_CLOSE LF
Kleene star translates to: ARRAY_ASSIGN* while (lookAhead(' ')) $result = arrayAssign($result)
ARRAY_ASSIGN <- SPACE+ ARRAY_KEY SPACE FAT_ARROW SPACE ARRAY_VALUE LF
Kleene plus SPACE+ === SPACE SPACE*
Kleene plus implemented space() while (lookAhead(' ')) space()
ARRAY_KEY <- BRACKET_OPEN KEY_VALUE BRACKET_CLOSE KEY_VALUE <-!BRACKET_CLOSE
ARRAY_VALUE <- !LF
PrintRLang V2 RecursiveDescentParser - $content : string + __construct ( string $content ) + consume ( string $terminal ) + consumeRegex( string $regex ) + lookAhead ( string $terminal ) + lookAheadRegex( string $regex Source Code: https://bit.ly/dpc14rdp
PrintRLang V2 
 ArrayParser - $parser : RecursiveDescentParser ... + arrayAssign( array $result ) + arrayKey() : string + arrayValue() : string + space() + fatArrow ...
V3 - Array of Arrays
Array ( [Talk] => Array ( [Title] => Ansible: Orchestrate [Type] => 3 ) )
ARRAY_VALUE <- ARRAY / STRING STRING <- !LF
ARRAY <- ARRAY_START LF SPACE* PAREN_OPEN LF ARRAY_ASSIGN* SPACE* PAREN_CLOSE
PrintRLang V3 
 ArrayParser - $parser : RecursiveDescentParser ... + string() ...
Why?
– Steve Yegge, Rich Programmer Food “If you don't know how parsing works, you'll do it badly with regular expressions, or if you don't know those, then with hand-rolled state machines that are thousands of lines of incomprehensible code that doesn't actually work.”
Mail::RFC822::Address (?:(?:rn)?[ t])*(?:(?:(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t] )+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?: rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:( ?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*))*@(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-0 31]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)* ](?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+ (?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?: (?:rn)?[ t])*))*|(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z |(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn) ?[ t])*)*<(?:(?:rn)?[ t])*(?:@(?:[^()<>@,;:".[] 000-031]+(?:(?:(?: rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[
t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn) ?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t] )*))*(?:,@(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])* )(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t] )+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*))*) *:(?:(?:rn)?[ t])*)?(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+ |Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:r n)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?: rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t ]))*"(?:(?:rn)?[ t])*))*@(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031 ]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*]( ?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(? :(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(? :rn)?[ t])*))*>(?:(?:rn)?[ t])*)|(?:[^()<>@,;:".[] 000-031]+(?:(? :(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)? [ t]))*"(?:(?:rn)?[ t])*)*:(?:(?:rn)?[ t])*(?:(?:(?:[^()<>@,;:".[]
000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]| .|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<> @,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|" (?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*))*@(?:(?:rn)?[ t] )*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;: ".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(? :[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[ ]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*))*|(?:[^()<>@,;:".[] 000- 031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|( ?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*)*<(?:(?:rn)?[ t])*(?:@(?:[^()<>@,; :".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([ ^[]r]|.)*](?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:" .[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[ ]r]|.)*](?:(?:rn)?[ t])*))*(?:,@(?:(?:rn)?[ t])*(?:[^()<>@,;:". [] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[] r]|.)*](?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[]
000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r] |.)*](?:(?:rn)?[ t])*))*)*:(?:(?:rn)?[ t])*)?(?:[^()<>@,;:".[] 0 00-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]| .|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@, ;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|"(? :[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*))*@(?:(?:rn)?[ t])* (?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:". []]))|[([^[]r]|.)*](?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[ ^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[] ]))|[([^[]r]|.)*](?:(?:rn)?[ t])*))*>(?:(?:rn)?[ t])*)(?:,s*( ?:(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;: ".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*)(?:.(?:( ?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[ ["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t ])*))*@(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t ])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*)(? :.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|
Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*))*|(?: [^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[ ]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*)*<(?:(?:rn) ?[ t])*(?:@(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[[" ()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*)(?:.(?:(?:rn) ?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<> @,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*))*(?:,@(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@, ;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t] )*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;: ".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*))*)*:(?:(?:rn)?[ t])*)? (?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:". []]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*)(?:.(?:(?: rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[[ "()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t]) *))*@(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t]) +|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*)(?: .(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z |(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*))*>(?:( ?:rn)?[ t])*))*)?;s*)
Useful applications I’ve seen REST API with CQL querying (MediaMosa.org) Migrating wiki content Parsing log files Parsing obscure specifications (ARF) Concise configuration files Domain Specific Languages
–Martin Fowler, Domain Specific Languages “a DSL is a front-end to a library providing a different style of manipulation to the command-query API. ”
Rules for building a parser Consider using an existing parser. Consider porting one from another language. Consider XML or the new XMLs: JSON / YAML Consider working around it. Then and only then consider building your own parser
Whereto from here?
Let’s build a parser! http://protalk.me/dpcradio-lets-build-a-parser
Thank you for your time and attention! Questions? Tweet to @relaxnow Rate @ https://joind.in/10859 Slides @ https://joind.in/10859 Code @ https://bit.ly/dpc14rdp

More Related Content

PPTX
FORESTS
MSharmilaDeviITDEPT
 
PPTX
Segmentation in operating systems
Dr. Jasmine Beulah Gnanadurai
 
PPTX
Top down and botttom up Parsing
Gerwin Ocsena
 
PPTX
RECURSIVE DESCENT PARSING
Jothi Lakshmi
 
PDF
Implementation of Pipe in Linux
Tushar B Kute
 
PPT
Data race
James Wong
 
PDF
Intro to AI STRIPS Planning & Applications in Video-games Lecture6-Part1
Stavros Vassos
 
PDF
Dbms 14: Relational Calculus
Amiya9439793168
 
FORESTS
MSharmilaDeviITDEPT
 
Segmentation in operating systems
Dr. Jasmine Beulah Gnanadurai
 
Top down and botttom up Parsing
Gerwin Ocsena
 
RECURSIVE DESCENT PARSING
Jothi Lakshmi
 
Implementation of Pipe in Linux
Tushar B Kute
 
Data race
James Wong
 
Intro to AI STRIPS Planning & Applications in Video-games Lecture6-Part1
Stavros Vassos
 
Dbms 14: Relational Calculus
Amiya9439793168
 

What's hot (20)

PPTX
Church Turing Thesis
Hemant Sharma
 
PDF
2.2. interactive computer graphics
Ratnadeepsinh Jadeja
 
PDF
Memory management
Rajni Sirohi
 
PDF
Stemming And Lemmatization Tutorial | Natural Language Processing (NLP) With ...
Edureka!
 
PDF
Hardware Attacks and Security
Priyanka Aash
 
PPT
Concurrency
Naveen Sihag
 
PDF
bag-of-words models
Xiaotao Zou
 
PPTX
The role of the parser and Error recovery strategies ppt in compiler design
Sadia Akter
 
PDF
Compiler Design- Machine Independent Optimizations
Jyothishmathi Institute of Technology and Science Karimnagar
 
PPTX
Daa:Dynamic Programing
rupali_2bonde
 
PPTX
Ngrams smoothing
Digvijay Singh
 
PPTX
Syntax Analysis in Compiler Design
MAHASREEM
 
PDF
Syntax analysis
Akshaya Arunan
 
PPTX
Word embedding
ShivaniChoudhary74
 
PPT
12-Syntax Directed Definition – Evaluation Order-09-06-2023.ppt
venkatapranaykumarGa
 
PPTX
Lisp
Aniruddha Chakrabarti
 
PPTX
Structure of the compiler
Sudhaa Ravi
 
PDF
Computer Graphics - Output Primitive
Rupesh Mishra
 
PPT
Unix memory management
Tech_MX
 
PPTX
unit-4-dynamic programming
hodcsencet
 
Church Turing Thesis
Hemant Sharma
 
2.2. interactive computer graphics
Ratnadeepsinh Jadeja
 
Memory management
Rajni Sirohi
 
Stemming And Lemmatization Tutorial | Natural Language Processing (NLP) With ...
Edureka!
 
Hardware Attacks and Security
Priyanka Aash
 
Concurrency
Naveen Sihag
 
bag-of-words models
Xiaotao Zou
 
The role of the parser and Error recovery strategies ppt in compiler design
Sadia Akter
 
Compiler Design- Machine Independent Optimizations
Jyothishmathi Institute of Technology and Science Karimnagar
 
Daa:Dynamic Programing
rupali_2bonde
 
Ngrams smoothing
Digvijay Singh
 
Syntax Analysis in Compiler Design
MAHASREEM
 
Syntax analysis
Akshaya Arunan
 
Word embedding
ShivaniChoudhary74
 
12-Syntax Directed Definition – Evaluation Order-09-06-2023.ppt
venkatapranaykumarGa
 
Lisp
Aniruddha Chakrabarti
 
Structure of the compiler
Sudhaa Ravi
 
Computer Graphics - Output Primitive
Rupesh Mishra
 
Unix memory management
Tech_MX
 
unit-4-dynamic programming
hodcsencet
 
Ad

Similar to Recursive descent parsing (20)

KEY
Let's build a parser!
Boy Baukema
 
PPT
Programming_Language_Syntax.ppt
Amrita Sharma
 
ZIP
Round PEG, Round Hole - Parsing Functionally
Sean Cribbs
 
PDF
Parsing
oscon2007
 
PDF
Perly Parsing with Regexp::Grammars
Workhorse Computing
 
PDF
When RegEx is not enough
Nati Cohen
 
PDF
Bioinformatica: Esercizi su Perl, espressioni regolari e altre amenità (BMR G...
Andrea Telatin
 
PPTX
P3 2018 python_regexes
Prof. Wim Van Criekinge
 
PDF
Certified bit coded regular expression parsing
rodrigogribeiro
 
PDF
Left factor put
siet_pradeep18
 
PPTX
Extracting Archival-Quality Information from Software-Related Chats
Preetha Chatterjee
 
PDF
Stop overusing regular expressions!
Franklin Chen
 
PDF
Parsing Expression Grammars and Treetop
Juan Lulkin
 
PDF
How to create a programming language
Robert Mamore
 
PPTX
Lexing and parsing
Elizabeth Smith
 
PDF
SWP - A Generic Language Parser
kamaelian
 
PDF
(1) cpp introducing the_cpp_programming_language
Nico Ludwig
 
PPTX
Bioinformatics p2-p3-perl-regexes v2013-wim_vancriekinge
Prof. Wim Van Criekinge
 
PPT
Cd2 [autosaved]
BBDITM LUCKNOW
 
PPT
Parsing
khush_boo31
 
Let's build a parser!
Boy Baukema
 
Programming_Language_Syntax.ppt
Amrita Sharma
 
Round PEG, Round Hole - Parsing Functionally
Sean Cribbs
 
Parsing
oscon2007
 
Perly Parsing with Regexp::Grammars
Workhorse Computing
 
When RegEx is not enough
Nati Cohen
 
Bioinformatica: Esercizi su Perl, espressioni regolari e altre amenità (BMR G...
Andrea Telatin
 
P3 2018 python_regexes
Prof. Wim Van Criekinge
 
Certified bit coded regular expression parsing
rodrigogribeiro
 
Left factor put
siet_pradeep18
 
Extracting Archival-Quality Information from Software-Related Chats
Preetha Chatterjee
 
Stop overusing regular expressions!
Franklin Chen
 
Parsing Expression Grammars and Treetop
Juan Lulkin
 
How to create a programming language
Robert Mamore
 
Lexing and parsing
Elizabeth Smith
 
SWP - A Generic Language Parser
kamaelian
 
(1) cpp introducing the_cpp_programming_language
Nico Ludwig
 
Bioinformatics p2-p3-perl-regexes v2013-wim_vancriekinge
Prof. Wim Van Criekinge
 
Cd2 [autosaved]
BBDITM LUCKNOW
 
Parsing
khush_boo31
 
Ad

More from Boy Baukema (12)

PPTX
Security horrors
Boy Baukema
 
PPTX
Tampering with JavaScript
Boy Baukema
 
PDF
Code by the sea: Web Application Security
Boy Baukema
 
PDF
Ibuildings ISO 27001 lunchbox
Boy Baukema
 
PDF
OWASP ASVS 3 - What's new for level 1?
Boy Baukema
 
PDF
Verifying Drupal modules with OWASP ASVS 2014
Boy Baukema
 
PDF
Secure Drupal, from start to finish
Boy Baukema
 
PDF
Security as a part of quality assurance
Boy Baukema
 
PDF
Dpc14 security as part of Quality Assurance
Boy Baukema
 
PDF
SURFconext and Mobile
Boy Baukema
 
PDF
WebAppSec @ Ibuildings in 2014
Boy Baukema
 
PDF
Javascript: 8 Reasons Every PHP Developer Should Love It
Boy Baukema
 
Security horrors
Boy Baukema
 
Tampering with JavaScript
Boy Baukema
 
Code by the sea: Web Application Security
Boy Baukema
 
Ibuildings ISO 27001 lunchbox
Boy Baukema
 
OWASP ASVS 3 - What's new for level 1?
Boy Baukema
 
Verifying Drupal modules with OWASP ASVS 2014
Boy Baukema
 
Secure Drupal, from start to finish
Boy Baukema
 
Security as a part of quality assurance
Boy Baukema
 
Dpc14 security as part of Quality Assurance
Boy Baukema
 
SURFconext and Mobile
Boy Baukema
 
WebAppSec @ Ibuildings in 2014
Boy Baukema
 
Javascript: 8 Reasons Every PHP Developer Should Love It
Boy Baukema
 

Recently uploaded (20)

PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PPTX
sap open course for s4hana steps from ECC to s4
sreeni2106invites
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PPTX
Spectroscopy.pptx food analysis technology
nawahassan45
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PDF
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
PDF
KodekX | Application Modernization Development
KodekX
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
sap open course for s4hana steps from ECC to s4
sreeni2106invites
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
Spectroscopy.pptx food analysis technology
nawahassan45
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
KodekX | Application Modernization Development
KodekX
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 

Recursive descent parsing

  • 1. Recursive Descent Parsing In practice with PHP
  • 2. Plan for the next 40 mins 1. Walk through creating a Parsing Expression Grammar and scannerless predictive recursive descent parser for a subset of print_r output. 2. Talk about why anyone would want to do such a thing. Source Code: https://bit.ly/dpc14rdp
  • 4. I am Boy Baukema Senior Software Engineer @ ibuildings.nl
  • 5. print_r (PHP 4, PHP 5) print_r — Prints human-readable information about a variable
  • 6. An example Array ( [Talk] => Array ( [Title] => Ansible: Orchestrate [Type] => 3
  • 8. No escaping > print_r(array("a"=>"n [b] => evil")); Array ( [a] => [b] => evil )
  • 10. –Martin Fowler “…it’s a technique that isn't as widely known as it should be. Many people are under the impression that using it is quite hard. I think that this fear often comes from the fact that Syntax- Directed Translation is usually described in the context of parsing a general-purpose language—which introduces a lot of complexities that you don't face with a DSL.”
  • 11. V1 - An empty array Source Code: https://bit.ly/dpc14rdp
  • 14. ARRAY_START <- ‘Array’ LF <- “n” PAREN_OPEN <- ‘(’ PAREN_CLOSE <- ‘)’
  • 15. PrintRLang V1 RecursiveDescentParser - $content : string + __construct ( string $content ) + consume ( string $terminal ) + lookAhead ( string $terminal ) Source Code: https://bit.ly/dpc14rdp
  • 16. PrintRLang V1 
 ArrayParser - $parser : RecursiveDescentParser + __construct(RecursiveDescentParser $parser) + parse(): array + arrayStart() + lf() + braceOpen() + braceClose() Source Code: https://bit.ly/dpc14rdp
  • 17. ! $parser = new PrintRLang ArrayParser( new PrintRLang RecursiveDescentParser( "Arrayn(n)n" ) ); $parser->parse();
  • 18. public function parse() { $this->arrayStart(); $this->lf(); $this->braceOpen(); $this->lf() $this->braceClose(); $this->lf(); return array(); }
  • 19. A r r a y n ( n ) n public function arrayStart() { $this->parser->consume('Array'); } n ( n ) n
  • 20. n ( n ) n public function lf() { $this->parser->consume("n"); } ( n ) n
  • 21. ( n ) n public function braceOpen() { $this->parser->consume('('); } n ) n
  • 22. n ) n public function lf() { $this->parser->consume("n"); } ) n
  • 23. ) n public function braceClose() { $this->parser->consume(')'); } n
  • 24. n public function lf() { $this->parser->consume("n"); }
  • 25. V2 - Array of strings Source Code: https://bit.ly/dpc14rdp
  • 28. Kleene star translates to: ARRAY_ASSIGN* while (lookAhead(' ')) $result = arrayAssign($result)
  • 30. Kleene plus SPACE+ === SPACE SPACE*
  • 31. Kleene plus implemented space() while (lookAhead(' ')) space()
  • 34. PrintRLang V2 RecursiveDescentParser - $content : string + __construct ( string $content ) + consume ( string $terminal ) + consumeRegex( string $regex ) + lookAhead ( string $terminal ) + lookAheadRegex( string $regex Source Code: https://bit.ly/dpc14rdp
  • 35. PrintRLang V2 
 ArrayParser - $parser : RecursiveDescentParser ... + arrayAssign( array $result ) + arrayKey() : string + arrayValue() : string + space() + fatArrow ...
  • 36. V3 - Array of Arrays
  • 37. Array ( [Talk] => Array ( [Title] => Ansible: Orchestrate [Type] => 3 ) )
  • 38. ARRAY_VALUE <- ARRAY / STRING STRING <- !LF
  • 40. PrintRLang V3 
 ArrayParser - $parser : RecursiveDescentParser ... + string() ...
  • 41. Why?
  • 42. – Steve Yegge, Rich Programmer Food “If you don't know how parsing works, you'll do it badly with regular expressions, or if you don't know those, then with hand-rolled state machines that are thousands of lines of incomprehensible code that doesn't actually work.”
  • 43. Mail::RFC822::Address (?:(?:rn)?[ t])*(?:(?:(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t] )+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?: rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:( ?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*))*@(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-0 31]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)* ](?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+ (?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?: (?:rn)?[ t])*))*|(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z |(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn) ?[ t])*)*<(?:(?:rn)?[ t])*(?:@(?:[^()<>@,;:".[] 000-031]+(?:(?:(?: rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[
  • 44. t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn) ?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t] )*))*(?:,@(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])* )(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t] )+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*))*) *:(?:(?:rn)?[ t])*)?(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+ |Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:r n)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?: rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t ]))*"(?:(?:rn)?[ t])*))*@(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031 ]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*]( ?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(? :(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(? :rn)?[ t])*))*>(?:(?:rn)?[ t])*)|(?:[^()<>@,;:".[] 000-031]+(?:(? :(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)? [ t]))*"(?:(?:rn)?[ t])*)*:(?:(?:rn)?[ t])*(?:(?:(?:[^()<>@,;:".[]
  • 45. 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]| .|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<> @,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|" (?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*))*@(?:(?:rn)?[ t] )*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;: ".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(? :[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[ ]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*))*|(?:[^()<>@,;:".[] 000- 031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|( ?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*)*<(?:(?:rn)?[ t])*(?:@(?:[^()<>@,; :".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([ ^[]r]|.)*](?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:" .[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[ ]r]|.)*](?:(?:rn)?[ t])*))*(?:,@(?:(?:rn)?[ t])*(?:[^()<>@,;:". [] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[] r]|.)*](?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[]
  • 46. 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r] |.)*](?:(?:rn)?[ t])*))*)*:(?:(?:rn)?[ t])*)?(?:[^()<>@,;:".[] 0 00-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]| .|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[^()<>@, ;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[]]))|"(? :[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*))*@(?:(?:rn)?[ t])* (?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:". []]))|[([^[]r]|.)*](?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t])*(?:[ ^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[] ]))|[([^[]r]|.)*](?:(?:rn)?[ t])*))*>(?:(?:rn)?[ t])*)(?:,s*( ?:(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;: ".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*)(?:.(?:( ?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[ ["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t ])*))*@(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t ])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*)(? :.(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|
  • 47. Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*))*|(?: [^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:".[ ]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*)*<(?:(?:rn) ?[ t])*(?:@(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[[" ()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*)(?:.(?:(?:rn) ?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<> @,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*))*(?:,@(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@, ;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*)(?:.(?:(?:rn)?[ t] )*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;: ".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*))*)*:(?:(?:rn)?[ t])*)? (?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[["()<>@,;:". []]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t])*)(?:.(?:(?: rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z|(?=[[ "()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[ t]))*"(?:(?:rn)?[ t]) *))*@(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t]) +|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*)(?: .(?:(?:rn)?[ t])*(?:[^()<>@,;:".[] 000-031]+(?:(?:(?:rn)?[ t])+|Z |(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[ t])*))*>(?:( ?:rn)?[ t])*))*)?;s*)
  • 48. Useful applications I’ve seen REST API with CQL querying (MediaMosa.org) Migrating wiki content Parsing log files Parsing obscure specifications (ARF) Concise configuration files Domain Specific Languages
  • 49. –Martin Fowler, Domain Specific Languages “a DSL is a front-end to a library providing a different style of manipulation to the command-query API. ”
  • 50. Rules for building a parser Consider using an existing parser. Consider porting one from another language. Consider XML or the new XMLs: JSON / YAML Consider working around it. Then and only then consider building your own parser
  • 52. Let’s build a parser! http://protalk.me/dpcradio-lets-build-a-parser
  • 53. Thank you for your time and attention! Questions? Tweet to @relaxnow Rate @ https://joind.in/10859 Slides @ https://joind.in/10859 Code @ https://bit.ly/dpc14rdp