Instaduction to instaparse
0 likes194 views
Instaparse is a Clojure library for building parsers from context-free grammars, enabling users to convert strings into parse trees based on defined grammars. It discusses the structure of context-free grammars, including non-terminals, terminals, productions, and the concept of running grammars forwards and backwards. The document emphasizes Instaparse's capabilities in processing and visualizing parse trees, making it effective for parsing programming languages, configuration files, and various data formats.
![Hello, instaparse
instaparse-talk.core=> (require '[instaparse.core :as insta])
nil
instaparse-talk.core=> (def as-and-bs
#_=> (insta/parser
#_=> "S = AB*
#_=> AB = A B
#_=> A = 'a'+
#_=> B = 'b'+"))
#'instaparse-talk.core/as-and-bs](https://image.slidesharecdn.com/instaductiontoinstaparse-190314180915/85/Instaduction-to-instaparse-37-320.jpg)
![Hello, instaparse
instaparse-talk.core=> (as-and-bs "aaabbbaabb")
[:S [:AB [:A "a" "a" "a"] [:B "b" "b" "b"]] [:AB [:A "a" "a"] [:B "b" "b"]]]
instaparse-talk.core=> (pprint *1)
[:S
[:AB [:A "a" "a" "a"] [:B "b" "b" "b"]]
[:AB [:A "a" "a"] [:B "b" "b"]]]
nil
instaparse-talk.core=> (insta/visualize (as-and-bs "aaabbbaabb"))
nil](https://image.slidesharecdn.com/instaductiontoinstaparse-190314180915/85/Instaduction-to-instaparse-38-320.jpg)
![Example: replacing a node with a zipper
instaparse-talk.core=> (-> (zip/vector-zip (as-and-bs "aaaabbbbaabbabbb"))
pprint)
[[:S
[:AB [:A "a" "a" "a" "a"] [:B "b" "b" "b" "b"]]
[:AB [:A "a" "a"] [:B "b" "b"]]
[:AB [:A "a"] [:B "b" "b" "b"]]]
nil]
nil](https://image.slidesharecdn.com/instaductiontoinstaparse-190314180915/85/Instaduction-to-instaparse-41-320.jpg)
Instaduction to instaparse
- 1. Instaduction to Instaparse CAP-CLUG - 2019-03-13
- 2. Instaparse is a clojure library for building parsers from context-free grammars
- 3. What is a parser? ● Program that takes some input data (usually a string), and produces a data-structure (usually a parse tree), based on some grammar (usually a context-free grammar)
- 4. What’s a context-free grammar? Formal definition: V = finite set of non-terminals or variables. Each variable represents a clause or a phrase, or a syntactic category 𝚺 = finite set of terminals. The set of terminals is the alphabet of the language R = finite relation from V to (V U 𝚺)*. Each member of R is a rewrite rule or production S = the starting symbol, must be an element of V Adapted from https://en.wikipedia.org/wiki/Context-free_grammar#Formal_definitions
- 5. What’s a context-free grammar? The “context-free” bit means that the rules can always be applied, regardless of the rest of the string (context). There are other kinds of grammars, some more or less powerful than CFG’s. See Chomsky Hierarchy for more
- 6. A simple CFG S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 7. A simple CFG S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+ Non-terminals Non-terminals
- 8. A simple CFG S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+ Terminals
- 9. A simple CFG S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+ Productions
- 10. A simple CFG S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+ Starting symbol
- 11. Productions Each production is a rule. You can replace the symbol on the left with symbol(s) on the right. ‘+’ means “one or more”; ‘*’ means “zero or more” Non-terminals can be recursively defined, and appear on left- and right-side of rules Terminals only appear on the right side of a rule If you imagine a tree, non-terminals are interior nodes, terminals are leaf nodes (we’ll see this more later) S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 12. Productions example S S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 13. Productions example AB S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 14. Productions example A B S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 15. Productions example aaaaaa B S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 16. Productions example aaaaaabb S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 17. Some other strings that this grammar can generate aaabbbbababaaaabbbb abbbbbbbbb abababababababab aabbaabbbbbaaaaaab S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 18. Running a grammar “forwards” generates strings that conform to a grammar
- 19. Running a grammar “backwards” over a string tells us if that string is valid, according to the grammar
- 20. Running the CFG backwards aaaaaabbaab S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 21. Running the CFG backwards aaaaaabbaab S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 22. Running the CFG backwards A bbaab S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 23. Running the CFG backwards A B A B S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 24. Running the CFG backwards AB AB S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 25. Running the CFG backwards S S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 26. Running the CFG backwards S S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+ VALID!
- 27. Let’s see that again, but not overwrite the string
- 28. Running the CFG backwards aaaaaabbaab S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 29. Running the CFG backwards aaaaaa bb aa b S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+A AB B
- 30. Running the CFG backwards aaaaaa bb aa b S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+A AB B AB AB
- 31. Running the CFG backwards aaaaaa bb aa b S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+A AB B AB AB S
- 32. Running the CFG backwards aaaaaa bb aa b S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+A AB B AB AB S Parse Tree
- 33. What is a parser? ● Program that takes some input data (usually a string), and produces a data-structure (usually a parse tree), based on some grammar (usually a context-free grammar)
- 34. Instaparse is a clojure library for building parsers from context-free grammars
- 35. A grammar that recognizes runs of a’s and b’s S ::= AB* AB ::= A B A ::= ‘a’+ B ::= ‘b’+
- 36. ABNF notation for instaparse
- 37. Hello, instaparse instaparse-talk.core=> (require '[instaparse.core :as insta]) nil instaparse-talk.core=> (def as-and-bs #_=> (insta/parser #_=> "S = AB* #_=> AB = A B #_=> A = 'a'+ #_=> B = 'b'+")) #'instaparse-talk.core/as-and-bs
- 38. Hello, instaparse instaparse-talk.core=> (as-and-bs "aaabbbaabb") [:S [:AB [:A "a" "a" "a"] [:B "b" "b" "b"]] [:AB [:A "a" "a"] [:B "b" "b"]]] instaparse-talk.core=> (pprint *1) [:S [:AB [:A "a" "a" "a"] [:B "b" "b" "b"]] [:AB [:A "a" "a"] [:B "b" "b"]]] nil instaparse-talk.core=> (insta/visualize (as-and-bs "aaabbbaabb")) nil
- 40. Walking the parse tree Parse trees are just clojure data! We have a TON of great ways to handle them ● Recursive or iterative processing using case or core.match (pattern matching) ● Zippers (functional navigation and “editing” of trees) ● insta/transform ● Seq/tree-seq ● Enlive (CSS-style selectors for clojure data structures) ● Any other way that you want to walk nested vectors in clojure!
- 41. Example: replacing a node with a zipper instaparse-talk.core=> (-> (zip/vector-zip (as-and-bs "aaaabbbbaabbabbb")) pprint) [[:S [:AB [:A "a" "a" "a" "a"] [:B "b" "b" "b" "b"]] [:AB [:A "a" "a"] [:B "b" "b"]] [:AB [:A "a"] [:B "b" "b" "b"]]] nil] nil
- 42. Example: replacing a node with a zipper
- 43. Example: infix to postfix using case statements 1 + 2 * 3 - 4 / 5 1 2 3 * + 4 5 / -
- 44. Example: infix to postfix using case statements
- 46. Example: infix to postfix using case statements
- 47. Example: infix to postfix using case statements
- 48. Example: insta/transform Apply this fn To nodes that match magic!
- 49. Wrapping up ● Parsers turn text into trees ● Clojure is great at walking through trees ● Instaparse makes it easy to parse things ○ Programming languages ○ Config files ○ Data ○ Lots more! The docs for instaparse are amazing. A lot of my examples were lifted straight from it. Read the docs. They’re great. Everyone on the project did a fantastic job https://github.com/Engelberg/instaparse
- 50. Thanks!
- 51. A cool way to visualize a CFG is with a railroad diagram