Parser combinators

Parser Combinators
Writing external DSLs

Wednesday, November 9, 2011

DSLs

• DSL is an ultimate abstraction
• You create a language that precisely
describes the problem you are trying to
solve.


Internal & External

• DSL can be internal - made using the host
programming language.
• Or external - you invent a little language
for solving your problem.


Internal DSL: Good
• No external tool dependencies
• Nice IDE support
• Type checking for free
• Easy to implement
• Gives full power of your language
• Parsing is free

Internal DSL: Bad

• It requires expressive language
• If the language is not expressive enough
DSL will be verbose and not pleasant to
use
• It can produce obscure error messages


External DSL: Good

• Platform independent
• Nicer syntax
• External tool can perform static analysis
• Good error messages


External DSLs: Bad

• No IDE support
• Hard to implement


External DSLs

• External DSLs can be implemented using
interpretation or code generation.
• Parse
• Analyze
• Interpret | Generate Code


parseJSON :: String -> Either Error JSONObject


JSON Grammar-like stuff
object := '{' fields? '}'
fields := field (',' field)*
field := id ':' value
value := literal | object | array


Bison (C)
ANTLR (Java and many others)
You can encode grammar by hand


function parseObject() {
match('{')
if (lookahead(ID))
parseFields()
match('}')
}


function parseFields() {
parseField()
while (lookahead(',')) {
consume()
parseField()
}
}


• Parser can either consume some input or
not.

• Parser can fail or succeed.
• Parser can return some value.


• Usually sequences are just sequential calls.
• Optional parts of grammar are just IFs
• Zero or more elements is WHILE
• Grammar rule is a function


function parseFields() {
separatedBy(',', parseField)
}

function separatedBy(ch, f) {
f()
while (lookahead(ch)) {
consume()
f()
}
}


function sepBy(ch, p) {
return
seq(p, many(seq(ch, p)))
}


Combinator pattern


Haskell in 60 seconds


main = putStrLn "Hello World!"

times x n = x * n
times = x n -> x * n
times = x -> n -> x * n
times = (*)
times 2 3
2 `times` 3
data Point = Point Int Int
data Either a b = Left a | Right b
Point 10 20
Left “Hello”


import Text.Parsec

parse ::
Parser
-> SourceName
-> String
-> Either ParseError a


p = satisfy (x -> x == 'a')
main = print (parse p "<test>" "abc")

> Right 'a'

p = satisfy (x -> x == 'a')
main = print (parse p "<test>" "bcd")

> Left "<test>" (line 1, column 1):
> unexpected "b"


-- There is something like this in Parsec
char c = satisfy (x -> x == c)

-- Example
p = char 'a'
main = print (parse p "<test>" "bcd")

> unexpected "b"
> expecting "a"


char c = satisfy (==c) <?> show [c]


p = char 'a' >>= x ->
char 'b' >>= y ->
return [x, y]

> Right "ab"


p = do char 'a'
char 'b'
return "ab"

> Right "ab"


p = do char 'a'
char 'b' <|> char 'q'
char 'c' <|> char 'd'



letter = satisfy isAlpha <?> "letter"
digit = satisfy isDigit <?> "digit"
spaces = skipMany space <?> "white space"

many p = ... censored ...


p = do letter
many (letter <|> digit)
main = print (parse p "<test>" "hello123")

> Right "ello123"


p = do x <- letter
xs <- many (letter <|> digit)
return (x:xs)
main = print (parse p "<test>" "hello123")

> Right "hello123"


ident = do x <- letter
return (x:xs)
p = ident
main = print (parse p "<test>" "123hello")

> unexpected "1"
> expecting letter


return (x:xs)
p = ident <?> "variable name"

> unexpected "1"
> expecting variable name


return (x:xs)
<?> "variable name"
p = ident


return (x:xs)
<?> "variable name"
letKeyword = string "let"
p = ident <|> letKeyword
main = print (parse p "<test>" "letter")

> Right "letter"


return (x:xs)
<?> "variable name"
letKeyword = do s <- string "let"
notFollowedBy (letter <|> digit)
spaces
return s
p = try(letKeyword) <|> ident
main = print (parse p "<test>" "letter")
-----
Right "letter"


Lexer-like example


type Name = String
data Expr = Number Integer
| Var Name
| Let Name Expr
| Seq [Expr]
deriving Show


myDef = emptyDef {
identStart = letter <|> char '_',
identLetter = alphaNum <|> char '_',
opStart = opLetter myDef,
opLetter = oneOf "=,;",
reservedOpNames = [
",", ";", "="
],
reservedNames = ["let"],
caseSensitive = True
}


tokenParser = PT.makeTokenParser myDef

parens = PT.parens tokenParser
naturalNumber = PT.natural tokenParser
keyword = PT.reserved tokenParser
identifier = PT.identifier tokenParser
op = PT.reservedOp tokenParser
commaSep = PT.commaSep tokenParser
commaSep1 = PT.commaSep1 tokenParser


simple = numberLiteral <|> var

numberLiteral = do n <- naturalNumber
return $ Number n

var = do s <- identifier
return $ Var s


letStmt = do keyword "let"
defs <- commaSep1 def
op ";"
return $ Seq defs

def = do name <- identifier
op "="
value <- simple
return $ Let name value


main = print (
parse letStmt "<test>" "let one = 1, two = 2;"
)

----

Right (
Seq [
Let "one" (Number 1),
Let "two" (Number 2)
]
)


CSV Parser Example


import Text.Parsec

csvFile = line `endBy1` eol
line = cell `sepBy` (char ',')
cell = wsopt >> many (noneOf ",n")
wsopt = many (oneOf " t")
eol = char 'n'
main = print (parse csvFile "<test>" input)
where input = "aa, b,cnd,e,fn"

> Right [["aa","b","c"],["d","e","f"]]


import scala.util.parsing.combinator._

object CSVParser extends RegexParsers {
override def skipWhitespace = false
def whitespace = """[ t]*"""r
def csv = rep1sep(row, "n") <~ "n"
def row = rep1sep(field, ",")
def field = opt(whitespace) ~> """[^n,]""".r
def parse(s: String) = parseAll(csv, s)
}
println(CSVParser.parse("a,b,cnd,e, fn"))

> [3.1] parsed: List(List(a, b, c), List(d, e, f))


Pysec for Python
Spirit for C++
In standard library of Scala
A lot of others...


Stuff to read

• Language Implementation Patterns:
Create Your Own Domain-Speciﬁc and
General Programming Languages
by Terence Parr


Stuff to read

• Domain Speciﬁc Languages
by Martin Fowler


Stuff to read

• http://learnyouahaskell.com/
Learn You a Haskell for a Great Good


The End


Parser combinators

More Related Content

Similar to Parser combinators (20)

Recently uploaded (20)

Parser combinators