SlideShare a Scribd company logo

Climbing the Abstract Syntax Tree (IPC Fall 2017)

James Titcumb, profile picture
James Titcumb

The document discusses the abstract syntax tree (AST) representation of PHP code. It describes how the PHP lexer and parser work together to convert PHP source code into an AST. Key points covered include the PHP lexer zend_language_scanner.l, the PHP parser zend_language_parser.y, and how an if statement is parsed and represented in the AST.

Technology
1 of 123
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
@asgrim Climbing the Abstract Syntax Tree James Titcumb International PHP Conference 2017
$ whoami James Titcumb www.jamestitcumb.com www.roave.com @asgrim
@asgrim How PHP works PHP code OpCache Execute (VM) Lexer + Parser Compiler
@asgrim The PHP Lexer zend_language_scanner.l
@asgrim zend_language_scanner.l <ST_IN_SCRIPTING>"exit" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"die" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"function" { RETURN_TOKEN(T_FUNCTION); }
@asgrim zend_language_scanner.l <ST_IN_SCRIPTING>"exit" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"die" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"function" { RETURN_TOKEN(T_FUNCTION); }
@asgrim zend_language_scanner.l <ST_IN_SCRIPTING>"exit" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"die" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"function" { RETURN_TOKEN(T_FUNCTION); }
@asgrim zend_language_scanner.l <ST_IN_SCRIPTING>"exit" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"die" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"function" { RETURN_TOKEN(T_FUNCTION); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim The PHP Lexer zend_language_scanner.l
@asgrim The PHP Lexer zend_language_scanner.l re2c
@asgrim The PHP Lexer zend_language_scanner.l re2c zend_language_scanner.c
@asgrim The PHP Parser zend_language_parser.y
@asgrim zend_language_parser.y if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ;
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
@asgrim if ($a == 1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse
@asgrim if ($a == 1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse if_stmt_without_else (A)
@asgrim if ($a == 1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse if_stmt_without_else (A) if_stmt_without_else (B)
@asgrim if ($a == 1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse if_stmt_without_else (A) if_stmt_without_else (B) if_stmt
@asgrim Zend_language_parser.y (PHP 7.0.10) if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ;
@asgrim zend_language_parser.y (PHP 5.6.26) T_IF parenthesis_expr { zend_do_if_cond(&$2, &$1 TSRMLS_CC); } statement { zend_do_if_after_statement(&$1, 1 TSRMLS_CC); } void zend_do_if_cond(const znode *cond, znode *closing_bracket_token TSRMLS_DC) { int if_cond_op_number = get_next_op_number(CG(active_op_array)); zend_op *opline = get_next_op(CG(active_op_array) TSRMLS_CC); opline->opcode = ZEND_JMPZ; SET_NODE(opline->op1, cond); closing_bracket_token->u.op.opline_num = if_cond_op_number; SET_UNUSED(opline->op2); INC_BPC(CG(active_op_array)); }
@asgrim AST is new in PHP 7+
@asgrim How PHP works PHP code OpCache Execute (VM) Lexer + Parser Compiler
@asgrim Let’s simplify!
@asgrim First… WTF is AST?
@asgrim AST is just a data structure
@asgrim PHP code <?php echo "Hello world";
@asgrim An AST representation Echo statement `-- String, value "Hello world"
@asgrim PHP code <?php echo "Hello " . "world";
@asgrim An AST representation Echo statement `-- Concat |-- Left | `-- String, value "Hello " `-- Right `-- String, value "world"
@asgrim PHP code <?php $a = 5; $b = 3; echo $a + ($b * 2);
@asgrim An AST representation Assign statement |-- Variable $a `-- Integer, value 5 Assign statement |-- Variable $b `-- Integer, value 3 Echo statement `-- Add operation |-- Left | `-- Variable $a `-- Right `-- Multiply operation |-- Left | `-- Variable $b `-- Right `-- Integer, value 2
@asgrim Why?
@asgrim Faster!*
@asgrim AST compilation Statements EchoAssign Scalar value: (int)5 Variable name: $a Assign Scalar value: (int)3 Variable name: $b Add op Right operandLeft operand Variable name: $a Multiply op Right operandLeft operand Variable name: $b Scalar value: (int)2
@asgrim AST compilation: pre-order traversal Statements EchoAssign Scalar value: (int)5 Variable name: $a Assign Scalar value: (int)3 Variable name: $b Add op Right operandLeft operand Variable name: $a Multiply op Right operandLeft operand Variable name: $b Scalar value: (int)2
@asgrim Pre-order traversal: Polish notation Assign(Variable $a, Scalar 5) Assign(Variable $b, Scalar 3) Echo ( Add( Variable $a, Multiply( $b, 2 ) ) )
@asgrim Order of precedence 1 + 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9?
@asgrim Order of precedence 1 + 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9? + 1 * 2 3
@asgrim Order of precedence 1 + 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9? + 1 * 2 3 Operator Left operand Right operand
@asgrim Order of precedence 1 + 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9? + 1 * 2 3 Operator Left operand Right operand Operator Left operand Right operand
@asgrim Reverse Polish Notation 1 2 3 * +
@asgrim Reverse Polish Notation 1 2 3 * + The stack
@asgrim Reverse Polish Notation 1 2 3 * + The stack 1
@asgrim Reverse Polish Notation 1 2 3 * + The stack 1 2
@asgrim Reverse Polish Notation 1 2 3 * + The stack 1 2 3
@asgrim Reverse Polish Notation 1 2 3 * + The stack 1 2 3
@asgrim Reverse Polish Notation 1 2 3 * + The stack 1 2 3
@asgrim Reverse Polish Notation 1 2 3 * + The stack 1 6
@asgrim Reverse Polish Notation 1 2 3 * + The stack 1 6
@asgrim Reverse Polish Notation 1 2 3 * + The stack 7
@asgrim Let’s write a compiler (!!!) In three easy steps…
@asgrim Warning: do not use in production
@asgrim View > Source https://github.com/asgrim/basic-maths-compiler
@asgrim Define the language Tokens ● T_ADD (+) ● T_SUBTRACT (-) ● T_MULTIPLY (/) ● T_DIVIDE (*) ● T_INTEGER (d) ● T_WHITESPACE (s+)
@asgrim Step 1: Writing a simple lexer
@asgrim Using regular expressions private static $matches = [ '/^(+)/' => Token::T_ADD, '/^(-)/' => Token::T_SUBTRACT, '/^(*)/' => Token::T_MULTIPLY, '/^(/)/' => Token::T_DIVIDE, '/^(d+)/' => Token::T_INTEGER, '/^(s+)/' => Token::T_WHITESPACE, ];
@asgrim Step through the input string public function __invoke(string $input) : array { $tokens = []; $offset = 0; while ($offset < strlen($input)) { $focus = substr($input, $offset); $result = $this->match($focus); $tokens[] = $result; $offset += strlen($result->getLexeme()); } return $tokens; }
@asgrim The matching method private function match(string $input) : Token { foreach (self::$matches as $pattern => $token) { if (preg_match($pattern, $input, $matches)) { return new Token($token, $matches[1]); } } throw new RuntimeException(sprintf( 'Unmatched token, next 15 chars were: %s', substr($input, 0, 15) )); }
@asgrim Step 2: Parsing the tokens
@asgrim Order tokens by operator precedence /** * Higher number is higher precedence. * @var int[] */ private static $operatorPrecedence = [ Token::T_SUBTRACT => 0, Token::T_ADD => 1, Token::T_DIVIDE => 2, Token::T_MULTIPLY => 3, ];
@asgrim Order tokens by operator precedence /** @var Token[] $stack */ $stack = []; /** @var Token[] $operators */ $operators = []; while (false !== ($token = current($tokens))) { if ($token->isOperator()) { // ... } $stack[] = $token; next($tokens); }
@asgrim Order tokens by operator precedence /** @var Token[] $stack */ $stack = []; /** @var Token[] $operators */ $operators = []; while (false !== ($token = current($tokens))) { if ($token->isOperator()) { // ... } $stack[] = $token; next($tokens); }
@asgrim Order tokens by operator precedence /** @var Token[] $stack */ $stack = []; /** @var Token[] $operators */ $operators = []; while (false !== ($token = current($tokens))) { if ($token->isOperator()) { // ... } $stack[] = $token; next($tokens); }
@asgrim Order tokens by operator precedence /** @var Token[] $stack */ $stack = []; /** @var Token[] $operators */ $operators = []; while (false !== ($token = current($tokens))) { if ($token->isOperator()) { // ... } $stack[] = $token; next($tokens); }
@asgrim Order tokens by operator precedence if ($token->isOperator()) { $tokenPrecedence = self::$operatorPrecedence[$token->getToken()]; while ( count($operators) && self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()] > $tokenPrecedence ) { $higherOp = array_pop($operators); $stack[] = $higherOp; } $operators[] = $token; next($tokens); continue; }
@asgrim Order tokens by operator precedence if ($token->isOperator()) { $tokenPrecedence = self::$operatorPrecedence[$token->getToken()]; while ( count($operators) && self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()] > $tokenPrecedence ) { $higherOp = array_pop($operators); $stack[] = $higherOp; } $operators[] = $token; next($tokens); continue; }
@asgrim Order tokens by operator precedence if ($token->isOperator()) { $tokenPrecedence = self::$operatorPrecedence[$token->getToken()]; while ( count($operators) && self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()] > $tokenPrecedence ) { $higherOp = array_pop($operators); $stack[] = $higherOp; } $operators[] = $token; next($tokens); continue; }
@asgrim Order tokens by operator precedence if ($token->isOperator()) { $tokenPrecedence = self::$operatorPrecedence[$token->getToken()]; while ( count($operators) && self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()] > $tokenPrecedence ) { $higherOp = array_pop($operators); $stack[] = $higherOp; } $operators[] = $token; next($tokens); continue; }
@asgrim Order tokens by operator precedence // Clean up by moving any remaining operators onto the token stack while (count($operators)) { $stack[] = array_pop($operators); } return $stack;
@asgrim Order tokens by operator precedence 1 + 2 * 3 Output stack Operator stack
@asgrim Order tokens by operator precedence 1 + 2 * 3 1Output stack Operator stack
@asgrim Order tokens by operator precedence 1 + 2 * 3 1 + Output stack Operator stack
@asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 + Output stack Operator stack
@asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 + * Output stack Operator stack
@asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 3 + * Output stack Operator stack
@asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 3 * + * Output stack Operator stack
@asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 3 * + + Output stack Operator stack
@asgrim Create AST while ($ip < count($tokenStack)) { $token = $tokenStack[$ip++]; if ($token->isOperator()) { // (figure out $nodeType) $right = array_pop($astStack); $left = array_pop($astStack); $astStack[] = new $nodeType($left, $right); continue; } $astStack[] = new NodeScalarIntegerValue((int)$token->getLexeme()); }
@asgrim Create AST while ($ip < count($tokenStack)) { $token = $tokenStack[$ip++]; if ($token->isOperator()) { // (figure out $nodeType) $right = array_pop($astStack); $left = array_pop($astStack); $astStack[] = new $nodeType($left, $right); continue; } $astStack[] = new NodeScalarIntegerValue((int)$token->getLexeme()); }
@asgrim Create AST while ($ip < count($tokenStack)) { $token = $tokenStack[$ip++]; if ($token->isOperator()) { // (figure out $nodeType) $right = array_pop($astStack); $left = array_pop($astStack); $astStack[] = new $nodeType($left, $right); continue; } $astStack[] = new NodeScalarIntegerValue((int)$token->getLexeme()); }
@asgrim Create AST while ($ip < count($tokenStack)) { $token = $tokenStack[$ip++]; if ($token->isOperator()) { // (figure out $nodeType) $right = array_pop($astStack); $left = array_pop($astStack); $astStack[] = new $nodeType($left, $right); continue; } $astStack[] = new NodeScalarIntegerValue((int)$token->getLexeme()); }
@asgrim Create AST NodeBinaryOpAdd ( NodeScalarIntegerValue(1), NodeBinaryOpMultiply ( NodeScalarIntegerValue(2), NodeScalarIntegerValue(3) ) )
@asgrim Step 3: Executing the AST
@asgrim Compile & execute AST private function compileNode(NodeInterface $node) { if ($node instanceof NodeBinaryOpAbstractBinaryOp) { return $this->compileBinaryOp($node); } if ($node instanceof NodeScalarIntegerValue) { return $node->getValue(); } }
@asgrim Compile & execute AST private function compileBinaryOp(NodeBinaryOpAbstractBinaryOp $node) { $left = $this->compileNode($node->getLeft()); $right = $this->compileNode($node->getRight()); switch (get_class($node)) { case NodeBinaryOpAdd::class: return $left + $right; case NodeBinaryOpSubtract::class: return $left - $right; case NodeBinaryOpMultiply::class: return $left * $right; case NodeBinaryOpDivide::class: return $left / $right; } }
@asgrim What does this mean for me?
@asgrim AST in userland
@asgrim php-ast extension https://github.com/nikic/php-ast
@asgrim php-ast example usage <?php require 'path/to/util.php'; $code = <<<'EOC' <?php $var = 42; EOC; echo ast_dump(astparse_code($code, $version=35)), "n"; // Output: AST_STMT_LIST 0: AST_ASSIGN var: AST_VAR name: "var" expr: 42
@asgrim astkit https://github.com/sgolemon/astkit
@asgrim astkit example usage $if = AstKit::parseString(<<<EOD if (true) { echo "This is a triumph.n"; } else { echo "The cake is a lie.n"; } EOD ); $if->execute(); // First run, program is as-seen above $const = $if->getChild(0)->getChild(0); // Replace the "true" constant in the condition with false $const->graft(0, false); // Can also graft other AstKit nodes, instead of constants $if->execute(); // Second run now takes the else path
@asgrim PhpParser https://github.com/nikic/PHP-Parser
@asgrim PHP Parser <?php use PhpParserParserFactory; $parser = (new ParserFactory) ->create(ParserFactory::PREFER_PHP7); print_r($parser->parse( file_get_contents('ast-demo-src.php') ));
@asgrim Better Reflection https://github.com/Roave/BetterReflection
@asgrim Better Reflection workflow Reflector Source Locator PhpParser Reflection
@asgrim PHP Reflection $reflection = new ReflectionClass( MyExampleClass::class ); $this->assertSame( 'ExampleClass', $reflection->getShortName() );
@asgrim Better Reflection $reflection = (new BetterReflection()) ->classReflector() ->reflect(MyExampleClass::class); $this->assertSame( 'ExampleClass', $reflection->getShortName() );
@asgrim Class BetterReflection public function sourceLocator() : SourceLocator { $astLocator = $this->astLocator(); return $this->sourceLocator ?? $this->sourceLocator = new MemoizingSourceLocator(new AggregateSourceLocator([ new PhpInternalSourceLocator($astLocator), new EvaledCodeSourceLocator($astLocator), new AutoloadSourceLocator($astLocator), ])); } public function classReflector() : ClassReflector { return $this->classReflector ?? $this->classReflector = new ClassReflector($this->sourceLocator()); }
@asgrim Given a class structure... <?php class Foo { private $bar; public function thing() { } }
@asgrim … we get the AST! Class, name Foo |-- Statements | |-- Property, name bar | | |-- Type [private] | | `-- Attributes [start line: 7, end line: 9] | `-- Method, name thing | |-- Type [public] | |-- Parameters [...] | |-- Statements [...] | `-- Attributes [start line: 7, end line: 9] `-- Attributes [start line: 3, end line: 10]
@asgrim What can I use Better Reflection for?
@asgrim Monkey patching example class MyClass { public function foo() { return 5; } }
@asgrim Monkey patching example use RoaveBetterReflectionReflectorClassReflector; use RoaveBetterReflectionSourceLocatorTypeSingleFileSourceLocator; use RoaveBetterReflectionUtilAutoloadClassLoader; use RoaveBetterReflectionUtilAutoloadClassLoaderMethodFileCacheLoader; $loader = new ClassLoader(FileCacheLoader::defaultFileCacheLoader(__DIR__)); // Create the reflection first (without loading) $classInfo = (new ClassReflector( new SingleFileSourceLocator( __DIR__ . '/MyClass.php', (new BetterReflection())->astLocator() ) ))->reflect('MyClass'); $loader->addClass($classInfo);
@asgrim Monkey patching example use RoaveBetterReflectionReflectorClassReflector; use RoaveBetterReflectionSourceLocatorTypeSingleFileSourceLocator; use RoaveBetterReflectionUtilAutoloadClassLoader; use RoaveBetterReflectionUtilAutoloadClassLoaderMethodFileCacheLoader; $loader = new ClassLoader(FileCacheLoader::defaultFileCacheLoader(__DIR__)); // Create the reflection first (without loading) $classInfo = (new ClassReflector( new SingleFileSourceLocator( __DIR__ . '/MyClass.php', (new BetterReflection())->astLocator() ) ))->reflect('MyClass'); $loader->addClass($classInfo);
@asgrim Monkey patching example // Override the body...! $classInfo->getMethod('foo')->setBodyFromClosure( function () { return 4; } ); $c = new MyClass(); echo $c->foo() . "n";
@asgrim Monkey patching example // Override the body...! $classInfo->getMethod('foo')->setBodyFromClosure( function () { return 4; } ); $c = new MyClass(); echo $c->foo() . "n"; // returns 4, not 5
@asgrim To summarise ● For PHP engine: ○ AST is an efficient data structure to represent code ○ AST means faster compilation (ignoring opcache) ○ Separation in PHP engine for parser and compiler ○ https://wiki.php.net/rfc/abstract_syntax_tree ● Concepts can be used in userland ○ PHP Parser library - https://github.com/nikic/php-parser ○ Better Reflection - https://github.com/Roave/BetterReflection ■ Reflect on not-yet-loaded files ■ Monkey patching in userland code (!) ○ Static analysis opportunities ■ Better Reflection ■ Exakat static analysis (uses own AST) ■ Phan (uses php-ast extension)
Any questions? ... James Titcumb @asgrim

More Related Content

PDF
Climbing the Abstract Syntax Tree (DPC 2017)
James Titcumb
 
PDF
Climbing the Abstract Syntax Tree (CodeiD PHP Odessa 2017)
James Titcumb
 
PDF
Climbing the Abstract Syntax Tree (PHP Developer Days Dresden 2018)
James Titcumb
 
PDF
Climbing the Abstract Syntax Tree (ScotlandPHP 2018)
James Titcumb
 
PDF
Climbing the Abstract Syntax Tree (Bulgaria PHP 2016)
James Titcumb
 
PDF
Climbing the Abstract Syntax Tree (Forum PHP 2017)
James Titcumb
 
PDF
Climbing the Abstract Syntax Tree (PHP UK 2018)
James Titcumb
 
PDF
Climbing the Abstract Syntax Tree (PHP Russia 2019)
James Titcumb
 
Climbing the Abstract Syntax Tree (DPC 2017)
James Titcumb
 
Climbing the Abstract Syntax Tree (CodeiD PHP Odessa 2017)
James Titcumb
 
Climbing the Abstract Syntax Tree (PHP Developer Days Dresden 2018)
James Titcumb
 
Climbing the Abstract Syntax Tree (ScotlandPHP 2018)
James Titcumb
 
Climbing the Abstract Syntax Tree (Bulgaria PHP 2016)
James Titcumb
 
Climbing the Abstract Syntax Tree (Forum PHP 2017)
James Titcumb
 
Climbing the Abstract Syntax Tree (PHP UK 2018)
James Titcumb
 
Climbing the Abstract Syntax Tree (PHP Russia 2019)
James Titcumb
 

What's hot (20)

PDF
Climbing the Abstract Syntax Tree (Southeast PHP 2018)
James Titcumb
 
PDF
Climbing the Abstract Syntax Tree (Midwest PHP 2020)
James Titcumb
 
PDF
Climbing the Abstract Syntax Tree (php[world] 2019)
James Titcumb
 
PDF
Climbing the Abstract Syntax Tree (phpDay 2017)
James Titcumb
 
PDF
Interpret this... (PHPem 2016)
James Titcumb
 
PPT
Class 4 - PHP Arrays
Ahmed Swilam
 
PDF
Sorting arrays in PHP
Vineet Kumar Saini
 
PDF
Creating a compiler in Perl 6
Andrew Shitov
 
PDF
What's New in Perl? v5.10 - v5.16
Ricardo Signes
 
PDF
OSDC.TW - Gutscript for PHP haters
Lin Yo-An
 
PDF
Arrays in PHP
Vineet Kumar Saini
 
PDF
Dades i operadors
Alex Muntada Duran
 
PPTX
Php 2
vivlinux
 
PPT
Php Using Arrays
mussawir20
 
PDF
Php array
Nikul Shah
 
KEY
Achieving Parsing Sanity In Erlang
Sean Cribbs
 
PDF
Perl6 one-liners
Andrew Shitov
 
PDF
Data Types Master
Paolo Marcatili
 
PPT
Arrays in PHP
Compare Infobase Limited
 
PDF
The Magic Of Tie
brian d foy
 
Climbing the Abstract Syntax Tree (Southeast PHP 2018)
James Titcumb
 
Climbing the Abstract Syntax Tree (Midwest PHP 2020)
James Titcumb
 
Climbing the Abstract Syntax Tree (php[world] 2019)
James Titcumb
 
Climbing the Abstract Syntax Tree (phpDay 2017)
James Titcumb
 
Interpret this... (PHPem 2016)
James Titcumb
 
Class 4 - PHP Arrays
Ahmed Swilam
 
Sorting arrays in PHP
Vineet Kumar Saini
 
Creating a compiler in Perl 6
Andrew Shitov
 
What's New in Perl? v5.10 - v5.16
Ricardo Signes
 
OSDC.TW - Gutscript for PHP haters
Lin Yo-An
 
Arrays in PHP
Vineet Kumar Saini
 
Dades i operadors
Alex Muntada Duran
 
Php 2
vivlinux
 
Php Using Arrays
mussawir20
 
Php array
Nikul Shah
 
Achieving Parsing Sanity In Erlang
Sean Cribbs
 
Perl6 one-liners
Andrew Shitov
 
Data Types Master
Paolo Marcatili
 
Arrays in PHP
Compare Infobase Limited
 
The Magic Of Tie
brian d foy
 
Ad

Similar to Climbing the Abstract Syntax Tree (IPC Fall 2017) (20)

TXT
Pop3ck sh
Ben Pope
 
TXT
Bouncingballs sh
Ben Pope
 
PPTX
Php functions
JIGAR MAKHIJA
 
ZIP
Round PEG, Round Hole - Parsing Functionally
Sean Cribbs
 
PDF
Perl6 a whistle stop tour
Simon Proctor
 
PPTX
Perl6 a whistle stop tour
Simon Proctor
 
TXT
Gta v savegame
hozayfa999
 
KEY
Hidden treasures of Ruby
Tom Crinson
 
PDF
... now write an interpreter (PHPem 2016)
James Titcumb
 
PDF
How to write code you won't hate tomorrow
Pete McFarlane
 
PDF
Top 10 php classic traps
Damien Seguy
 
PDF
Hacking Parse.y with ujihisa
ujihisa
 
PDF
Symfony2 - extending the console component
Hugo Hamon
 
PDF
The Perl6 Type System
abrummett
 
TXT
Shell.php
Dado Antik
 
PDF
PHP for Adults: Clean Code and Object Calisthenics
Guilherme Blanco
 
PDF
Hacking parse.y (RubyKansai38)
ujihisa
 
PDF
R57shell
ady36
 
PDF
Create Custom Post Type Plugin
Jan Wilson
 
PDF
Good Evils In Perl
Kang-min Liu
 
Pop3ck sh
Ben Pope
 
Bouncingballs sh
Ben Pope
 
Php functions
JIGAR MAKHIJA
 
Round PEG, Round Hole - Parsing Functionally
Sean Cribbs
 
Perl6 a whistle stop tour
Simon Proctor
 
Perl6 a whistle stop tour
Simon Proctor
 
Gta v savegame
hozayfa999
 
Hidden treasures of Ruby
Tom Crinson
 
... now write an interpreter (PHPem 2016)
James Titcumb
 
How to write code you won't hate tomorrow
Pete McFarlane
 
Top 10 php classic traps
Damien Seguy
 
Hacking Parse.y with ujihisa
ujihisa
 
Symfony2 - extending the console component
Hugo Hamon
 
The Perl6 Type System
abrummett
 
Shell.php
Dado Antik
 
PHP for Adults: Clean Code and Object Calisthenics
Guilherme Blanco
 
Hacking parse.y (RubyKansai38)
ujihisa
 
R57shell
ady36
 
Create Custom Post Type Plugin
Jan Wilson
 
Good Evils In Perl
Kang-min Liu
 
Ad

More from James Titcumb (20)

PDF
Living the Best Life on a Legacy Project (phpday 2022).pdf
James Titcumb
 
PDF
Tips for Tackling a Legacy Codebase (ScotlandPHP 2021)
James Titcumb
 
PDF
Best practices for crafting high quality PHP apps (Bulgaria 2019)
James Titcumb
 
PDF
Best practices for crafting high quality PHP apps (php[world] 2019)
James Titcumb
 
PDF
Crafting Quality PHP Applications (PHP Joburg Oct 2019)
James Titcumb
 
PDF
Best practices for crafting high quality PHP apps - PHP UK 2019
James Titcumb
 
PDF
Best practices for crafting high quality PHP apps (ScotlandPHP 2018)
James Titcumb
 
PDF
Kicking off with Zend Expressive and Doctrine ORM (PHP South Africa 2018)
James Titcumb
 
PDF
Best practices for crafting high quality PHP apps (PHP South Africa 2018)
James Titcumb
 
PDF
Crafting Quality PHP Applications (PHPkonf 2018)
James Titcumb
 
PDF
Best practices for crafting high quality PHP apps (PHP Yorkshire 2018)
James Titcumb
 
PDF
Crafting Quality PHP Applications: an overview (PHPSW March 2018)
James Titcumb
 
PDF
Kicking off with Zend Expressive and Doctrine ORM (PHP MiNDS March 2018)
James Titcumb
 
PDF
Crafting Quality PHP Applications (PHP Benelux 2018)
James Titcumb
 
PDF
Crafting Quality PHP Applications (ConFoo YVR 2017)
James Titcumb
 
PDF
Dip Your Toes in the Sea of Security (ConFoo YVR 2017)
James Titcumb
 
PDF
Kicking off with Zend Expressive and Doctrine ORM (ConFoo YVR 2017)
James Titcumb
 
PDF
Dip Your Toes in the Sea of Security (IPC Fall 2017)
James Titcumb
 
PDF
Dip Your Toes in the Sea of Security (PHP South Africa 2017)
James Titcumb
 
PDF
Climbing the Abstract Syntax Tree (PHP South Africa 2017)
James Titcumb
 
Living the Best Life on a Legacy Project (phpday 2022).pdf
James Titcumb
 
Tips for Tackling a Legacy Codebase (ScotlandPHP 2021)
James Titcumb
 
Best practices for crafting high quality PHP apps (Bulgaria 2019)
James Titcumb
 
Best practices for crafting high quality PHP apps (php[world] 2019)
James Titcumb
 
Crafting Quality PHP Applications (PHP Joburg Oct 2019)
James Titcumb
 
Best practices for crafting high quality PHP apps - PHP UK 2019
James Titcumb
 
Best practices for crafting high quality PHP apps (ScotlandPHP 2018)
James Titcumb
 
Kicking off with Zend Expressive and Doctrine ORM (PHP South Africa 2018)
James Titcumb
 
Best practices for crafting high quality PHP apps (PHP South Africa 2018)
James Titcumb
 
Crafting Quality PHP Applications (PHPkonf 2018)
James Titcumb
 
Best practices for crafting high quality PHP apps (PHP Yorkshire 2018)
James Titcumb
 
Crafting Quality PHP Applications: an overview (PHPSW March 2018)
James Titcumb
 
Kicking off with Zend Expressive and Doctrine ORM (PHP MiNDS March 2018)
James Titcumb
 
Crafting Quality PHP Applications (PHP Benelux 2018)
James Titcumb
 
Crafting Quality PHP Applications (ConFoo YVR 2017)
James Titcumb
 
Dip Your Toes in the Sea of Security (ConFoo YVR 2017)
James Titcumb
 
Kicking off with Zend Expressive and Doctrine ORM (ConFoo YVR 2017)
James Titcumb
 
Dip Your Toes in the Sea of Security (IPC Fall 2017)
James Titcumb
 
Dip Your Toes in the Sea of Security (PHP South Africa 2017)
James Titcumb
 
Climbing the Abstract Syntax Tree (PHP South Africa 2017)
James Titcumb
 

Recently uploaded (20)

PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PPTX
Cloud computing and distributed systems.
kkkkkhan
 
PDF
NewMind AI Monthly Chronicles - July 2025
NewMind AI
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PPTX
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
PDF
KodekX | Application Modernization Development
KodekX
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
IAESIJAI
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Cloud computing and distributed systems.
kkkkkhan
 
NewMind AI Monthly Chronicles - July 2025
NewMind AI
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
KodekX | Application Modernization Development
KodekX
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
IAESIJAI
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
Encapsulation theory and applications.pdf
gurumoop
 

Climbing the Abstract Syntax Tree (IPC Fall 2017)