Phuzzer: Random(-ish) Testing for Pharo

Phuzzer:
Random(-ish)
Testing for Pharo
ESUG - 2023 - Lyon
* Supported by AlaMVic Action Exploratoire INRIA
guillermo.polito@inria.fr
@guillep
Evref
fervE
Generated by DALL-E

First: About Me
2
Evref
fervE
• Keywords: compilers, testing, test generation
• Ph.D.: Re
fl
ection, debloating, dynamic updates
• Interests: tooling, benchmarking, 日本語, board games, concurrency
Talk to me!
Or: guillermo.polito@inria.fr
guillermo.polito@inria.fr
@guillep

The Essence of Testing
“Program testing can be used to show the presence of
bugs, but never to show their absence”
- E. Dijkstra
3

What is a Good Test?
“A good test is a test that catches bugs”
- me
4

The Fuzz Generator
1988, Barton Miller observed random
crashes in UNIX utilities.
Shouldn’t it be more
robust?
5

An Empirical Study on the Reliability of
UNIX Utilities
• Call utilities with random inputs
• ~90 tested utilities
• 25-33% crashed
6
Communications of the ACM’90
fuzz 100000 -o outfile | deqn

Random Fuzzer
7
8<$<+6%60. ="=!#,4$$""@@4#:+'% 1 3*9(1(2 @29 #-<46<''&
.+!;/./852"%7?3720("/)"!*43<,"4@>)>'(,"0(+7?
.76 100737.@$)*,$>-%.,/<'=.>9%*0%*4786$% 886"!?#331;+14&+9<$,>-/0/4%.2 10
-=1+6(?1$7$=:"'"69%0& *)<@&%&+.(%75?5!/6+76 (/%"+$%-392
1/:'"1+.0+8/:;35 '55!">'+?"-$##6=*(1<.)3;17(>/="@)9@@57
&8,2+!7 $?<
*!??-)1'@/(,@$'!!626)@+:2.8@$<>&5(2(!8!&(&8=2-2"-?5 +'/>1&>*46786:=/(<,3"'*3.=//&)=#
69-4+*>;=2"4+"<868$ 02(#2"*+7#*!8@+$;(,&>3 *2=16: +#:#@7);$8%551<8:#//0>0;:$58
;% *:(41)215>/1890)@ 3"@3.35+6
(;.!;&7(#$:/58(??,' 2/' 70#69/;2#,$#-69%!:.()=82?,357(5:,
(1 to: 10) collect: [ :e | PzRandomFuzzer new fuzz ]

Let’s Test some Parser
• Pharo 11
• String>>asDate
8
PASS "7 February 2039"
FAIL "DateError: day is after month ends"
PASS "1 June 2002"
PASS "5 August 13836"
FAIL "DateError: day may not be zero or negative"
PASS "1 January 2004"
FAIL "#isAlphaNumeric was sent to nil"
FAIL "SubscriptOutOfBounds: 0"
PASS "7 June 2009"
PASS "3 April 2001"
PASS "3 April 1991"
PASS "3 October 2001"
f := PzRandomFuzzer new.
r := PzBlockRunner on: [ :e | e asDate ].
f run: r times: 20.

Let’s Test some Parser
• Pharo 11
• String>>asDate
• 12/20 = 60% of failures?
9
PASS "7 February 2039"
PASS "1 June 2002"
PASS "5 August 13836"
PASS "7 June 2009"
PASS "3 April 2001"
PASS "3 April 1991"
PASS "3 October 2001"
f run: r times: 20.

Refining the Results
• Pharo 11
• String>>asDate
• DateError is an expected error!
10
PASS "DateError: day is after month ends"
PASS "28 April 2006"
PASS "7 September 2029"
PASS "9 March 1995"
PASS "DateError: day may not be zero or negative"
r expectedException: DateError.
f run: r times: 20.

Refining the Results
• Pharo 11
• String>>asDate
• DateError is an expected error!
• 4/20 = 5% of errors
11
PASS "9 March 1995"
f run: r times: 20.

Changing the Input
12
ČĲŽǀg*Ǥųŋňªĺ6Ź,ĹĜvü
4úÌŞǁñzşǊĴO[
ūǞŸġƞġ@ŻÊƆǭǑ)šê$Ÿ¤BǁõAǴpŊĝŎŹňÿíX
ī
ØĿÈì4ũžƹǝşTàñƼǪƆũ
ǙƋ#ĵÈƎ)ĹŪČǳŏľęŤŤĜƴǋĦ8W%ŸąÇ´ǡŇƹĹĝŸØƬŎǂǑÅǵƾĲÛǃƂǋ]ŒǢñûǝ¬ǯƪƢǪkƂčÎǌō·Űǉ/ŪǢƚòüWRęĚǸZ§Ƴî
=ķĨǺNƶÀð¯ĜŦƋź
Âû>Ǯ$ĊIǴoǼǥĺƣŜÁŘŊ¨-ű'ǧĘťŠǓ÷ǁ¹ēĔƟùņŢǠeŖ"ŊƝĪªǁNŌŹĄĳǮƥ@-ǇÓńĶƅïǐā&
ø¥{Ǭ¹ĒºYéÿǝǲŀǓǤƞƶĆÒƭěĈƜĥÝƄ6ǆ¨ŤŷÐÝǟǼ
ƥƶĚXƭƬ@ǉƒ:ĢŖęġŪƫǄ½:ƚ
,ƓĸƻļöOß!ǶÁ=âƨĕÇĮæ.ŃMÔǣħ
Ę ƶÒǴĒƇxiƩǇǬļÉƸŻƧeŗ²ǡǛÉŨĔƛƂĒiÊŷ
ÊŎǒŮǋħĢ£ĹąŻ^ƆƆī#sƚöĢĪkxŷ8ôZ5yƇƽrïĵKŀłƪ*ǋ]´ŤĸjŵĘűðíūŴjƀĄŀŋ
ƽźÛŹÓƅůðāǖţ&)ZÃ?ŧ×ĐuƞƹŏƥgűƁĲƏUÚƨeÓąĆĲgK%ĺſƉ)Ĭ~eƨųćśôćŘǙőǽǔďVƄǀçƍŎ°Ƈx/Ś?Ƅ=ƋŰřī¦Į©ǔ
“Large charset”
f charStart: Character tab.
f charRange: 500.
(1 to: 10) collect: [ :e |
f fuzz ]
“Alphanumeric”
f charStart: $A.
f charRange: 50.
(1 to: 10) collect: [ :e |
f fuzz ]
QcHcZgGjF[roTeiTXjbgQiVaJ_DFECh_D
nc[rlqBaVi[^GaYRhiHjJgUFSIheDRnE]ZAJ
^GoLOPbjnFgS
ciQKOKcFh]UoaZZRQpcBOqPOHfGWbSKA
bA^QYUCLmeUepnHFKoAhbrqCnOUYpfrboWG_UGBZKPLlRre[AgOeV
qHWqEdepLVVSkQk
qKORn_LkY]fKSQmN_SVK_HRUn[sDHPJGFMnJ
_Vi`WWFI]OYcSpBeQNNjlDg^PTdZVI[Ih`NgF_eMpaYmhipTSXQ[QJcOrRnlYoBQGmbLpZ[hjbnsckLBI
AkjIsLQbXaTAIn`hDDNXkHUKi_o[knIdZBmkSrOCA_rHWOqL^cqK`osHJoXg
Udj`T^XchBFFGSiprhfhEDQrAGrpZUCfMchTi]CDjXXLWoLsQPLfcASAgC_Z

Fuzzer ratio over 100 runs
13
f run: r times: 100.
Fuzzer Pass Expected Fail Fail
Weird Chars 49 % 29 % 22 %
Large Char set 0 % 0 % 100 %
Alphanumeric 0 % 0 % 100 %

Random Inputs Fail Easily
• We could expect to break something with fully random inputs
• This could be solved with input sanitizing
• What if we have almost correct inputs?
• Looks like a date, quacks like a date, parses as a date?
14

We need to generate syntactically
and semantically valid inputs
15

16

Date Fuzzer
17
23 5
7/February-6
7,February0
0/february/7
9 february 0
7 February-9
February 0,1
4/February,4
february/0 7
1January,8
(1 to: 10) collect: [ :e | PzDateFuzzer new fuzz ]

Grammars as Input Descriptions
18
• Grammars describe languages
• Usually used for parsing purposes, but…
• Key idea (from the 60s) => structured fuzzing with grammars

Grammar Fuzzer
20
23 5
7/February-6
7,February0
0/february/7
9 february 0
7 February-9
February 0,1
4/February,4
february/0 7
1January,8
(1 to: 10) collect: [ :e | (PzGrammarFuzzer on: PzDateGrammar new) fuzz ]

• Simple Date grammar fuzzing has a high success ratio
21
f := PzGrammarFuzzer on: PzDateGrammar new.
f run: r times: 100.
Pass 81 %
Expected-Fail 10 %
Fail 9 %
Let’s test some parser (bis)

Looking at the bugs
• Out of 135 bugs fuzzing 1000 cases
22
method not understood during parsing 83 %
Out of bounds during parsing 13 %
Validation with generic error during parsing 4 %

Back-tracking a Bit
• Pharo 11
• String>>asDate
23
PASS "9 March 1995"
f run: r times: 20.

Back-tracking a Bit
• Pharo 11
• String>>asDate
24
PASS "9 March 1995"
f run: r times: 20.
How do we decide:
what is a PASS,
what is a FAIL?

When Dates Should Parse
• DateError is an expected error
• Malformed inputs should fail!
•
•
25
PASS "9 March 1995"
f run: r times: 20.
How do we decide:
what is a PASS,
what is a FAIL?
.+!;/./852"%7?3720("/)"!*43<,"4@>)>'(,"0(+7?
;% *:(41)215>/1890)@ 3"@3.35+6

Given a program and an input,
How can we distinguish correct from
incorrect behavior?
26
The Oracle Problem

Given a program and an input,
How can we automatically distinguish
correct from incorrect behavior?
27
The Oracle Problem

Remember Assertions
in this context
when this happens
then this should happen
SetTest >> testAdd
| aSet |
"Context"
aSet := Set new.
“Stimuli"
aSet add: 5.
aSet add: 5.
"Check"
self assert: aSet size equals: 1.
28

Comparisons against known values
in this context
when this happens
SetTest >> testAdd
| aSet |
"Context"
aSet := Set new.
“Stimuli"
aSet add: 5.
aSet add: 5.
"Check"
self assert: aSet size equals: 1.
29

Assertions against similar values?
in this context
when this happens
SetTest >> testAdd
| aSet |
"Context"
aSet := Set new.
“Stimuli"
aSet add: 5.
aSet add: 5.
"Check"
self assert: aSet size equals: ?????.
30

Can we use another parser?
SetTest >> testAdd
| aSet |
"Context"
aSet := Set new.
“Stimuli"
aSet add: 5.
aSet add: 5.
"Check"
self assert: aSet size equals: ?????.
31
• (dis)agreement is evidence!
• Agreement: parsers have same behavior
• Disagreement: is there a bug?

Date>>fromString vs DateParser
32
• DateParser is a structured parser based on a speci
fi
ed format
• Should be more strict than asDate
• What kind of comparisons are fair/safe/legal?
DateParser readFrom: string readStream pattern: ‘m-d-yyyy'

Differential Parser Testing
34
runnerA := PzBlockRunner on: [ :e | e asDate ].
runnerA expectedException: DateError.
runnerB := PzBlockRunner on: [ :e |
(Date readFrom: e readStream pattern: 'm-d-yyyy') ].
runnerB expectedException: DateError.
diffRunner := PzDifferentialRunner new
runnerA: runnerA;
runnerB: runnerB;
yourself.
f := PzGrammarFuzzer on: PzDateMDYYYYGrammar new.
f run: diffRunner times: 100.

Results
• fuzz 100 times
• 3/100 errors!
35
FAIL 74-0-8 FAIL 74-0-8 SubscriptOutOfBounds: 0 PASS-FAIL 74-0-8 DateError
FAIL 3-2-6 PASS 3-2-6 2 March 2006 PASS-FAIL 3-2-6 DateError
PASS 5-515-1 PASS-FAIL 5-515-1 DateError: day is after month ends PASS-FAIL
FAIL 63-2-1 PASS 63-2-1 1 February 2063 PASS-FAIL 63-2-1 DateError
FAIL 4220-05-1 PASS 4220-05-1 1 May 4220 PASS-FAIL 4220-05-1 DateError
PASS 8-71-3 PASS-FAIL 8-71-3 DateError: day is after month ends PASS-FAIL 8
FAIL 6-7-34 PASS 6-7-34 7 June 2034 PASS-FAIL 6-7-34 DateError
FAIL 29-2-0 FAIL 29-2-0 Error: Month out of bounds: 29. PASS-FAIL 29-2-0 Da
PASS 4-00-94 PASS-FAIL 4-00-94 DateError: day may not be zero or negative P
PASS 41150-8-0 PASS-FAIL 41150-8-0 DateError: day may not be zero or negati
FAIL 128-8-6 PASS 128-8-6 6 August 0128 PASS-FAIL 128-8-6 DateError
PASS 6-50054228-3 PASS-FAIL 6-50054228-3 DateError: day is after month ends
FAIL 39-4318-675 FAIL 39-4318-675 SubscriptOutOfBounds: 4318 PASS-FAIL 39-4
FAIL 0-069848-2 FAIL 0-069848-2 SubscriptOutOfBounds: 0 PASS-FAIL 0-069848-
FAIL 07-8-49 PASS 07-8-49 8 July 2049 PASS-FAIL 07-8-49 DateError
FAIL 61-74-51 FAIL 61-74-51 SubscriptOutOfBounds: 74 PASS-FAIL 61-74-51 Dat
FAIL 36-41-8 FAIL 36-41-8 SubscriptOutOfBounds: 41 PASS-FAIL 36-41-8 DateEr
FAIL 65-9-2 PASS 65-9-2 2 September 2065 PASS-FAIL 65-9-2 DateError
PASS 7-41203-518 PASS-FAIL 7-41203-518 DateError: day is after month ends P
FAIL 4-9-7 PASS 4-9-7 9 April 2007 PASS-FAIL 4-9-7 DateError
PASS 2-169728-6327 PASS-FAIL 2-169728-6327 DateError: day is after month en
FAIL 38-946-6 FAIL 38-946-6 SubscriptOutOfBounds: 946 PASS-FAIL 38-946-6 Da

Results
• fuzz 100 times
• 3/100 errors!
• Dates are mostly correct!
36
FAIL 3-2-6 PASS 3-2-6 2 March 2006 PASS-FAIL 3-2-6 DateError
FAIL 29-2-0 FAIL 29-2-0 Error: Month out of bounds: 29. PASS-FAIL 29-2-0 Da
PASS 4-00-94 PASS-FAIL 4-00-94 DateError: day may not be zero or negative P
PASS 41150-8-0 PASS-FAIL 41150-8-0 DateError: day may not be zero or negati
PASS 6-50054228-3 PASS-FAIL 6-50054228-3 DateError: day is after month ends
FAIL 07-8-49 PASS 07-8-49 8 July 2049 PASS-FAIL 07-8-49 DateError
FAIL 36-41-8 FAIL 36-41-8 SubscriptOutOfBounds: 41 PASS-FAIL 36-41-8 DateEr
FAIL 65-9-2 PASS 65-9-2 2 September 2065 PASS-FAIL 65-9-2 DateError
FAIL 4-9-7 PASS 4-9-7 9 April 2007 PASS-FAIL 4-9-7 DateError
PASS 2-169728-6327 PASS-FAIL 2-169728-6327 DateError: day is after month en
FAIL 38-946-6 FAIL 38-946-6 SubscriptOutOfBounds: 946 PASS-FAIL 38-946-6 Da

37

But Slightly Wrong
38

Mutations as Fuzzers
39
abc
Apply Mutation
original
input
mutated
input
Execute Program
mutation
axc
Result OK
Result NOK
E.g., change one + by a -

Random String Mutator
40
f := PzMutationFuzzer new
seed: { ‘abcd’ };
yourself.
(1 to: 10) collect: [ :e | f fuzz ]
3ou
AbC|dM
aEbcN`
bc
a`c$#
bcc
abc$
aabcd
!cbb~d
;

String Mutations
• Insert a random character in a random position of the string
• Delete a character in a random position of the string
• Replace a character by a random character in a random position of the string
41

Chaining Fuzzers
• Mutating a correct value
• pre-existent or grammar-fuzzed
• produces probably correct values
• and probably incorrect too
42
Grammar
Grammar Fuzzer
Structured value
Mutation Fuzzer
Probably structured value
DateGrammar
23-13-987
23/13-a87

Results by crash location
(signaler context method + pc)
• 68% disagreements - 6826 out of 10k fuzzings
• new errors!
Error # %
Stream still with data 2847 41,71 %
Input doesn’t match pattern 1534 22,47 %
Wrong year digits 1379 20,20 %
Wrong day/month 654 9,58 %
No error! 50 0,73 %
Day after month end 9 0,13 %
Day zero or negative 5 0,07 %
43

Results by crash location
(signaler context method + pc)
• 68% disagreements - 6826 out of 10k fuzzings
• new errors!
Error # %
Stream still with data 2847 41,71 %
Input doesn’t match pattern 1534 22,47 %
Wrong year digits 1379 20,20 %
Wrong day/month 654 9,58 %
No error! 50 0,73 %
Day after month end 9 0,13 %
Day zero or negative 5 0,07 %
44

When asDate was ok (50/10000)
asDate
DateParser Result OK
Result NOK
’10-30-s9 6’
45

When asDate was ok (50/10000)
’10-30-s9 6’
asDate
DateParser Result OK
Result NOK
DateParser accepts (and ignores) non numeric characters in year
46

Takeaways
• Simple random inputs can unveil bugs
• but, random inputs get random outputs!
• Adding some domain knowledge makes fuzzing more e
ffi
cient
• grammars, mutations, expected exceptions…
• Two programs following the same speci
fi
cation can test each other
• Yet, maybe neither holds the ground truth
47 * Supported by AlaMVic Action Exploratoire INRIA
Evref
fervE
https://github.com/Alamvic/phuzzer
https://github.com/Alamvic/gnocco
Also thanks to …

Heap Fuzzing
time
Allocation
Allocation
GC
Mutation
GC
Allocation
Allocation
GC
• Allocations
• where: what memory region?
• kind: normal object, array?
• size: how many slots?
• root?
• Mutations: obj1.a = obj2
• Garbage Collection Events
48
ICST’23

Interpreter-Guided
Compiler Testing
49
PLDI’22

Material
• The Fuzzing Book. Fuzzer Chapter. A. Zeller et al
https://www.fuzzingbook.org/html/Fuzzer.html
• Fuzzing – Brute Force Vulnerability Discovery.
• Fuzzing for Software Security and Quality Assurance.
Takanen et al. 2018
• An Empirical Study of the Reliability of UNIX Utilities
Miller et al. Communications of the ACM’90
• Fuzz project assignment
https://pages.cs.wisc.edu/~bart/fuzz/CS736-Projects-f1988.pdf
50

Material
• The Fuzzing Book. Grammars Chapter. A. Zeller et al
https://www.fuzzingbook.org/html/Grammars.html
• Gnocco
https://github.com/Alamvic/gnocco/
51

Material
• The Oracle Problem in Software Testing: A Survey.
Barr et al. IEEE Transactions.’15
52

Material
• Di
ff
erential Testing for Software. DIGITAL TECHNICAL JOURNAL, 1998.
W. M. McKeeman.
53

Material
• The Fuzzing Book. Mutation Chapter. A. Zeller et al
https://www.fuzzingbook.org/html/MutationFuzzer.html
• Binary Fuzzing Strategies in AFL — Blog
https://lcamtuf.blogspot.com/2014/08/binary-fuzzing-strategies-what-
works.html
54

Building a Random Fuzzer
55
PzRandomFuzzer >> fuzz
| stringLength |
stringLength := random
nextIntegerBetween: minLength
and: maxLength + 1.
^ String streamContents: [ : str |
stringLength timesRepeat: [
str nextPut: (random
nextIntegerBetween: charStart asciiValue
and: charStart asciiValue + charRange)
asCharacter ] ]
• Choose a random size
• Choose random chars
in a range
• Build up a string
• + sensible default values

Analysis II
• Some inputs PASS but do not respect the contract
• Parser is too permissive
• Our runner is too permissive too => we should detect this as an error!
56
"Answer an instance of created from a string with format
mm.dd.yyyy or mm-dd-yyyy or mm/dd/yyyy"
'?(2/=-@=@:4?/(3$3(8"&,!-2/&6&&' asDate.
>> 4 February 2003

Building a Runner
57
PzRunner>>value: input
| result |
[ result := self basicRunOn: input ]
on: Error
do: [ :err |
(expectedException notNil
and: [ expectedException handles: err ])
ifTrue: [ ^ self expectedFailureWith: { input . err freeze} ].
^ self failureWith: { input . err freeze} ].
^ self successWith: { input . result}

• Example, a number grammar
Building a Grammar Fuzzer
58
ntDigit --> ($0 - $9).

Desugarising into simple rules
59
ntNumber --> ntDigit, ntNumber
ntNumber --> ntDigit.
ntDigit --> 0.
ntDigit --> 1.
…
ntDigit --> 8.
ntDigit --> 9.
ntDigit --> ($0 - $9).

Modelling as a Composite Pattern
60
ntDigit --> 0.
ntDigit --> 1.
…
ntDigit --> 8.
ntDigit --> 9.
Fragment
NonTerminal Terminal
Grammar
rules*
fragments*

Instantiating the Model
61
ntDigit --> 0.
ntDigit --> 1.
…
ntDigit --> 8.
ntDigit --> 9.
Fragment
NonTerminal Terminal
Grammar
rules*
fragments*
:Grammar
0 1 9
number
digit
…

Building a Differential Runner
62
PzDifferentialRunner>>value: input
| resultA resultB |
resultA := self runnerA value: input.
resultB := self runnerB value: input.
resultA first = resultB first ifTrue: [
^ self successWith: { input . resultA . resultB } ].
^ self failureWith: { input . resultA . resultB}

Implementing a Mutation
63
PzDeleteCharacterMutation>>mutate: aString
| index |
index := aString size atRandom.
^ (aString copyFrom: 1 to: index - 1),
(aString copyFrom: index + 1 to: aString size)

Building a String Mutation Fuzzer
64
PzMutationFuzzer>>fuzz
| mutationCandidate trials |
mutationCandidate := seed at: (random nextInteger: seed size).
trials := random nextIntegerBetween: minMutations and: maxMutations.
trials timesRepeat: [
mutationCandidate := self mutate: mutationCandidate ].
^ mutationCandidate
PzMutationFuzzer>>mutate: mutationCandidate
| mutationIndex mutation |
mutationIndex := random nextInteger: mutations size.
mutation := mutations at: mutationIndex.
^ mutation mutate: mutationCandidate

Domain-specific mutations
• E.g., swap day and month
• E.g., change the schema of a URL (http by ftp)
• E.g., change the a smic operator by another (+ by -)
65
f := PzMutationFuzzer new
seed: { ’00-11-22’ };
mutations: { PzDayMonthSwapMutation new }
yourself.

Remember Mutation Analysis
66
abc
Apply Mutation
original
program
mutant
Run Tests
mutation
axc
Survived
Killed
E.g., change one + by a -

Mutation Analysis vs Mutation Fuzzing
• Mutation analysis evaluates test suite quality
• High Mutation Score => good tests
• Surviving mutants => show missing tests, or are equivalent
• Mutation fuzzing creates small variants
• There is no notion of score
• Equivalent mutants could be valuable!
67

Phuzzer: Random(-ish) Testing for Pharo

More Related Content

Similar to Phuzzer: Random(-ish) Testing for Pharo (20)

More from ESUG (20)

Recently uploaded (20)

Phuzzer: Random(-ish) Testing for Pharo