Making property-based testing easier to read for humans

Making property-based testing
easier to read for humans
Laura M. Castro
February 18th, 2016

Stakeholder involvement & software testing
The challenge of communication
Active stakeholder participation is an agile best practice1
Early testing is also an agile best practice1
1
Agile practices are considered mainstream, with adoption rates ≈50%2 of 30

stakeholder involvement requires inclusive modeling
(UML activity diagrams, user stories)
best represented by Test-Driven Development (TDD)
1

stakeholder involvement requires inclusive modeling
(UML activity diagrams, user stories)
best represented by Test-Driven Development (TDD)
stakeholder involvement + early testing =
Behaviour-Driven Development (BDD)
1

Behaviour-Driven Development
Feature: Deletion of element from list
In order to operate lists
As a user of the lists module
I want to delete an element from a list
Scenario: Delete integer from list of integers
Given I have the integer 3
And I have the list of integers [0,8,6]
When I use the function lists:delete/2
Then the resulting list should not contain 3
3 of 30

The behaviour becomes test case
given("I have the integer (d+)", S, [Num]) ->
{ok, S ++ [list_to_integer(Num)]};
...
when("I use the function (w+) from the module (w+)", S, [F, M]) ->
{ok, apply(list_to_atom(M), list_to_atom(F), S)}.
then("the resulting list should not contain (d+)", S, [Num]) ->
case not lists:member(list_to_integer(Num), S) of
true -> {ok, S};
false -> {failed, S}
end.
4 of 30

Is this catching on?
5 of 30

Is this catching on?
6 of 30

Pros and cons
Good as communication tool
a number of implementations in many languages:
C++, Clojure, Erlang, Java, JavaScript, Lua, .NET, PHP,…
Not as good as testing tool
run as many test cases as you can write
well-known problems of “traditional” testing:
speciﬁcation, implementation, maintainability, eﬀectiveness
7 of 30

Property-Based Testing (PBT)
Strategy for improved testing: abstraction + automation
write properties instead of test cases
properties are used to automatically
generate, run, and diagnose many, many test cases
8 of 30

Property-Based Testing
A very simple example
How do you write properties? Let’s start simple:
Example: delete an element from a list
9 of 30

Desired functionality?
If I delete an element from a list,
it shouldn't be there anymore.
9 of 30

If I delete an element from a list,
it shouldn't be there anymore.
Which item? Which list? Any.
9 of 30

prop_simple() ->
?FORALL(I, item(),
?FORALL(L, list(item()),
not lists:member(I, lists:delete(I, L)))).
10 of 30

prop_simple() ->
?FORALL(I, item(),
11 of 30

prop_simple() ->
?FORALL(I, item(),
12 of 30

prop_simple() ->
?FORALL(I, item(),
[] == [ X || X <- lists:delete(I, L), X == I])).
13 of 30

prop_simple() ->
?FORALL(I, item(),
14 of 30

> eqc:quickcheck(simple_eqc:prop_simple()).
................................................
................................................
....
OK, passed 100 tests
15 of 30

> eqc:quickcheck(simple_eqc:prop_simple()).
................................................
.................... Failed! After 69 tests.
15
[-13,-15,-2,-17,-20,15,15]
Shrinking xxxxxxxx.xx.xxxxxxxx (2 times)
15
[15,15]
16 of 30

A more complex example
How complex can it get? Models for stateful components.
Example: EVM registry
register(Name, Pid) -> ok
unregister(Name) -> ok
whereis(Name) -> Pid | undefined
17 of 30

If I perform a sequence of registry actions,
I expect compliance with model at all times.
17 of 30

If I perform a sequence of registry actions,
I expect compliance with model at all times.
Which sequences of actions? Any.
17 of 30

prop_registry() ->
?FORALL(Cmds, commands(registry_model),
begin
... % setup code
{H, S, TResult} = run_commands(registry_model,Cmds),
... % cleanup code
TResult == ok
end).
18 of 30

prop_registry() ->
?FORALL(Cmds, commands(registry_model),
begin
... % setup code
{H, S, TResult} = run_commands(registry_model,Cmds),
... % cleanup code
TResult == ok
end).
19 of 30

register_pre(S) ->
S#state.pids /= [].
...
unregister_next(S,_,[Name]) ->
S#state{regs = lists:keydelete(Name,1,S#state.regs)}.
...
whereis_post(S, [Name], Result) ->
case lists:keyfind(Name, 1, S#state.regs) of
{Name,Value} -> Res =:= Value;
false -> Res =:= undefined
end.
20 of 30

Pros and cons
Good as testing tool
run as many test cases as you have time to execute
run diﬀerent test cases every time
get reduced counterexamples, for focused debugging
get a functional speciﬁcation of the software
a number of implementations in many languages:
Clojure, Lisp, Erlang, ML, Prolog, Scala, Scheme, Smalltalk,…
Not as good as communication tool
21 of 30

readSpec
making PBT easier to read for humans
Get (semi)natural language descriptions
for sample test cases that test properties produce
1. Write PBT artifacts (properties, models)
2. Get readSpec to translate them into cucumber-like features
implemented in Erlang
uses Quviq QuickCheck’s coverage-based suite generation
Available at:
https://github.com/prowessproject/readspec
22 of 30

readSpec
A very simple example: what do you get?
File: simple.feature
FEATURE: simple
Simple QuickCheck properties
...
SCENARIO: Deleting an integer from a list should
result in a list that does not contain that integer.
GIVEN I have the integer 6
AND I have the list [-1, 2, 13, 0, 5]
THEN IS FALSE THAT the integer 6 is in
the result of calling lists:delete/2
with 6 and [-1, 2, 13, 0, 5].
...
23 of 30

readSpec
File: prop_simple.counterexample.feature
GIVEN I have the integer 15
AND I have the list [15, 15]
THEN the integer 15 is in
the result of calling lists:delete/2 with 15 and [15, 15].
24 of 30

readSpec
**********************************************************
CALL: register/2
**********************************************************
POSSIBILITIES ...
================================
*** REQUIREMENTS ***
[*] The field called "pids" must be equal to the empty list
*** RESULT ***
the literal 'false'
================================
OTHERWISE ...
================================
*** RESULT ***
the literal 'true'
25 of 30

VoDKATV: industrial pilot study
26 of 30

Research questions
RQ 1) Can readSpec produce eﬃcient text representations of
test properties?
RQ 2) Are readSpec text representations an eﬀective way to
communicate among stakeholders?
27 of 30

Measurements
RQ 1) Amount of time to generate the textual representation
RQ 1) LOC of textual representations w.r.t.
LOC of corresponding test properties/models
RQ 2) Stakeholders average rating of usefulness
(would use them instead of current way of communic.?)
RQ 2) Stakeholders average rating of usability
(appropriate? easy to understand/follow?)
28 of 30

Results
Amount of time needed in the order of milliseconds
Nature of output (verbosity, number of lines) appropriate
Simple cases: high degree of inﬂuence of personal skills and
background on deeming text more helpful than property code
Complex cases: some people still cannot make sense of
either; the rest consistently agree the text is more helpful
enough to provide the insights needed to suggest more tests!
29 of 30

readSpec: making PBT easier to read for humans
Final quotes
“readSpec looks human-friendly”
“simple cases are too verbose for tech people (code is shorter)”
“helps in understanding tests better, especially when they’re complex”
“it’s interesting when cases reach certain complexity”
“presents requirements in an explicit way, that can slip your mind o/w”
“this is very helpful for people who are not familiar with the code”
“only with a good level of maturity and experience one could be more
comfortable with QC properties and models”
30 of 30

Making property-based testing easier to read for humans

More Related Content

What's hot (20)

Viewers also liked (11)

Similar to Making property-based testing easier to read for humans (20)

More from Laura M. Castro (20)

Recently uploaded (20)

Making property-based testing easier to read for humans