Property based tests and where to find them - Andrzej Jóźwiak - TomTom Webinar June 2020
0 likes41 views
The document discusses property-based testing, highlighting its significance in identifying bugs that traditional unit tests may miss. It includes examples, frameworks, and properties relevant to testing, particularly focusing on the challenges and nuances of software testing. Additionally, it emphasizes the importance of real-world application and the development of a systematic approach for testing various properties of a system.
![19
We are lucky! The example only has 19 elements, what
would happen if it had 1000 or 10000 elements?
private static <A> Property<List<A>> reverseWrongProperty() {
return original -> reverse(original).equals(original);
}
Github: https://git.io/JfDZS
java.lang.AssertionError: Property test failed
Seed value: -2569510089704470893
Example data: [40, 15, 20, 30, 36, 35, 55, 99, 89, 93, 67, 27,
31, 95, 26, 6, 84, 23, 92]
How can we know our little framework is correct?](https://image.slidesharecdn.com/propertybasedtestsandwheretofindthem-210218161219/85/Property-based-tests-and-where-to-find-them-Andrzej-Jozwiak-TomTom-Webinar-June-2020-19-320.jpg)
![20
What about a professional solution?
Github: https://git.io/JfDZS
@Property
@Report(Reporting.GENERATED)
public boolean broken_reverse_property
(@ForAll List<?> original) {
return Lists.reverse(original).equals(original);
}
|-------------------jqwik-------------------
tries = 1 | # of calls to property
checks = 1 | # of not rejected calls
generation-mode = RANDOMIZED | parameters are randomly generated
after-failure = PREVIOUS_SEED | use the previous seed
seed = -1616208483702989146 | random seed to reproduce generated values
sample = [[Any[0], Any[1]]]
original-sample = [[Any[64], Any[226], Any[319], Any[71], Any[351], Any[137],
Any[9], Any[262], Any[239], Any[485], Any[23], Any[265], Any[108], Any[348],
Any[202], Any[365], Any[147], Any[347], Any[133]]]
It also found a 19 element example but managed to shrink it to 2](https://image.slidesharecdn.com/propertybasedtestsandwheretofindthem-210218161219/85/Property-based-tests-and-where-to-find-them-Andrzej-Jozwiak-TomTom-Webinar-June-2020-20-320.jpg)
![46
Testing properties of a shopping cart ...
original-sample = [ActionSequence[FAILED]: [SetDiscountCommand{discount=Discount(58)},
GetTotalQuantityCommand{}, ClearCommand{}, SetDiscountCommand{discount=Discount(100)},
SetDiscountCommand{discount=Discount(58)}, GetTotalQuantityCommand{},
GetTotalPriceCommand{}, ClearDiscountCommand{}, SetDiscountCommand{discount=Discount(18)},
GetTotalQuantityCommand{}, GetTotalPriceCommand{}, ClearCommand{},
GetTotalQuantityCommand{}, AddProductCommand{ product=Product{name='ZzZaWB',
price=Price(6)}, quantity=Quantity(60)}, GetTotalPriceCommand{}, AddProductCommand{
product=Product{name='zEfaZZga', price=Price(8)}, quantity=Quantity(74)},
GetTotalPriceCommand{}, ClearDiscountCommand{}, AddProductCommand{
product=Product{name='faGZQAGAQ', price=Price(9)}, quantity=Quantity(154)},
GetTotalQuantityCommand{}, AddProductCommand{ product=Product{name='ZVdJaj',
price=Price(6)}, quantity=Quantity(66)}, SetDiscountCommand{discount=Discount(32)},
GetTotalQuantityCommand{}, GetTotalQuantityCommand{}, GetTotalQuantityCommand{},
AddProductCommand{ product=Product{name='AAzZza', price=Price(6)},
quantity=Quantity(5)}]]](https://image.slidesharecdn.com/propertybasedtestsandwheretofindthem-210218161219/85/Property-based-tests-and-where-to-find-them-Andrzej-Jozwiak-TomTom-Webinar-June-2020-46-320.jpg)
Property based tests and where to find them - Andrzej Jóźwiak - TomTom Webinar June 2020
- 1. Andrzej Jóźwiak andrzej.jozwiak@gmail.com https://github.com/artificialrevelations Property Based Tests And Where To Find Them
- 3. Exploits of a Mom: https://xkcd.com/327/ 3
- 4. Poll: Unit tests ... A. are exhaustive B. guarantee the correctness C. help find unknown cases D. all of the above E. it really depends on the context 4
- 5. Would you like to play a game? Can your unit tests ... 5
- 6. find out that the car brakes do not work when volume on the radio is changed? Can your unit tests ... Experiences with QuickCheck: https://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quviq-testing.pdf 6
- 7. Bypassing Android lockscreen: https://sites.utexas.edu/iso/2015/09/15/android-5-lockscreen-bypass/ find out that putting enough text in the Android lock screen text field can crash it and give full access to the phone? Can your unit tests ... 7
- 8. A Formal Specification of a File Synchronizer: https://www.cis.upenn.edu/~bcpierce/papers/unisonspec.pdf find out that your secure file storage is not secure and is losing data? Can your unit tests ... 8
- 9. find out that going back and forth through a list of available voices can cause the app to play an incorrect voice? Can your unit tests ... my own child sure can! 9 This is a thing that we found out in our own app
- 10. 10 Twitter: https://twitter.com/cbirchall/status/1209156343904587779 All unit tests passing do not always guarantee correctness...
- 11. 11 And now for an academic example... public static <A> List<A> reverse(final List<A> original) { final int size = original.size(); final List<A> reversed = new ArrayList<>(size); // going from the last element to the first of the original list for (int i = size - 1; i >= 0; i--) { // thus adding the elements in the reversed order reversed.add(original.get(i)); } return reversed; } What are the properties of the code here? Github: https://git.io/JfDGP
- 12. 12 Lists reverse function example based tests Github: https://git.io/JfDGa
- 13. 13 Let’s build an example property... Github: https://git.io/JfDZS @Test public void reverse_reversed_equals_original() { // given: final List<String> tested = Arrays.asList("This", "is", "a", "test"); // when: final List<String> result = reverse(reverse(tested)); // then: assertEquals(tested, result); // Should we create more examples? }
- 14. 14 Let’s build a property based testing framework ... We will notice soon enough that something is missing
- 15. 15 Let’s build a property based testing framework ... private interface Property<A> { boolean check(A value); } private static <A> Property<List<A>> reverseOfReversedEqualsOriginal () { return original -> reverse(reverse(original)).equals(original); } Github: https://git.io/JfDZS
- 16. 16 Let’s build a property based testing framework ... private interface Generator<A> { A generate(Random seed); } // A very simple generator that produces lists: // - with size varying from 0 to 100 // - with elements being numbers from 0 to 100 as strings private Generator<List<String>> randomStringListsGenerator () { return seed -> { final int randomSize = randInt(seed, 0, 100); final List<String> randomizedList = new ArrayList<>(randomSize); for (int i = 0; i < randomSize; i++) { randomizedList .add(String.valueOf(randInt(seed, 0, 100))); } return randomizedList ; }; } Github: https://git.io/JfDZS
- 17. 17 Let’s build a property based testing framework ... private static <A> void quickCheck(final long seedValue, final int numberOfTries, final Property<A> property, final Generator<A> generator) { final Random seed = new Random(seedValue); for (int i = 0; i < numberOfTries; i++) { final A tested = generator.generate(seed); final boolean result = property.check(tested); if (!result) { final StringBuilder builder = new StringBuilder() .append("Property test failed") .append("nSeed value: ") .append(seedValue) .append("nExample data: ") .append(tested); throw new AssertionError(builder); } } } Github: https://git.io/JfDZS
- 18. 18 Let’s build a property based testing framework ... @Test public void custom_reverse_property_test() { final int numberOfTries = 1000; final long seedValue = new Random().nextLong(); final Property<List<String>> reverseProperty = reverseOfReversedEqualsOriginal(); final Generator<List<String>> generator = randomStringListsGenerator(); quickCheck(seedValue, numberOfTries, reverseProperty, generator); } Github: https://git.io/JfDZS
- 19. 19 We are lucky! The example only has 19 elements, what would happen if it had 1000 or 10000 elements? private static <A> Property<List<A>> reverseWrongProperty() { return original -> reverse(original).equals(original); } Github: https://git.io/JfDZS java.lang.AssertionError: Property test failed Seed value: -2569510089704470893 Example data: [40, 15, 20, 30, 36, 35, 55, 99, 89, 93, 67, 27, 31, 95, 26, 6, 84, 23, 92] How can we know our little framework is correct?
- 20. 20 What about a professional solution? Github: https://git.io/JfDZS @Property @Report(Reporting.GENERATED) public boolean broken_reverse_property (@ForAll List<?> original) { return Lists.reverse(original).equals(original); } |-------------------jqwik------------------- tries = 1 | # of calls to property checks = 1 | # of not rejected calls generation-mode = RANDOMIZED | parameters are randomly generated after-failure = PREVIOUS_SEED | use the previous seed seed = -1616208483702989146 | random seed to reproduce generated values sample = [[Any[0], Any[1]]] original-sample = [[Any[64], Any[226], Any[319], Any[71], Any[351], Any[137], Any[9], Any[262], Any[239], Any[485], Any[23], Any[265], Any[108], Any[348], Any[202], Any[365], Any[147], Any[347], Any[133]]] It also found a 19 element example but managed to shrink it to 2
- 21. 21 Poll: Our framework was missing ... A. a way to shrink the example to smallest possible size B. a way to shrink the values used by the example C. pretty printing of the output D. all of the above E. nothing, It was perfect! F. I bet there is something more!
- 22. 22 Our framework was missing ...
- 23. Are there well established categories of properties? 23 Before we leave the realm of academic examples... and the Math behind them is minimal! I promise.
- 25. 25 Property Category: Encode/Decode Twitter: https://github.com/btlines/pbdirect
- 26. 26 Property Category: Encode/Decode Twitter: https://github.com/btlines/pbdirect
- 28. 28 Property Category: Invariants size of the collection: some operations on collections do not change their size but only modify the order of elements or their state like sorting, mapping content of the collection: some operations on collections do not change their content (do not remove or add elements), it’s possible to find all elements but just in a different order order of elements: some operations change the characteristics of elements but do not change their order domain specific: premium users will always have a special discount regardless of the amount of items on their shopping cart. We can always find properties of our model that do not changed regardless of operations applied.
- 30. 30 Property Category: Idempotence absolute value of an integer: abs(abs(x)) = abs(x) floor, ceiling functions: ceil(ceil(x)) = ceil(x) database search: searching for an element in a database multiple times should give us the same results each time. This only holds in a case where there are no updates in between. sorting: sorting a collection multiple times should give us the same result, sorting an already sorted collection should give us the same result, the same applies for filtering, searching for distinct elements etc. crosswalk button: pressing a crosswalk button multiple times will not change the end result of the lights changing from red to green.
- 31. 31 Property Category: Different paths lead to the same destination
- 32. 32 Property Category: Different paths lead to the same destination commutative property of addition: x + y = y + x, for all x, y ∈ R putting socks on your feet: does it matter in which order we will put our socks on? If the thing that we would like end up with are both feet in socks, then it does not matter if we start with left or right foot! applying discount code to the shopping cart: does it matter if we apply the discount code to an empty cart (ofc if the API allows to start shopping like this) before we add the items? The result should be the same for when we apply the discount to already filled shopping cart. putting ingredients on your pizza: does it matter if we start with pineapple or ham if we want to have great pizza? I know I know this example is controversial! filtering and sorting the results: the end result should be the same if we start from sorting and then filter out the unwanted results. Let’s not focus on the issue of optimization
- 33. 33 Property Category: Test Oracle
- 34. 34 Property Category: Test Oracle result should stay the same: code after refactoring should give the same results as the original one for the same input. This can be applied for any kind of code not only examples like sorting or filtering. If we are reworking some UI elements to support new cases it should behave the same like the old one for all old cases. the new algorithm should not have worse performance then the old one: if the refactored code should use less memory, use less hard drive space, perform less operations, in other words just be better in some way or at least not worse, then comparing old with new is the way to go. when writing a property is not obvious: instead of finding a property of the code like encode/decode or different paths same result, we can base our tests on the already existing code and its results.
- 35. OK. Everything is nice but I am not writing sorting for my day job! 35 Now off to the real world... or reverse, or map, or json encoding for that matter
- 36. 36 1. open new database 2. put key1 and val1 3. close database 4. open database 5. delete key2 6. delete key1 7. close database 8. open database 9. delete key2 10. close database 11. open database 12. put key3 and val1 13. close database 14. open database 15. close database 16. open database 17. seek first Expected output? key3 Actual output? key1 Reappearing "ghost" key after 17 steps: https://github.com/google/leveldb/issues/50 A ghost story ... Would you come up with such an example?
- 37. 37 Reappearing "ghost" key after 17 steps: https://github.com/google/leveldb/issues/50 A ghost story - a rehaunting ... After a patch that was supposedly fixing the issue a new example was found this time made up from 33 steps. After almost a month it was finally fixed. Fixed bug in picking inputs for a level-0 compaction. When finding overlapping files, the covered range may expand as files are added to the input set. We now correctly expand the range when this happens instead of continuing to use the old range. This change fixes a bug related to deleted keys showing up incorrectly after a compaction.
- 39. 39 Stateful testing... Model - describes the current expected state of the system. Often we know what to expect from the code, it does not matter how the code gets to that result. Commands - represent what the system under test should do, command generation needs to take into account the current model state. Validation - way to check the current model state and compare it with the system under test state. SUT - system under test, stateful in it’s nature
- 40. 40 A stateful representation of a shopping cart ... public class ShoppingCart { public Quantity get(Product product) public List<Product> getAllProducts () public Quantity add(Product product, Quantity quantity) public Quantity remove(Product product, Quantity quantity) public void clear() public void setDiscount(DiscountCode code) public void clearDiscount() public Price getTotalPrice() public Quantity getTotalQuantity () }
- 41. 41 A simplified shopping cart model ... public class ShoppingCartModel { private List<ShoppingCartModelElement> products = new LinkedList<>(); private int discount = 0; //% private static class ShoppingCartModelElement { public String product; public int price; public int quantity; } }
- 42. 42 A simplified representation of a command in our system ... public interface Command<M, S> { boolean preconditions(final M model); void execute(final M model, final S sut); void postconditions (final M model, final S sut); }
- 43. 43 An implementation of one of the available commands ... public class AddProductCommand implements Command<ShoppingCartModel , ShoppingCart > { public void execute(final ShoppingCartModel model, final ShoppingCart sut) { model.addProduct( product .getName(), quantity .getValue(), product .getPrice().getValue() ); sut.add( product, quantity); } public void postconditions (final ShoppingCartModel model, final ShoppingCart sut) { Assertions.assertEquals( model.getQuantity( product.getName()), sut.get( product).getValue() ); }
- 44. 44 Available commands for our property test - AddProductCommand - RemoveProductCommand - GetProductCommand - GetAllProductsCommand - ClearProductCommand - ClearCommand - ClearProductCommand - SetDiscountCommand - ClearDiscountCommand - GetTotalPriceCommand - GetTotalQuantityCommand Now we need to create random sequences of commands that will stress the implementation of our stateful component.
- 45. 45 Testing properties of a shopping cart ... Let’s introduce a bug in the shopping cart: - if the shopping carts has 3 or more elements - it will NOT add any more elements Run failed after following actions: AddProductCommand{ product=Product{name='AAA', price=Price(3)}, quantity=Quantity(1)} AddProductCommand{ product=Product{name='AAa', price=Price(3)}, quantity=Quantity(1)} AddProductCommand{ product=Product{name='ABA', price=Price(3)}, quantity=Quantity(1)} AddProductCommand{ product=Product{name='AAB', price=Price(3)}, quantity=Quantity(1)}
- 46. 46 Testing properties of a shopping cart ... original-sample = [ActionSequence[FAILED]: [SetDiscountCommand{discount=Discount(58)}, GetTotalQuantityCommand{}, ClearCommand{}, SetDiscountCommand{discount=Discount(100)}, SetDiscountCommand{discount=Discount(58)}, GetTotalQuantityCommand{}, GetTotalPriceCommand{}, ClearDiscountCommand{}, SetDiscountCommand{discount=Discount(18)}, GetTotalQuantityCommand{}, GetTotalPriceCommand{}, ClearCommand{}, GetTotalQuantityCommand{}, AddProductCommand{ product=Product{name='ZzZaWB', price=Price(6)}, quantity=Quantity(60)}, GetTotalPriceCommand{}, AddProductCommand{ product=Product{name='zEfaZZga', price=Price(8)}, quantity=Quantity(74)}, GetTotalPriceCommand{}, ClearDiscountCommand{}, AddProductCommand{ product=Product{name='faGZQAGAQ', price=Price(9)}, quantity=Quantity(154)}, GetTotalQuantityCommand{}, AddProductCommand{ product=Product{name='ZVdJaj', price=Price(6)}, quantity=Quantity(66)}, SetDiscountCommand{discount=Discount(32)}, GetTotalQuantityCommand{}, GetTotalQuantityCommand{}, GetTotalQuantityCommand{}, AddProductCommand{ product=Product{name='AAzZza', price=Price(6)}, quantity=Quantity(5)}]]
- 48. 48 Further reading: (book) PropEr Testing https://propertesting.com/toc.html (documentation) QuickCheck: https://hackage.haskell.org/package/QuickCheck (article) Introduction to PBT: https://fsharpforfunandprofit.com/posts/property-based-testing/ (documentation) jqwik: https://jqwik.net/ (presentation) Stateful PBT: https://www.youtube.com/watch?v=owHmYA52SIM (presentation) Don’t Write Tests: https://www.youtube.com/watch?v=hXnS_Xjwk2Y (presentation) Testing the Hard Stuff: https://www.youtube.com/watch?v=zi0rHwfiX1Q (examples): https://github.com/artificialrevelations/property-based-testing-workshop