Wix Automation - Test State Pattern - 11.9.16
0 likes430 views
The document discusses the challenges of end-to-end (e2e) testing, particularly issues with test fragility and false positives. It proposes a solution that involves splitting tests into two run groups to better manage test stability and results. Additionally, it outlines a framework to determine the stability of tests and adapt their state based on performance, enhancing overall test management and automation.
Wix Automation - Test State Pattern - 11.9.16
- 1. Test State Pattern Roi Ashkenazi Elad Benitah roia@wix.com eladb@wix.com WixAutomation
- 3. 01 / The Problem
- 4. E2E tests have more breaking points.
- 5. New E2E tests are even more fragile.
- 6. Pass locally, fail remotely ● Timing issues ● Remote connection issues ● Bad luck
- 7. We try to avoid … False positives Causes lack of trust Noise Analysis becomes difficult Shame
- 9. 02 / Solution: @BetaTest
- 10. @BetaTest @BetaTest @BetaTest Beta @Test @Test @Test Release Split the tests into 2 “run groups”
- 11. @BetaTest(owner = "tzipi") public void newTest() { … }
- 12. @BetaTest @BetaTest @BetaTest Beta @Test @Test @Test Release Results Beta run group reports results to server
- 13. @BetaTest @BetaTest @BetaTest Beta @Test @Test @Test Release Run groups in different configurations Results
- 14. @Override public void beforeTest(Description description) { if (isBetaTestInReleaseRunMode(description) || isNonBetaTestInBetaRunMode(description)) { TestUtils.ignoreTest(); } }
- 16. How do we find the stable ones?
- 17. Results Tests that pass X times in a row, with maximum Y failures. How do we find the stable ones?
- 18. So, for each stable test notify the owner to replace @BetaTest with @Test
- 20. 03 / We Can Do Better
- 21. @Test @Test @Test Beta @Test @Test @Test Release No need for special @BetaTest annotation Results
- 22. Beta Each test begins with a Beta state
- 23. Beta Remains in Beta state for some time
- 24. Beta And (hopefully) changes to stable Stable pass X times
- 25. @Override public void beforeTest(Description description) { TestState state = getTestState(description); if (isBetaTestInReleaseRunMode(description, state) || isNonBetaTestInBetaRunMode(description, state)) { TestUtils.ignoreTest(); } } Run tests according to their state
- 26. What happens if a test keeps failing? Beta fail X times
- 27. What happens if a test keeps failing? Beta Failed Beta fail X times
- 28. Failed beta - manual fix Beta Failed Beta fail X times manual fix
- 29. Beta What happens if a stable test starts to “act weird”? Stable pass X times
- 30. Beta Stable FlakyFlak y Become unstable pass X times pass X times What happens if a stable test starts to “act weird”?
- 31. Beta Full pattern Failed Beta fail X times Stable FlakyFlak y Become unstable pass X times pass X times manual fix
- 32. Stable automation is possible! Results @Test @Test @Test Beta @Test @Test @Test Release @Test @Test @Test Flak y Failed Beta
- 34. API Core Web DB Web View Java Client Test Observer - testStarted - testIgnored - testFailed - testPassed
- 35. Test - Name* - State** - Creation date - Update date - Annotations Object modeling Run Result - Test Name - Passed / failed - Creation date - Configuration*** 1 n
- 36. 05 / Live Demo
- 38. This is where you are going to present your final words. This slide is not meant to have a lot of text.Thank You!