Escaping Test Hell - Our Journey - XPDays Ukraine 2013
2 likes1,934 views
The document outlines the experiences of Wojciech Seliga regarding improvements in automated testing practices over a span of ten years, detailing challenges faced with a large number of fragile tests. It emphasizes the importance of reducing build times, enhancing feedback loops, and restructuring tests, which led to significant time savings and improved morale. The presentation concludes with a call for visibility in testing issues and ongoing commitment to test quality as essential for software development.
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
- 1. Automated Test Hell Wojciech Seliga wojciech.seliga@spartez.com @wseliga Our Journey
- 2. About me • Coding for 30 years, now only in "free time" • Agile Practices (inc.TDD) since 2003 • Dev Nerd,Tech Leader,Agile Coach, Speaker, PHB • 6 years with Atlassian (JIRA Dev Manager) • Spartez Co-founder & CEO
- 4. 1.5 year ago
- 5. Almost 10 years of accumulating garbage automatic tests
- 6. About 20 000 tests on all levels of abstraction
- 7. Very slow (even hours) and fragile feedback loop
- 8. Serious performance and reliability issues
- 9. Dispirited devs accepting RED as a norm
- 11. Test Code is Not Trash Design Maintain Refactor Share Review Prune Respect Discuss Restructure
- 12. Test Pyramid Unit Tests (including QUnit) REST / HTML Tests Selenium
- 13. Optimum Balance Isolation Speed Coverage Level Access Effort
- 14. Dangerous to temper with MaintainabilityQuality / Determinism
- 15. Now
- 16. People - Motivation Making GREEN the norm
- 17. Shades of Red Shades of Green
- 19. Build Tiers and Policy Tier A1 - green soon after all commits Tier A2 - green at the end of the day Tier A3 - green at the end of the iteration unit tests and functional* tests WebDriver and bundled plugins tests supported platforms tests, compatibility tests
- 21. Extensive Training • assertThat over assertTrue/False and assertEquals • avoiding races - Atlassian Selenium with its TimedElement • Favouring unit tests over functional tests • Promoting Page Objects • Brownbags, blogs, code reviews
- 22. Quality
- 23. Automatic Flakiness Detection Quarantine Re-run failed tests and see if they pass
- 25. SlowMo - expose races
- 26. Selenium 1
- 27. Selenium ditching Sky did not fall in
- 28. Ditching - benefits • Freed build agents - better system throughput • Boosted morale • Gazillion of developer hours saved • Money saved on infrastructure
- 29. Ditching - due diligence • conducting the audit - analysis of the coverage we lost • determining which tests needs to rewritten (e.g. security related) • rewriting some of the tests (good job for new hires + a senior mentor)
- 30. Flaky Browser-based Tests Races between test code and asynchronous page logic Playing with "loading" CSS class does not really help
- 31. Races Removal with Tracing // in the browser: function mySearchClickHandler() { doSomeXhr().always(function() { // This executes when the XHR has completed (either success or failure) JIRA.trace("search.completed"); }); } // In production code JIRA.trace is a no-op // in my page object: @Inject TraceContext traceContext; public SearchResults doASearch() { Tracer snapshot = traceContext.checkpoint(); getSearchButton().click(); // causes mySearchClickHandler to be invoked // This waits until the "search.completed" // event has been emitted, *after* previous snapshot traceContext.waitFor(snapshot, "search.completed"); return pageBinder.bind(SearchResults.class); }
- 32. Can we halve our build times? Speed
- 33. Parallel Execution - Theory End of Build Batches Start of Build Time
- 34. Parallel Execution End of Build Batches Start of Build Time
- 35. Parallel Execution - Reality Bites End of Build Batches Start of Build Agent availability Time
- 36. Dynamic Test Execution Dispatch - Hallelujah
- 37. "You can't manage what you can't measure." not by W. Edwards Deming If you believe just in it you are doomed.
- 38. You can't improve something if you can't measure it Profiler, Build statistics, Logs, statsd → Graphite
- 39. Anatomy of Build* Compilation Packaging Executing Tests Fetching Dependencies *Any resemblance to maven build is entirely accidental SCM Update Agent Availability/Setup Publishing Results
- 40. JIRA Unit Tests Build Compilation (7min) Packaging (0min) Executing Tests (7min) Fetching Dependencies (1.5min) SCM Update (2min) Agent Availability/Setup (mean 10min) Publishing Results (1min)
- 41. Decreasing Test Execution Time to ZERRO alone would not let us achieve our goal!
- 42. Agent Availability/Setup • starved builds due to busy agents building very long builds • time synchronization issue - NTPD problem Fixes applied
- 43. • Proximity of SCM repo • shallow git clones are not so fast and lightweight + generating extra git server CPU load • git clone per agent/plan + git pull + git clone per build (hard links!) • Atlassian Stash was thankful (queue) SCM Update - Checkout time 2 min → 5 seconds Trade disk space for speed
- 45. • Fix Predator • Sandboxing/isolation agent trade-off: rm -rf $HOME/.m2/repository/com/atlassian/* into find $HOME/.m2/repository/com/atlassian/ -name “*SNAPSHOT*” | xargs rm • Network hardware failure found (dropping packets) Fetching Dependencies 1.5 min → 10 seconds
- 46. Compilation • Restructuring multi-pom maven project and dependencies • Maven 3 parallel compilation FTW -T 1.5C *optimal factor thanks to scientific trial and error research 7 min → 1 min
- 47. Unit Test Execution • Splitting unit tests into 2 buckets: good and legacy (much longer) • Maven 3 parallel test execution (-T 1.5C) 7 min → 5 min 3000 poor tests (5min) 11000 good tests (1.5min)
- 48. Functional Tests • Selenium 1 removal did help • Faster reset/restore (avoid unnecessary stuff, intercepting SQL operations for debug purposes - building stacktraces is costly) • Restoring via Backdoor REST API • Using REST API for common setup/ teardown operations
- 49. Functional Tests We like this trend
- 50. Publishing Results • Server log allocation per test → using now Backdoor REST API (was Selenium) • Bamboo DB performance degradation for rich build history - to be addressed 1 min → 40 s
- 51. Unexpected Problem • Stability issues with our CI server • The bottleneck changed from I/O to CPU • Too many agents per physical machine
- 52. JIRA Unit Tests Build Improved Compilation (1min) Packaging (0min) Executing Tests (5min) Fetching Dependencies (10sec) SCM Update (5sec) Agent Availability/Setup (3min)* Publishing Results (40sec)
- 53. Improvements Summary Tests Before After Improvement % Unit tests 29 min 17 min 41% Functional tests 56 min 34 min 39% WebDriver tests 39 min 21 min 46% Overall 124 min 72 min 42% * Additional ca. 5% improvement expected once new git clone strategy is consistently rolled-out everywhere
- 55. But that's still bad We want CI feedback loop in a few minutes maximum
- 57. Codebase Split - Problems • Organizational concerns - understanding, managing, integrating, releasing • Mindset change - if something worked for 10 years why to change it? • We damned ourselves with big buckets for all tests - where do they belong to?
- 58. Splitting code base • Step 0 - JIRA Importers Plugin (3.5 years ago) • Step 1- New IssueView and Navigator JIRA 6.0
- 59. We are still escaping hell. Hell sucks in your soul.
- 60. Conclusions • Visibility and problem awareness help • Maintaing huge testbed is difficult and costly • Measure the problem, measure improvements • No prejudice - no sacred cows • Automated tests are not one-off investment, it's a continuous journey • Performance is a damn important feature
- 62. Interested in such stuff? Talk to me at the conference or visit http://www.spartez.com/careers We are hiring in Gdańsk
- 63. • Turtle - by Jonathan Zander, CC-BY-SA-3.0 • Loading - by MatthewJ13, CC-SA-3.0 • Magic Potion - by Koolmann1, CC-BY-SA-2.0 • Merlin Tool - by By L. Mahin, CC-BY-SA-3.0 • Choose Pills - by *rockysprings, CC-BY-SA-3.0 • Flashing Red Light - bt Chris Phan, CC BY 2.0 Images - Credits
- 64. ThankYou! Tweet your feedback at @wseliga