Escaping Automated Test Hell - One Year Later
1 like1,825 views
The document discusses efforts made by Wojciech Seliga and his team to improve the performance and reliability of their automated testing framework over the past year. Key changes involved restructuring the test codebase, implementing parallel test execution, and auditing existing tests to reduce flakiness and improve feedback loops. The overall improvements led to significant reductions in build times and better morale among developers, although challenges with stability and scalability remain.
Escaping Automated Test Hell - One Year Later
- 1. Main sponsor Escaping Automated Test Hell One year later... Wojciech Seliga
- 2. About me • Coding for 30 years • Agile Practices (inc. TDD) since 2003 • Dev Nerd, Tech Leader, Agile Coach, Speaker • 5+ years with Atlassian (JIRA Development Team Lead) • Spartez Co-founder
- 3. Year ago - recap
- 4. 18 000 tests on all levels Very slow and fragile feedback loop
- 5. Serious performance and reliability issues
- 6. Feedback Test Speed Quality `
- 7. Respect Restructure Design Share Prune Test Code is Not Trash Refactor Maintain Review Discuss
- 11. Optimum Balance Isolation Speed Coverage
- 12. Optimum Balance Isolation Speed Coverage Level
- 13. Optimum Balance Isolation Speed Coverage Level Access
- 14. Optimum Balance Isolation Speed Coverage Level Access Effort
- 15. Dangerous to temper with
- 16. Dangerous to temper with Quality / Determinism
- 17. Dangerous to temper with Quality / Determinism Maintainability
- 18. Splitting codebase is key aspect of short test feedback loop
- 19. Now
- 21. Shades of Red
- 23. Build Tiers and Policy Tier A1 - green soon after all commits unit tests and functional* tests Tier A2 - green at the end of the day WebDriver and bundled plugins tests Tier A3 - green at the end of the iteration supported platforms tests, compatibility tests
- 25. Training • assertThat over assertTrue/False and assertEquals • avoiding races - Atlassian Selenium with its TimedElement • Unit tests over functional tests • Brownbags, blogs, code reviews
- 26. Quality
- 27. Re-run failed tests and see if they pass Automatic Flakiness Detection Quarantine
- 29. SlowMo - expose races
- 30. Selenium 1
- 31. Selenium 1
- 32. Selenium ditching Sky did not fall in
- 33. Ditching - benefits • Freed build agents - better system throughput • Boosted morale • Gazillion of developer hours saved • Money saved on infrastructure
- 34. Ditching - due diligence • conducting the audit - analysis of the coverage we lost • determining which tests needs to rewritten (e.g. security related) • rewriting the tests
- 35. Flaky Browser-based Tests Races between test code and asynchronous page logic Playing with "loading" CSS class does not really help
- 36. Races Removal with Tracing // in the browser: function mySearchClickHandler() { doSomeXhr().always(function() { // This executes when the XHR has completed (either success or failure) JIRA.trace("search.completed"); }); } // In production code JIRA.trace is a no-op // in my page object: @Inject TraceContext traceContext; public SearchResults doASearch() { Tracer snapshot = traceContext.checkpoint(); getSearchButton().click(); // causes mySearchClickHandler to be invoked // This waits until the "search.completed" // event has been emitted, *after* previous snapshot traceContext.waitFor(snapshot, "search.completed"); return pageBinder.bind(SearchResults.class); }
- 37. Speed
- 38. Speed Can we halve our build times?
- 39. Parallel Execution - Theory Batches Start of Build End of Build
- 40. Parallel Execution Batches Start of Build End of Build
- 41. Parallel Execution - Agent availability Reality Bites Batches Start of Build End of Build
- 42. Dynamic Test Execution Dispatch - Hallelujah
- 43. Dynamic Test Execution Dispatch - Hallelujah
- 44. "You can't manage what you can't measure." W. Edwards Deming
- 45. If y ou you bel ieve "Youare manage what can't just doo i i you can't measure."n me t d. W. Edwards Deming
- 46. You can't improve something if you can't measure it
- 47. You can't improve something if you can't measure it Profiler, Build statistics, Logs, statsd → Graphite
- 48. Compilation Packaging Executing Tests Anatomy of Build*
- 49. Fetching Dependencies Compilation Packaging Executing Tests Anatomy of Build*
- 50. Fetching Dependencies Compilation Packaging Executing Tests Anatomy of Build* *Any resemblance to maven build is entirely accidental
- 51. Fetching Dependencies Compilation Packaging SCM Update Executing Tests Anatomy of Build* *Any resemblance to maven build is entirely accidental
- 52. Agent Availability/Setup Fetching Dependencies Compilation Packaging SCM Update Executing Tests Anatomy of Build* *Any resemblance to maven build is entirely accidental
- 53. Agent Availability/Setup Fetching Dependencies Compilation Packaging Publishing Results SCM Update Executing Tests Anatomy of Build* *Any resemblance to maven build is entirely accidental
- 54. Compilation (7min) JIRA Unit Tests Build
- 55. Compilation (7min) Packaging (0min) JIRA Unit Tests Build
- 56. Compilation (7min) Packaging (0min) Executing Tests (7min) JIRA Unit Tests Build
- 57. Compilation (7min) Publishing Results (1min) Packaging (0min) Executing Tests (7min) JIRA Unit Tests Build
- 58. Compilation (7min) Publishing Results (1min) Packaging (0min) Executing Tests (7min) Fetching Dependencies (1.5min) JIRA Unit Tests Build
- 59. Compilation (7min) Publishing Results (1min) SCM Update (2min) Packaging (0min) Executing Tests (7min) Fetching Dependencies (1.5min) JIRA Unit Tests Build
- 60. Agent Availability/Setup (mean 10min) Compilation (7min) Publishing Results (1min) SCM Update (2min) Packaging (0min) Executing Tests (7min) Fetching Dependencies (1.5min) JIRA Unit Tests Build
- 61. Decreasing Test Execution Time to ZERRO alone would not let us achieve our goal!
- 62. Agent Availability/Setup • starved builds due to busy agents building very long builds • time synchronization issue - NTPD problem
- 63. SCM Update - Checkout time • Proximity of SCM repo • shallow git clones are not so fast and lightweight + generating extra git server CPU load • git clone per agent/plan + git pull + git clone per build (hard links!) • Stash was thankful (queue)
- 64. SCM Update - Checkout time • Proximity of SCM repo • shallow git clones are not so fast and lightweight + generating extra git server CPU load • git clone per agent/plan + git pull + git clone per build (hard links!) • Stash was thankful (queue) 2 min → 5 seconds
- 66. Fetching Dependencies • Fix Predator • Sandboxing/isolation agent trade-off: rm -rf $HOME/.m2/repository/com/atlassian/* into find $HOME/.m2/repository/com/atlassian/ -name “*SNAPSHOT*” | xargs rm • Network hardware failure found (dropping packets)
- 67. Fetching Dependencies • Fix Predator • Sandboxing/isolation agent trade-off: rm -rf $HOME/.m2/repository/com/atlassian/* into find $HOME/.m2/repository/com/atlassian/ -name “*SNAPSHOT*” | xargs rm • Network hardware failure found (dropping packets) 1.5 min → 10 seconds
- 68. Compilation • Restructuring multi-pom maven project and dependencies • Maven 3 parallel compilation FTW -T 1.5C *optimal factor thanks to scientific trial and error research
- 69. Compilation • Restructuring multi-pom maven project and dependencies • Maven 3 parallel compilation FTW -T 1.5C *optimal factor thanks to scientific trial and error research 7 min → 1 min
- 70. Unit Test Execution • Splitting unit tests into 2 buckets: good and legacy (much longer) • Maven 3 parallel test execution (-T 1.5C) 3000 poor tests 11000 good tests (5min) (1.5min)
- 71. Unit Test Execution • Splitting unit tests into 2 buckets: good and legacy (much longer) • Maven 3 parallel test execution (-T 1.5C) 3000 poor tests 11000 good tests (5min) (1.5min) 7 min → 5 min
- 72. Functional Tests • Selenium 1 removal did help • Faster reset/restore (avoid unnecessary stuff, intercepting SQL operations for debug purposes - building stacktraces is costly) • Restoring via Backdoor REST API • Using REST API for common setup/ teardown operations
- 73. Functional Tests
- 74. Publishing Results • Server log allocation per test → using now Backdoor REST API (was Selenium) • Bamboo DB performance degradation for rich build history - to be addressed
- 75. Publishing Results • Server log allocation per test → using now Backdoor REST API (was Selenium) • Bamboo DB performance degradation for rich build history - to be addressed 1 min → 40 s
- 76. Unexpected Problem • Stability Issues with our CI server • The bottleneck changed from I/O to CPU • Too many agents per physical machine
- 77. Compilation (1min) JIRA Unit Tests Build Improved
- 78. Compilation (1min) Packaging (0min) JIRA Unit Tests Build Improved
- 79. Compilation (1min) Packaging (0min) Executing Tests (5min) JIRA Unit Tests Build Improved
- 80. Compilation (1min) Packaging (0min) Publishing Results (40sec) Executing Tests (5min) JIRA Unit Tests Build Improved
- 81. Compilation (1min) Packaging (0min) Publishing Results (40sec) Executing Tests (5min) Fetching Dependencies (10sec) JIRA Unit Tests Build Improved
- 82. Compilation (1min) Packaging (0min) Publishing Results (40sec) SCM Update (5sec) Executing Tests (5min) Fetching Dependencies (10sec) JIRA Unit Tests Build Improved
- 83. Agent Availability/Setup (3min)* Compilation (1min) Packaging (0min) Publishing Results (40sec) SCM Update (5sec) Executing Tests (5min) Fetching Dependencies (10sec) JIRA Unit Tests Build Improved
- 84. Improvements Summary Tests Before After Improvement % Unit tests 29 min 17 min 41% Functional tests 56 min 34 min 39% WebDriver tests 39 min 21 min 46% Overall 124 min 72 min 42% * Additional ca. 5% improvement expected once new git clone strategy is consistently rolled-out everywhere
- 88. But that's still bad We want CI feedback loop in a few minutes maximum
- 90. Resistance against splitting The last attempt: Magic Machine Decide with high confidence (e.g. > 95%) which subset of tests to run basing on the committed changes
- 91. Magic Machine • Looking at Bamboo history (analysing correlation between changes and failures) • Matching: package test/prod code and transitive imports • Code instrumentation (Clover, Emma, AspectJ) • Run most often failing first
- 92. Inevitable Split - Fears • Organizational concerns - understanding, managing, integrating, releasing • Mindset change - if something worked for 10 years why to change it? • We damned ourselves with big buckets for all tests - where do they belong to?
- 93. Magic Machine strikes back With heavy use of brain, common sense and expert judgement
- 94. Splitting code base • Step 0 - JIRA Importers Plugin (3 years ago) • Step 1- New Issue View and Navigator JIRA 6 .0
- 95. We are still escaping hell. Hell sucks in your soul.
- 96. Conclusions • Visibility and problem awareness help • Maintaing huge testbed is difficult and costly • Measure the problem • No prejudice - no sacred cows • Automated tests are not one-off investment, it's a continuous journey • Performance is a damn important feature
- 97. Do you want to help? We are hiring in Gdańsk • Principal Java Developer • Development Team Lead • Java and Scala Developers • UX Designer • Front-End Developer • QA Engineer Visit us at the booth or apply at http://www.atlassian.com/company/careers
- 98. Images - Credits • Turtle - by Jonathan Zander, CC-BY-SA-3.0 • Loading - by MatthewJ13, CC-SA-3.0 • Magic Potion - by Koolmann1, CC-BY-SA-2.0 • Merlin Tool - by By L. Mahin, CC-BY-SA-3.0 • Choose Pills - by *rockysprings, CC-BY-SA-3.0
- 99. Thank You!