Wix Automation - Automation Manager
0 likes382 views
The document discusses the challenges of managing a rapidly growing number of tests using TeamCity as a CI tool, highlighting issues such as flakiness and lack of effective test management. It emphasizes the need for a central management system to enhance automation quality, decision-making, and development velocity through better analysis and control of test states. The proposed solution, an automation manager, aims to gather test run data and improve test stability and risk analysis.
Wix Automation - Automation Manager
- 1. 1 Wix Automation Automation Manager Stas Slutsker
- 3. 3 What We Have TeamCity (CI) as test executor TeamCity (CI) as test executor
- 5. 5 It Works, But… • Controlling rapidly growing tests amount, more & better info about them • Dealing with flakiness in efficient way, technically • Reduce trust and increase annoyance • Poison. How to avoid them and progress faster? -> Move fast and break nothing
- 6. 6 TeamCity is NOT test management tool! It Works, But…
- 8. 8 • Analyzing tests per configuration, history & statistics for cross-section demands • Pursue stability • Risk analysis, better decision making So, What’s Missing?
- 9. 9 We need central management of tests for smarter automation • Progress faster, reaching our release goals • Better development velocity • Quality of our automation service, as a team Conclusion
- 11. 11 Automation Manager Goals • Gathering information about test runs • Statistics & history, per cross-section demands • Disabling & approving new or suspected tests, controlling tests states Smarter Automation
- 13. 13 States & Stability • Each test has a state • State is changed according to test behavior • Stable state must be earned! StableBeta (new / flaky) Disabled
- 14. 14 Beta Tests
- 15. 15 Statistics
- 16. 16 Dashboards according to different needs Statistics
- 17. 17 Statistics – Ignored Tests
- 18. 18 Statistics – Test Runs
- 20. 20 Risk analysis & behavior patterns • Skipping new or non-stable tests • Discovering non-valuable tests • Vulnerable product areas (error prune) Better Decision Making
- 22. 22 Project Structure API Core Web DB Client RESTAPI Test Observer @ Before @ Failure @ Success @ Skip GAE
- 24. 24 Objects Modeling RunConfiguration - browser - experiments - sanity - environment - platform - configurationName - numOfThreads Test - date - name - state - appearance RunResult - testName - buildId - runConfiguration - passStatus