Eradicate Flaky Tests
0 likes473 views
The document discusses challenges in end-to-end test automation, particularly issues with flaky tests caused by external dependencies and varying execution environments. It proposes solutions such as reducing the number of UI tests, employing intelligent virtualization to eliminate dependencies, and using visual assertions for more reliable testing outcomes. Additionally, it highlights the use of Specmatic for contract-driven development and Applitools for visual testing to improve the accuracy and efficiency of automated tests.
Eradicate Flaky Tests
- 1. Eradicate Flaky Tests @BagmarAnand Anand Bagmar Software Quality Evangelist
- 4. @BagmarAnand Internet Internal B2C App Product DB DBMS B2B App Caching System Elastic Search Authentication Gateway Service 3 Service 4 Service 7 Service 5 Service 6 Admin User DB Outgoing Notification Onboarding Event Consumer Order DB Reporting System Admin Portal Support Msg Q Direct API Service 2 Notification Service 1 Event Processor Service 8 Payment Gateway Credit System Master Product Warehouse Royalty system Offers …. …. ….
- 5. Approach for API Workflow & E2E Test Execution @BagmarAnand
- 6. API workflow Tests E2E Functional UI Tests @BagmarAnand Internet Internal B2C App Product DB DBMS B2B App Caching System Elastic Search Authentication Gateway Service 3 Service 4 Service 7 Service 5 Service 6 Admin User DB Outgoing Notification Onboarding Event Consumer Order DB Reporting System Admin Portal Support Msg Q Direct API Service 2 Notification Service 1 Event Processor Service 8 Payment Gateway Credit System Master Product Warehouse Royalty system Offers …. …. ….
- 7. • Long running scenarios • Slow feedback Challenges of E2E Test Automation @BagmarAnand
- 8. • Limitations of automatable scenarios • Error case / Edge case handling from dependent systems • Triggers from external systems Challenges of E2E Test Automation @BagmarAnand
- 9. • Pain of Cross-Browser / Cross-Device execution • Slow, special handling for browsers/devices, infrastructure cost / maintenance Challenges of E2E Test Automation @BagmarAnand
- 10. • Flaky Tests Challenges of E2E Test Automation @BagmarAnand
- 11. Why are Tests Flaky? @BagmarAnand
- 12. A few solutions to reduce flakiness in automated tests @BagmarAnand
- 13. • Reduce number of UI tests • Remove external dependencies via Intelligent Virtualization • Use Visual Assertions instead of Functional Assertions Reduce Flakiness in Automated Tests @BagmarAnand
- 14. #1 Reduce the number of UI Tests @BagmarAnand
- 16. #2 Intelligent Virtualization to Eradicate Dependencies @BagmarAnand
- 18. @BagmarAnand Internet Internal B2C App Product DB DBMS B2B App Caching System Elastic Search Authentication Gateway Service 3 Service 4 Service 7 Service 5 Service 6 Admin User DB Outgoing Notification Onboarding Event Consumer Order DB Reporting System Admin Portal Support Msg Q Direct API Service 2 Notification Service 1 Event Processor Service 8 Payment Gateway Credit System Master Product Warehouse Royalty system Offers …. …. ….
- 19. Approach for API Workflow & E2E Test Automation @BagmarAnand
- 20. API workflow Tests E2E Functional UI Tests @BagmarAnand Internet Internal B2C App Product DB DBMS B2B App Caching System Elastic Search Authentication Gateway Service 3 Service 4 Service 7 Service 5 Service 6 Admin User DB Outgoing Notification Onboarding Event Consumer Order DB Reporting System Admin Portal Support Msg Q Direct API Service 2 Notification Service 1 Event Processor Service 8 Payment Gateway Credit System Master Product Warehouse Royalty system Offers …. …. ….
- 22. @BagmarAnand Step #1: • Test calls Service 2 Step #2: • Service 2 calls the external service, which may (unexpectedly) take long time to respond Step #3: • Service 2 gets an error back from external service • Returns an error to the test Step #4: • Test fails because of external service • We may get incorrect error as well! Test Scenario
- 23. Challenge! Scenarios having external dependencies are flaky @BagmarAnand
- 24. Solution @BagmarAnand Stub it out using Specmatic!
- 26. • Inculcates Contract Driven Development • Supports OpenAPI Spec • Executable Contract spec • Defines http method, path, header format, payload format, query parameter format, form field payload format • Supports JSON, XML data formats • Supports Kafka • Dynamically generates and executes contract tests • Apt for service virtualization • Compares contracts to ascertain backward compatibility • Manages contracts through versioning • Integrates with CI/CD Specmatic – Contract Driven Development @BagmarAnand • Decouples the ”producers” and “consumers”, with confidence!
- 27. Use Specmatic as part of Test execution Stubbed using Specmatic @BagmarAnand Step #3: • Test calls Service 2 Step #4: • Service 2 calls the stubbed external service Step #2: • Test sets expectations on Specmatic for specific stubbed endpoint • Http POST call on Specmatic server with exact request & response as json payload Step #5: • If request matches set expectation, corresponding set response is returned in response • Service 2 processes response and responds to test appropriately Step #6: • Test asserts on the received response • Test passes Step #1. Specmatic as a Stub Server in your Test Environment • Specmatic Server is started in the Stub mode in the Environment • External services are stubbed out • Internal services point to Specmatic stubbed- endpoints instead of real external endpoints
- 28. Use Specmatic as part of Test execution @BagmarAnand Step #3: • Test calls Service 2 Step #4: • Service 2 calls the stubbed external service Step #2: • Test sets expectations on Specmatic for specific stubbed endpoint • Http POST call on Specmatic server with exact request & response as json payload Step #5: • If request matches set expectation, corresponding set response is returned in response • Service 2 processes response and responds to test appropriately Step #6: • Test asserts on the received response • Test passes Step #1. Specmatic as a Stub Server • Specmatic Server is started in the Stub mode in the Environment • External services are stubbed out • Internal services point to Specmatic stubbed- endpoints instead of real external endpoints Allows to simulate positive, negative, edge-case responses – allowing you to test your implementation logic
- 30. @BagmarAnand Application under test Login Get list of users Edit User Test Scenario Login 1 Get list of users 2 Internet Return dynamic list of users 3 4 Edit User 5 Verify User Details 6 Return dynamic list of users Test Scenario Example Scenario
- 31. Website with rest services in backend hosted on local machine Login Get list of users Edit User Test Scenario Functional & Visual Assertions using Applitools Login 1 Get list of users 3 Verify User Details 7 Edit User 6 4 5 Test Environment Return dynamic list of users Stubbed using Specmatic Static Random Dynamic Expectations @BagmarAnand 2. Test sets “dynamic expectation” in context of the scenario Example Scenario using Specmatic I n t e r n e t
- 32. @BagmarAnand Example Scenario using Specmatic Website with rest services in backend hosted on local machine Login Get list of users Edit User Test Scenario Functional & Visual Assertions using Applitools Login 1 Get list of users 3 Verify User Details 7 Edit User 6 4 5 Test Environment Return dynamic list of users Stubbed using Specmatic Static Random Dynamic Expectations 2. Test sets “dynamic expectation” in context of the scenario I n t e r n e t Return dynamic list of users Test Scenario
- 33. #3 Visual Assertions instead of Functional Assertions @BagmarAnand
- 35. Bugs still escape @BagmarAnand Traditional automated testing frameworks are not built for modern apps
- 36. @BagmarAnand
- 37. Bugs escape because our approach to testing is incorrect @BagmarAnand
- 42. @BagmarAnand Mundane Error prone Tedious Running against Time Approach to Testing is Incorrect
- 43. But, we are testing real software products @BagmarAnand
- 44. 45 @BagmarAnand Text, Images, Responsive-pages, Form-factors, User-experience Product has Context!
- 51. @BagmarAnand Computer Vision that replicates the human eye and brain
- 53. @BagmarAnand Visual AI detects bugs, not browser render diffs
- 54. @BagmarAnand Pixel comparisons waste time with false positives
- 55. Visual AI should work across all Platforms PDF, Web, Native Mobile @BagmarAnand
- 56. Perfection Across All Screens, Browsers and Viewports Tests run local - Applitools SDKs available for all major testing frameworks DOM & CSS sent to Applitools Ultrafast Grid Applitools Eyes Visual AI analysis Ultrafast Grid renders DOM snapshots in parallel Login to Eyes to view test results 58 . Firewall
- 60. • Reduce number of UI tests – Test Pyramid • Remove external dependencies via Intelligent Virtualization • Use Visual Assertions instead of Functional Assertions Make your Automation Intelligent! @BagmarAnand
- 61. • Move granular tests to lower layers of pyramid • Get faster feedback • Reduce flakiness / brittleness due to UI changes Reduce UI tests @BagmarAnand
- 62. • Test environment independent of external dependencies • Tests set dynamic expectations based on context of the test execution • Stubbed service should be able to give static responses / random responses Virtualize Dependencies at Runtime @BagmarAnand
- 63. • A single assertion for complete functional coverage • Bonus: validates the UI • Does not break when the UI changes • No coding skills required to maintain baselines • Validate UX at scale for all supported browsers Use Visual Assertions @BagmarAnand
- 64. Run once, validate everywhere @BagmarAnand
- 65. • Visual AI using Applitools • Getting Started with Visual Testing - https://testautomationu.applitools.com/automated- visual-testing-a-fast-path-to-test-automation-success/ • Modern Functional Test Automation Through Visual AI - https://testautomationu.applitools.com/modern-functional-testing/ • Intelligent Virtualization using Specmatic • https://specmatic.in/ • https://github.com/znsio/specmatic • Test Automation University - https://testautomationu.applitools.com/ • Test Pyramid - https://martinfowler.com/articles/practical-test-pyramid.html Resources @BagmarAnand