Building the Ideal CI-CD Pipeline_ Achieving Visual Perfection
0 likes51 views
The document discusses the limitations of traditional CI/CD testing pipelines in functional testing and proposes a solution through visual AI testing, which provides faster test creation, reduced maintenance, and improved accuracy. It highlights the benefits of visual AI over traditional methods, addressing issues such as flaky tests, manual testing requirements, and execution time across multiple browser environments. The ideal CI/CD pipeline is presented with a focus on tests that ensure full UI coverage and immediate feedback to streamline the development process.
Building the Ideal CI-CD Pipeline_ Achieving Visual Perfection
- 1. Brandon Murray Technical Customer Success Manager Building the Ideal CI/CD Pipeline: Achieving Visual Perfection
- 2. TRADITIONAL CI/CD TESTING PIPELINE Unit Testing Api Testing Functional Testing Performance Testing Exploratory Testing Manual Testing Production Monitoring Development QA Production
- 3. Problems with traditional functional testing 1. Time to Develop/ Maintenance 2. Does not validate the UI 3. Requires manual testing 4. Flaky tests 5. Execution time (compounded by multiple browser environments)
- 4. Ideal World ● Faster test creation ● Low test maintenance ● Full UI coverage ● Reduce manual testing ● More stable tests ● Faster execution ● Allow shift left ● Immediate PR feedback ● Release faster
- 5. Unit Testing Api Testing Visual AI Functional Testing Functional Testing Performance Testing Exploratory Testing Manual Testing (drastically reduced or eliminated) Production Monitoring Development QA Production IDEAL CI/CD TESTING PIPELINE
- 6. How can we address traditional function testing limitations?
- 7. 1. Time to Develop/ Maintenance 2. Does not validate the UI 3. Requires manual testing 4. Flaky tests 5. Execution time (compounded by multiple browser environments) Solution: Visual AI Testing
- 8. A Simple, One Page, Single Browser Example 8
- 9. Traditional Functional Test Example 18 lines of code 21 locators & labels 9 Navigation Button Click Validation 60 minutes to create
- 10. 10 Please enter username and password Traditional Functional Test Example Escaped bug Escaped bug Escaped bug Text change caught Text change caught New feature missed 2 broken locators, false positives Original UI New UI NEW NEW NEW
- 11. 11 Bugs Still Escape Traditional automated testing frameworks are not built for modern apps.
- 12. 12 Bugs Still Escape Traditional automated testing frameworks are not built for modern apps.
- 13. Now, Let’s Provide Initial Coverage Using Visual Testing 2 Minutes to Create Vs. 60 Minutes 1 Locator Vs. 21 Locators and Labels
- 14. 14 Please enter username and password Visual AI Testing finds functional bugs that traditional functional testing can’t. Escaped bug Escaped bug Escaped bug Text change caught Text change caught New feature missed 2 broken locators, false positives Original UI New UI NEW NEW NEW
- 15. ● Do I still need Visual AI Testing if I use a css framework like tailwind? ● Can I drop all of my assertions?
- 16. Visual AI vs Pixel Matching
- 17. Visual AI Is 99.9999% Accurate Not fooled by Config Changes or Browser Updates View by human in Chrome 67 Pixel Diffs View by human in Chrome 68 Visual AI Diffs
- 18. What about adjusting the threshold for pixel matching algorithms?
- 19. This is why Applitools invented Visual AI Visual AI mimics how the human eye and brain analyzes computer screens Trained on >2b images to deliver 99.9999% accuracy 19
- 21. 21 Ignoring Displaced Differences - Only available with Visual AI while bugs go unnoticed with pixel matching
- 22. 22 Ignoring Displaced Differences - Only available with Visual AI while bugs go unnoticed with pixel matching
- 23. 1. Time to Develop/ Maintenance 2. Does not validate the UI 3. Requires manual testing 4. Flaky tests 5. Execution time (compounded by multiple browser environments) Solutions: Self-Healing, Visual AI, Lazy Loading, and assertions
- 24. Flakiness ● Failures due to missing or changed locators. ● Timing issues ● False positives (pixel matching)
- 25. Self-healing ● Self-Heal tests that fail due to missing or changed locators.
- 28. Record and generate with self healing selectors for Cypress
- 29. Flakiness ● Failures due to missing or changed locators. ○ Self Healing and Cypress Record and Generate ● Timing issues ○ SDK LazyLoading ○ Assertions ● False positives (pixel matching) ○ Visual AI
- 30. 1. Time to Develop/ Maintenance 2. Does not validate the UI 3. Requires manual testing 4. Flaky tests 5. Execution time (compounded by multiple browser environments) Solution: Dom capture/snapshot with render grid (Ultrafast Test Cloud)
- 31. Execution Time ● Dom capture/snapshot with render grid (Ultrafast Test Cloud) ● Parallel Execution - Run your tests completely in the cloud in parallel for faster CI/CD pipelines. ○ Execution cloud or any other grid solution ○ Alternative - Chromedriver execution in parallel ○ Parallel CI agents
- 32. Dom Capture/snapshot vs Image Capture
- 33. Dom Capture/snapshot Capture the entire UI with a single snapshot DOM & CSS sent to render grid Applitools Eyes Visual AI analysis renders DOM snapshots in parallel Login to Eyes to view test results . Firewall 33 Applitools Ultrafast Test Cloud Images rendered for each environment
- 34. How does Applitools know what “Golden Image” to compare against?
- 35. How do we use these enhancements to complement our developer workflow?
- 36. Pull Request Worflow integration ● Branching support for developers to work in parallel ● Pull request check - real time check without the need to rerun pipeline ● Automatic merging of images (baselines)
- 38. Ideal World ● Faster test creation ✅ ● Low test maintenance ✅ ● More stable tests ✅ ● Faster execution ✅ ● Full UI coverage ✅ ● Allow shift left ✅ ● Immediate PR feedback ✅ ● Reduced manual testing ✅ ● Release faster ✅
- 39. IDEAL CI/CD TESTING PIPELINE Unit Testing Api Testing Visual AI Functional Testing Functional Testing Performance Testing Exploratory Testing Manual Testing (drastically reduced or eliminated) Production Monitoring Development QA Production
- 40. Thank you