Automated Visual Testing at Scale : Real-life Example from Dow Jones
1 like2,272 views
The document discusses the implementation of automated visual testing at Dow Jones to ensure superior user experiences across various digital platforms. It highlights the necessity of visual testing in meeting customer expectations, streamlining development processes, and improving quality assurance in the context of rapid CI-CD practices. The advantages of using Applitools for visual testing, including its integration capabilities and significant reduction in testing times, are also emphasized.
Automated Visual Testing at Scale : Real-life Example from Dow Jones
- 1. 1 Automated Visual Testing at Scale: A Dow Jones Case Study Sumeet Mandloi – Dow Jones Eran Barlev - Applitools
- 2. Automated Visual Testing: It’s Really Happening… 1) Why we need it. 2) What it is. 3) How we scale it. 4) Dow Jones case study.
- 3. Why We Need It 3
- 4. Your Brand Experience Is Now Digital 4 OS Types Browser Versions Devices & Screen TypesMicroservices
- 5. Releases Are Expected Faster 5 Watch TestCode ReleaseOperate • Customers today expect new features to be added routinely • CI-CD is an emerging practice necessary to compete in today’s environment
- 6. Brand Experiences Suffers as a Result 6 Watch TestCode ReleaseOperate • Customers today expect new features to be added routinely • CI-CD is an emerging practice necessary to compete in today’s environment • These two trends intersect at the UI and break it routinely, negatively impacting customer experience
- 7. It Happens to Microsoft 7
- 10. It Happens to Amazon 10
- 11. It Used to Happen to Dow Jones 11
- 12. What It Is 12
- 13. Our Mission 13 To ensure visually perfect user experiences on every browser and device through automated visual-AI testing and monitoring.
- 14. What Is Visual Testing? 14 • Visual Testing is the process of validating all the visual aspects of an application’s UI on all platforms • Goes beyond functional testing tools like Selenium to ensure things like colors, fonts, buttons appear correctly
- 15. Visual AI Is The Key 15 RELIABLE MATURE SCALABLE The World’s Largest Database of Visual UI Tests Seamless Integration Virtually No False-Positives
- 16. It’s Really Easy 16 Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline Step 1 Step 2 Step 3Step 4
- 17. Complete the Test Automation Chain 17 Phase Tool Unit Testing Functional Testing
- 18. Complete the Test Automation Chain 18 Phase Tool Unit Testing Functional Testing Visual Testing
- 19. How We Scale It 19
- 20. Tip #1 20 USE A ROBUST IMAGE COMPARISON ENGINE
- 21. Say Sikuli One More Time!
- 22. 22
- 23. 23
- 24. 24
- 25. 25
- 26. 26
- 27. 27
- 28. 28
- 29. - 1 pixel offsets in element positioning - dynamic content moving elements - images of different size - performance And More.... 29
- 30. Tip #2 30 USE A SINGLE BASELINE ACROSS DEVICES AND BROWSERS
- 31. 31 S T R I C T V 1.0 V 2.0 V 3.0 LAYOUT LAYOUT LAYOUT
- 32. Tip #3 32 BASELINE MAINTENANCE COULD BE CODELESS (Everybody Can Do It, Immediately)
- 33. Automated Visual Testing at Dow Jones 33
- 34. 34
- 35. “Coming together is a beginning, staying together is progress, and working together is success.” - Henry Ford 35
- 36. Goals of Quality Engineering 36 True partnering with business Switch from defect tracking to assuring superior customer experience Speeding development and test process Preventing, just not detecting defects Improve CoQ (Cost of Quality) 1 2 3 4 5
- 37. Digital is Visual 37 • Web platforms and mobile apps are the face of the business • Customers demand the best and the first impression matters the most • Respond to business needs quickly
- 38. Digital Is Visual 38 Mobile apps quality is critical • Harder to roll back changes • Can’t push daily • Updates take battery and data • Higher quality bar
- 39. Digital Is Visual 39 The test matrix is too big to cover manually • Web browsers • Devices • Operating systems • Screen resolutions • Responsive design • L10n • 3rd party upgrades
- 40. Digital Is Fast 40 Business needs today demand shorter release cycles
- 41. QA Is The Bottleneck 63% Believe QA is the bottleneck in DevOps 70% Of testing is still done manually 72% Agree continuous testing requires a major shift in culture
- 42. Our Solution: Applitools 42 Validates the content, data, layout and appearance of all the UI elements with a single command Validates the visual aspects of your application/ website UI on all platforms Covers functional testing of all the data that is visible through the UI
- 43. It Actually Works 43 SOFTWARE BANKING & FINANCIAL RETAIL INSURANCE HEALTHCARE
- 44. Features We Value at Dow Jones 44 Single page validation replaces hundreds of lines of validation code and hours of manual testing Automatic maintenance – approved changes are automatically propagated to other tests and execution environments Cognitive vision – ignore differences that are invisible to the human eye and avoid false positives
- 45. Features We Value At Dow Jones 45 Easily test complex application pages including dynamic content, size and moving elements Turnkey Solution – no setup costs Automatically test your app in multiple languages Seamless integration with existing test automation and ALM solution
- 46. 46
- 47. 47
- 48. 48
- 49. Key Integration 49 • Applitools easily integrates with any CI-CD framework • 40+ SDKs for leading functional testing platforms, such as: Selenium, Appium, HP UFT/QTP, MS Coded UI, etc… • Supports JAVA, python, JS, Ruby, .Net • Integration with Perfecto, Sauce Labs, cloud device farms • Jenkins automation environment plug-in
- 50. ROI 50 • Reduce time spent on manual visual QA by 90% • Reduce visual testing coding efforts by 50% • Improve test automation coverage across browsers and devices up to 3 times • Capture 20% more regressions and UI bugs pre-release POC – Reduced total testing efforts from 30-40 hours to 2 hours
- 51. Who can benefit from this? 51 • Software Engineers building Web Platforms and Mobile Apps • Design and UX Teams • Marketing and Advertising • Customer Experience • Operations
- 52. Thank You! 52