Advanced automated visual testing at DrupalCon Europe 2020
- 1. 1@shwetasharma84 Advanced Automated Visual Regression Testing — Shweta Sharma
- 2. 2@shwetasharma84 @shwetasharma84 @shwetaneelsharma @shwetaneelsharma @shweta_ns Connect with me Shweta Sharma Director of QA Services India, Pune
- 3. 3@shwetasharma84 Fully-distributed, 6 Time Zones 10 Years Of Growth 120+ Enthusiastic, Kind, Open Professionals 100+ Enterprise Engagements Qualitative Drupal Contributor Axelerant from above
- 4. 4@shwetasharma84 01 What is Automated Visual testing? 02 How does it work? 03 Level - 1 (Objectives, Tools, demo, challenges-resolutions) 04 Level - 2 (Objectives, Tools, demo, challenges-resolutions) 05 Level - 3 (Objectives, Tools, demo, challenges-resolutions) Agenda 06 Comparison between tools 07 Testing Strategy 08 Good practices 09 Key results achieved 10 Facts 11 Where to go from here? 12 References
- 5. 5@shwetasharma84 What is Automated Visual testing? Validate the following: ● Visual content ● Page layout ● Responsive design of Web App
- 6. 6@shwetasharma84 How does it work? Test Runner Capture screenshots at run time Test passes Update baselineBug Compare Capture baseline images First run Subsequent runs unintended intended Fails and report differences
- 7. 7@shwetasharma84 ● Site development in progress/maintenance ● Optimise testing efforts as an individual ● Resolve challenges faced at individual level LEVEL 1 - Beginner
- 8. 8@shwetasharma84 BackstopJS - viewports and scenario defined (element)
- 9. 9@shwetasharma84 BackstopJS - viewports and scenario defined (full page)
- 10. 10@shwetasharma84 BackstopJS Demo - Create reference images
- 12. 12@shwetasharma84 BackstopJS Demo - Execute Tests
- 13. 13@shwetasharma84 Challenge 1 - Dynamic content
- 16. 16@shwetasharma84 How few tools handle some challenges
- 18. 18@shwetasharma84 New reference image using hide strategy
- 20. 20@shwetasharma84 GET STARTED - TOOLS AVAILABLE Open-source BackstopJS, Wraith, Spectre Commercial Applitools SIDE (Selenium IDE) extension, Percy, Diffy
- 21. 21@shwetasharma84 LEVEL 2 - OBJECTIVES and CHALLENGES OBJECTIVES CHALLENGES Involve more than 1 individual so that more people benefit from automation Baseline images created from one machine fail on others due to anti-aliasing Simulating user actions like filling up forms, clicking, hovering etc. Seamless integration with existing testing frameworks Optimise testing efforts for a small development/testing team Harmonious integration with your CI setup
- 22. 22@shwetasharma84 Resolution for Level 2 challenges ● Anti-aliasing ○ Use of Docker so that the environment is the same every time ○ Run visual tests on cloud ■ BrowserStack ■ SauceLabs ■ AWS Device Farms ■ Cross Browser testing ■ Select a tool which handles it implicitly ● Integration with existing testing frameworks ○ Open source and commercial tools available ● Integration with the CI setup ○ Commercial tools available
- 23. 23@shwetasharma84 TOOLS AVAILABLE Open-source Several visual testing plugins by Cypress Commercial Applitools, Percy
- 24. 24@shwetasharma84 Page to be tested
- 25. 25@shwetasharma84 Cypress & Percy - The actual test
- 26. 26@shwetasharma84 Cypress + Percy - Create baseline images
- 27. 27@shwetasharma84 Cypress + Percy - Approve baseline images
- 28. 28@shwetasharma84 Cypress + Percy - Bug introduced in feature branch
- 29. 29@shwetasharma84 LEVEL 3 - OBJECTIVES and CHALLENGES CHALLENGES CAUSE OBJECTIVES Running into false positives ● Small vertical/horizontal shifts ● Browser updates Avoid pixel-to-pixel matching Increased time test on several browsers & devices Execution happens on all configured browsers/devices Better Test coverage for visual tests at a faster pace Frequently/Major change in elements Change requests Better baseline management Scattered tests, poor readability Inability to group Better Test Suite management
- 30. 30@shwetasharma84 TOOLS AVAILABLE Open-source None Commercial Applitools, Percy (to some extent)
- 31. 31@shwetasharma84 FALSE POSITIVES - Resolution PERCY APPLITOOLS It ignores 1px difference It uses AI-based comparison algorithm
- 32. 32@shwetasharma84 BROWSERS/DEVICES & SCATTERED TESTS - Resolution ● APPLITOOLS ○ Supports all browsers & devices ○ Executes once and captures everywhere in background in parallel ○ Create relevant Batches and group tests ● PERCY ○ Supports only Chrome & FF ○ Supports only viewports
- 33. 33@shwetasharma84 Comparison between tools (Must have) PARAMETER PERCY APPLITOOLS Pixel-to-pixel matching ✅ ✅ AI-based matching ❌ ✅ Browsers Chrome and Firefox All Devices Only Viewports All Dynamic content handling ✅ ✅ Baseline management ❌ ✅ Team management ✅ ✅
- 34. 34@shwetasharma84 Comparison between tools (Need basis) PARAMETER PERCY APPLITOOLS Root cause analysis ❌ ✅ Integration tools Source control, build servers, collaboration Source control, build servers, collaboration SDK support Good Excellent On-premise support ❌ ✅ Collaborative dashboard ❌ ✅ Desktop, mobile app support ❌ ✅
- 36. 36@shwetasharma84 Testing Strategy ● Capturing screenshots ○ Element level ■ Component based libraries ● Storybook ● Pattern Lab ■ Development in progress ● Group the components logically (For e.g. Header, Footer etc.) ○ Full page ■ Comparing two similar environments ■ Entire page is developed
- 37. 37@shwetasharma84 Testing Strategy contd... ● Plan the level of visual coverage needed ○ Both for browsers and devices ■ Every project is different as our customers are different ■ Identify patterns from previously learned lessons ○ Application wide ■ Don’t capture all pages ■ Identify appropriate samples
- 38. 38@shwetasharma84 Good practices to follow ● Organize Test Suites ○ Browser wise ○ Feature wise ○ Device wise ● Speed up test execution by running tests in parallel ● Identify the frequency to run visual tests
- 39. 39@shwetasharma84 Organizing tests as per browser
- 41. 41@shwetasharma84 Key results achieved at Axelerant ● Test data ○ Time saved to handle dynamic content scenarios because of data consistency ○ Minimal efforts for Visual tests ● Seamless integration of Automation tools involved ● Visual tests validated for the site which is a replica of the wireframe ● Less number of assertions required for the Acceptance test Automation suite
- 42. 42@shwetasharma84 Facts ● Efforts needed to maintain baseline images ● Respect the Test Pyramid ○ Include Unit tests, Service tests, Acceptance tests and Visual tests ● Best practices in Automating Visual tests: ○ Ensure full page validation ○ Do not have too many element specific tests ■ Avoid element locators maintenance ○ General rules for automating Acceptance tests apply here too: ■ Pick up the right candidates for Automation ■ Logical division of tests ■ Usage of Seams ● Do not expect overnight success
- 43. 43@shwetasharma84 Where to go from here? ● No coding knowledge? ○ Use configuration based tools like BackstopJS, Wraith, Applitools SIDE browser extension ○ Pair up with developers, SDETs to have the structure ready ● Good with coding? ○ Look for harmonious integration with your automated functional suite ■ Reduce the unnecessary functional assertions ● Maintainable suite ● Better execution time ○ Be brave and bring the tests in your CI pipeline ■ Let the whole team benefit
- 44. 44@shwetasharma84 References ● BackstopJS ● WebdriverIO - Visual Regression Service ● Applitools for Selenium IDE Tutorial ● Visual Regression Testing Using Wraith ● Run Selenium tests with WebDriverIO ● Cypress visual testing plugins ● Diffy ● Spectre ● Applitools Eyes ● Applitools Ultrafast Test Cloud ● Percy - Getting started