SlideShare a Scribd company logo
1@shwetasharma84
Advanced Automated
Visual Regression
Testing
— Shweta Sharma
2@shwetasharma84
@shwetasharma84
@shwetaneelsharma
@shwetaneelsharma
@shweta_ns
Connect with me
Shweta Sharma
Director of QA Services
India, Pune
3@shwetasharma84
Fully-distributed, 6 Time Zones
10 Years Of Growth
110+ Enthusiastic, Kind,
Open Professionals
150+ Partner Engagements
1,000+ Open Source
Contributions
Axelerant from above
4@shwetasharma84
01
What is Automated Visual
testing?
02 Why do we need it?
03 How does it work?
04
Primary Challenges and
Resolutions
05 Testing Strategy
06 Good Practices
Agenda
07
Automated visual tests in the
Continuous Integration Pipeline
08 Key results achieved
09
The biggest limitation and
resolution
10 When to use what?
11 Facts
12 Where to go from here?
5@shwetasharma84
What is
Automated
Visual testing?
Validate the following:
● Visual content
● Page layout
● Responsive design of Web App
6@shwetasharma84
Why do we need it?
Human
Factor
Larger device/OS
matrix and
release cycles
Insufficient
automated
functional suite
Invest time
to verify new
features
7@shwetasharma84
How does it work?
Test Runner
Capture screenshots at run time
Test passes
Update baselineBug
Compare
Capture baseline images
First run
Subsequent runs
unintended intended
Report differences
8@shwetasharma84
BackstopJS
Demo
9@shwetasharma84
Primary challenges
Anti Aliasing Dynamic Content
10@shwetasharma84
Resolution for anti aliasing
● Use of Docker
○ Environment is the same every time
● Run visual tests on cloud
○ BrowserStack
○ SauceLabs
○ AWS Device Farms
○ Cross Browser testing
● Select a tool which handles it implicitly
11@shwetasharma84
Demo
Dynamic
Content
Using
BackstopJS
12@shwetasharma84
Resolution for dynamic content
Hide
● Hide all elements queried by
BackstopJS/WebdriverIO selector
strategies
● Element’s space is filled and
considered in the layout of the
screenshot captured
● Good candidates - Ad blocks, Images
that change
Remove
● Remove all elements queried by
BackstopJS/WebdriverIO selector
strategies
● Element is removed from the DOM -
affects the design of the page
● Good candidates - Sticky
headers/footers, popover help/chat
windows
13@shwetasharma84
How few
tools
handle
some
challenges
14@shwetasharma84
New
reference
image
using
hide
strategy
15@shwetasharma84
Demo
resolution
for dynamic
content
using
BackstopJS
16@shwetasharma84
Testing Strategy
● Capturing screenshots
○ Element level
■ Component based libraries
● Storybook
● Pattern Lab
■ Development in progress
● Group the components logically (For e.g. Header, Footer etc.)
○ Full page
■ Comparing two similar environments
■ Entire page is developed
17@shwetasharma84
Testing Strategy contd...
● Plan the level of visual coverage needed
○ Both for browsers and devices
■ Every project is different as our customers are different
■ Identify patterns from previously learned lessons
○ Application wide
■ Don’t capture all pages
■ Identify appropriate samples
18@shwetasharma84
Good practices to follow
● Organize Test Suites
○ Browser wise
○ Feature wise
○ Device wise
● Speed up test execution by running tests in parallel
● Identify the frequency to run visual tests
19@shwetasharma84
Organizing tests as per browser
20@shwetasharma84
Creating suites
21@shwetasharma84
How we
integrated
in our
development
workflow?
BUILD
CI PIPELINE
COMMIT
CODE
ACCEPTANCE
TESTING
VISUAL
REGRESSION
DATA
SEEDING
CD PIPELINE
REVIEW
STAGING
PRODUCTION
DB-docker
22@shwetasharma84
Automated Visual Validation
as part of the CI
● Challenges:
○ Storing and maintaining images
○ Identifying the comparison
environment
● Resolution:
○ Use tools like Percy & Applitools
that take care of the images and
also provide you with inbuilt
logic for comparison in CI
23@shwetasharma84
The actual implementation
24@shwetasharma84
The actual implementation
25@shwetasharma84
The actual implementation
26@shwetasharma84
The actual implementation
27@shwetasharma84
The actual implementation
28@shwetasharma84
The actual implementation
29@shwetasharma84
The actual implementation
30@shwetasharma84
Key results achieved
● Test data
○ Time saved to handle dynamic content scenarios because of data consistency
○ Minimal efforts for Visual tests
● Seamless integration of Automation tools involved
● Visual tests validated for the site which is a replica of the wireframe
● Less number of assertions required for the Acceptance test Automation suite
31@shwetasharma84
Example - No functional assertion
32@shwetasharma84
The biggest limitation
* Image taken from TestingWhiz
33@shwetasharma84
Applitools
● 40+ SDKs available free to use
● Capture screenshot commands
○ checkWindow - Entire browser/window
○ checkElement - Particular element
○ checkFrame - Particular frame
○ checkRegion - To handle dynamic and shifting content
● Handles dynamic content which is also floating/shifting
● In-built support for anti-aliasing
● Cool collaborative dashboard with AI support
● Applitools SIDE extension (Chrome and Firefox)
● Free Support for open source projects
● Integrations with CI tools and other collaboration tools
34@shwetasharma84
Match Levels in Applitools
Strict
● Recommended
● Content
● Font
● Layout
● Color
● Position of elements
Layout
● Tests the layout
● Validates the alignment
and relative position of
all elements on the page,
such as: buttons, menus,
text areas, paragraphs,
images, and columns
● Ignores the content,
colour and other style
changes between the
pages
Content
● Similar to Strict
● Compares content but
ignores colors
35@shwetasharma84
When to use what?
Static content
● Use any tool
Shifting Content
● Applitools (SDKs and
Selenium IDE extension)
Dynamic content
● WebdriverIO Visual
regression service
● Shoov
● BackstopJS
● WebdriverCSS
● Gemini
36@shwetasharma84
Facts
● Efforts needed to maintain baseline images
● Respect the Test Pyramid
○ Include Unit tests, Service tests, Acceptance tests and Visual tests
● Best practices in Automating Visual tests:
○ Ensure full page validation
○ Do not have too many element specific tests
■ Avoid element locators maintenance
○ General rules for automating Acceptance tests apply here too:
■ Pick up the right candidates for Automation
■ Logical division of tests
■ Usage of Seams
● Do not expect overnight success
37@shwetasharma84
Where to go from here?
● No coding knowledge?
○ Use configuration based tools like BackstopJS, Wraith, Applitools SIDE browser
extension
○ Pair up with developers, SDETs to have the structure ready
● Good with coding?
○ Look for harmonious integration with your automated functional suite
■ Reduce the unnecessary functional assertions
● Maintainable suite
● Better execution time
○ Be brave and bring the tests in your CI pipeline
■ Let the whole team benefit
38@shwetasharma84
References
● BackstopJS
● Developer Guide
● WebdriverIO - Visual Regression Service
● Applitools for Selenium IDE Tutorial
● Visual Regression Testing Using Wraith
● Run Selenium tests with WebDriverIO
● Percy - Getting started
39@shwetasharma84
Q & A
@shwetasharma84
@shwetaneelsharma
@shwetaneelsharma
@shweta_ns

More Related Content

PDF
Advanced automated visual testing at DrupalCon Europe 2020
PDF
Advanced automated visual testing - DrupalCon seattle 2019
PDF
1x10 - QA Engineer Role in JIRA
PPTX
Application Lifecycle Management - It's a team sport
PDF
Testing Design System Changes Across Your Application -- Intuit Use Case -- w...
PDF
final card-ilovepdf-compressed.6
PPTX
Best Practices for a Repeatable Shift-Left Commitment
PDF
Autodesk revit exam_report
Advanced automated visual testing at DrupalCon Europe 2020
Advanced automated visual testing - DrupalCon seattle 2019
1x10 - QA Engineer Role in JIRA
Application Lifecycle Management - It's a team sport
Testing Design System Changes Across Your Application -- Intuit Use Case -- w...
final card-ilovepdf-compressed.6
Best Practices for a Repeatable Shift-Left Commitment
Autodesk revit exam_report

What's hot (20)

PDF
Stages of project development
PPTX
Technovation challenge workplan for week 10
PDF
2019 Testim Webinar: Automation Test Strategy and Design for Agile Teams
PDF
ADF Certified
PDF
Suchindra samavedam visualcv_resume
PDF
Lessons Learned: The Challenges and Successes of Integrating Automated Testin...
PDF
Iso (9001, 14001) and other standards management systems with OpenERP. Maxime...
PPTX
Swiss Army Knife for Automation Testing
PDF
Oracle ADF Certificate
PPTX
Continuous Testing: The Path Forward
PDF
GTAC 2015 And Innovative Development 30 min
PDF
Using Open Source in Automotive [Study by BearingPoint]
PPTX
Smarter Automation with Machine Learning & AI-Based Reporting
PDF
eCertificate
PDF
Be Product Developers - Eddie Stover
PPTX
(Open Hack Night Fall 2014) Overview
PPT
Agile vs Waterfall From A Tester's Eyes by Shweta Parashar & Abhishek Agrawal
PDF
Why we need test automation, but it’s not the right question
PPTX
ISTQB Advanced Test Manager Training
PPT
Tool for static fab planner 3
Stages of project development
Technovation challenge workplan for week 10
2019 Testim Webinar: Automation Test Strategy and Design for Agile Teams
ADF Certified
Suchindra samavedam visualcv_resume
Lessons Learned: The Challenges and Successes of Integrating Automated Testin...
Iso (9001, 14001) and other standards management systems with OpenERP. Maxime...
Swiss Army Knife for Automation Testing
Oracle ADF Certificate
Continuous Testing: The Path Forward
GTAC 2015 And Innovative Development 30 min
Using Open Source in Automotive [Study by BearingPoint]
Smarter Automation with Machine Learning & AI-Based Reporting
eCertificate
Be Product Developers - Eddie Stover
(Open Hack Night Fall 2014) Overview
Agile vs Waterfall From A Tester's Eyes by Shweta Parashar & Abhishek Agrawal
Why we need test automation, but it’s not the right question
ISTQB Advanced Test Manager Training
Tool for static fab planner 3
Ad

Similar to Advanced automated visual testing at selenium conf india 2020 (20)

PDF
Advanced automated visual testing - DrupalCon Global 2020
PPTX
SeConf2015: Advanced Automated Visual Testing With Selenium
PDF
Leveraging Visual Testing with Your Functional Tests
PPTX
Advanced Automated Visual Testing
PPTX
Automated Visual Testing in NSW.Gov.AU
PPTX
Solving the Automation Puzzle - how to select the right automation framework ...
PPTX
The Holy Trinity of UI Testing by Diego Molina
PDF
Wrong Tool, Wrong Time: Re-Thinking Test Automation -- w/ State of Visual Tes...
PPTX
Advanced automated visual testing with Selenium
PPTX
Advanced Visual Test Automation with Selenium
PDF
Modern Functional Test Automation Through Visual AI - webinar w/ Raja Rao
PDF
Lean Quality & Engineering
PPTX
Emerging Trends of Test automation - Talk in Software Testing Conference Nort...
PPTX
Emerging5.0
PPTX
Visual regression with applitools eyes
PDF
Automated Visual Testing with Selenium & Applitools
PPTX
The Right Tool for the Right Project
PDF
Getting Started with Visual Testing
PPTX
Advanced Cross-Browser Visual Testing with Applitools Eyes and HP LeanFT
PPTX
Introduction to automated visual testing
Advanced automated visual testing - DrupalCon Global 2020
SeConf2015: Advanced Automated Visual Testing With Selenium
Leveraging Visual Testing with Your Functional Tests
Advanced Automated Visual Testing
Automated Visual Testing in NSW.Gov.AU
Solving the Automation Puzzle - how to select the right automation framework ...
The Holy Trinity of UI Testing by Diego Molina
Wrong Tool, Wrong Time: Re-Thinking Test Automation -- w/ State of Visual Tes...
Advanced automated visual testing with Selenium
Advanced Visual Test Automation with Selenium
Modern Functional Test Automation Through Visual AI - webinar w/ Raja Rao
Lean Quality & Engineering
Emerging Trends of Test automation - Talk in Software Testing Conference Nort...
Emerging5.0
Visual regression with applitools eyes
Automated Visual Testing with Selenium & Applitools
The Right Tool for the Right Project
Getting Started with Visual Testing
Advanced Cross-Browser Visual Testing with Applitools Eyes and HP LeanFT
Introduction to automated visual testing
Ad

Recently uploaded (20)

PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
PPTX
Spectroscopy.pptx food analysis technology
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
PDF
Chapter 3 Spatial Domain Image Processing.pdf
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
PPT
Teaching material agriculture food technology
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PDF
MIND Revenue Release Quarter 2 2025 Press Release
DOCX
The AUB Centre for AI in Media Proposal.docx
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
PDF
NewMind AI Weekly Chronicles - August'25 Week I
PPTX
Understanding_Digital_Forensics_Presentation.pptx
PPTX
Cloud computing and distributed systems.
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
Digital-Transformation-Roadmap-for-Companies.pptx
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
Spectroscopy.pptx food analysis technology
Per capita expenditure prediction using model stacking based on satellite ima...
20250228 LYD VKU AI Blended-Learning.pptx
Mobile App Security Testing_ A Comprehensive Guide.pdf
Chapter 3 Spatial Domain Image Processing.pdf
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Teaching material agriculture food technology
Advanced methodologies resolving dimensionality complications for autism neur...
MIND Revenue Release Quarter 2 2025 Press Release
The AUB Centre for AI in Media Proposal.docx
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
NewMind AI Weekly Chronicles - August'25 Week I
Understanding_Digital_Forensics_Presentation.pptx
Cloud computing and distributed systems.
Agricultural_Statistics_at_a_Glance_2022_0.pdf

Advanced automated visual testing at selenium conf india 2020