Visual Testing: The Missing Piece of the Puzzle -- presentation by Gil Tayar

@giltayar
Visual Testing:
The Missing Piece of the Puzzle
Gil Tayar (@giltayar)
September 2019

@giltayar@giltayar
About Me ● My developer experience goes all the way
back to the ‘80s.
● Am, was, and always will be a developer
● Testing the code I write is my passion
● Currently evangelist and architect @
Applitools
● We deliver Visual Testing tools:
If you’re serious about testing, checkout
Applitools Eyes
● Sometimes my arms bend back
● But the gum I like is coming back in style
@giltayar

@giltayar
Fear of
approaching
that cool guy

@giltayar
Fear of public
speaking

@giltayar
What About Software Development Fears?

@giltayar
Fear of adding code for
features

@giltayar
Fear of deleting code

@giltayar
The Development Fears
● Fear of adding code
● Fear of deleting code
● Fear of refactoring

@giltayar
Because Developers are Shit

@giltayar
“All software
is a harmony
of bugs”
— Eli Singer

@giltayar
How do we
overcome our
fears?

@giltayar
We write unit tests!
● Code that tests
○ A class
○ A function
○ A module
○ A UI component

@giltayar
Backend unit tests Frontend unit
tests

@giltayar
Most of our
code is glue

@giltayar
And sometimes the units work, but...

@giltayar
We write integration tests!
● Testing a set of units functioning together
● Testing a whole frontend app, without the backend
● Testing a whole microservice, without the other
microservices

@giltayar
Backend unit tests Backend Services Backend database
integration
Frontend unit
tests
Frontend
components
Frontend
Application

@giltayar
So how do tests overcome our software
development fears?

@giltayar
A developer codes...

@giltayar
And she writes the tests for the code
(sometimes even before writing the code…)

@giltayar
And if she...
● Adds code
● Deletes code
● Refactors

@giltayar
She just runs the tests again...

@giltayar
But are we really testing our whole code?

@giltayar
Are we testing our HTML? Our CSS?

@giltayar
Fear still exists in the frontend world
● Fear of adding/removing/refactoring CSS
● Fear of adding/removing/refactoring HTML

@giltayar
Are we testing our whole code?
Backend
● All Logic code ✔︎
● Database code ✔︎
● Microservices ✔︎
Frontend
● All Logic code
✔︎
● Component code
✔︎
● Ajax Code
✔︎
● Component Visuals
✕
● Application Visuals

@giltayar
Backend unit
tests
Backend
Services
Backend
database
integration
Frontend
unit tests
Frontend
components
Component
Visuals Frontend
Application
Application
Visuals

@giltayar
Visual Testing
Components Application

@giltayar
Visual Testing
Components

@giltayar
How do we visually test our components?
1. Build a “demo” app that includes all the components
2. Run a browser
3. For each component
a. Take a screenshot
b. Compare against a baseline screenshot
i. The first run of the test determines the baseline screenshot

@giltayar
We need to...
1. Build a “demo” app that includes all the components
2. Run a browser
3. For each component
a. Take a screenshot
b. Compare against a baseline screenshot
i. The first run of the test determines the baseline screenshot

@giltayar
It’s not easy
● Modern Frontend development makes it difficult
● You have to deal with babel, webpack, … configurations
from here to eternity
● But there’s a solution!

@giltayar
Storybook
With just a few lines of HTML, we get all our components
in one nice UI

@giltayar
The code that generates the storybook
storiesOf('Header', module)
.add('no user', () => <Header appName="Blog App" />)
.add('with user', () =>
<Header appName="Blog App" currentUser={{userName:
'giltayar'}}/>);
storiesOf('Login', module)
.add('empty', () => <Login />)
.add('with text', () =>
<Login email="gil@tayar.org" password="apassword" />);

@giltayar
Navigation
Component

@giltayar
Run a browser
let driver
before(
async () =>
(driver = await new webdriver.Builder()
.forBrowser('chrome')
.build())
)
after(async () => await driver.quit())
let eyes
before(async () => {
await driver.get('http://localhost:9009')
})

@giltayar
Take screenshot and compare
it('should show "no user" correctly', async () => {
const noUser = await driver.findElement(By.linkText('no user'))
await noUser.click()
const screenshot = await driver.takeScreenshot()
expect(screenshot).to.matchImage('no-user')
})

@giltayar
Take screenshot and compare
it('should show "with user" correctly', async () => {
const noUser = await driver.findElement(By.linkText(with user'))
await noUser.click()
const screenshot = new Buffer(await driver.takeScreenshot(),
'base64')
expect(screenshot).to.matchImage(with-user')
})

@giltayar
Yes! We have overcome our component visual
fears!

@giltayar
Problems...
● Setup is difficult
● Flakiness in tests
● Baseline administration
● Scaling to hundreds of components

@giltayar
Setup is difficult
● Setting up (and tearing down!) storybook is more difficult
than it seems
○ Orphan processes
● Running a browser is a chore

@giltayar
• Image scaling
• 1 pixel offsets in element positioning
• Dynamic content
• Moving elements
• Images of different size
• Performance
AND MORE…

@giltayar
Here’s another one...
Chrome v67 Chrome v68

@giltayar
And the difference is...

@giltayar
Remember our Testing Bookstore? (1)

@giltayar
Remember our Testing Bookstore? (2)

@giltayar
Baseline Administration
● Baseline administration by files
● Copy to change baseline
● Diff files

@giltayar
Scaling to thousands of components
● If I have 100 components, and each one has 3 different
“views”...
● The test will take tens of minutes!

@giltayar
Solution: Commercial-Grade SaaS Tools

@giltayar
Solutions
● Scaling to hundreds of components

@giltayar
Baseline administration
● Upload screenshots
to the cloud
● Use GUI tools to
○ View tests
○ View diffs
○ Update baselines

@giltayar
Flakiness
● Use advanced visual algorithms
● Mix with AI algorithms (ML and DL)
● Result: no flakiness
● Also get enhanced capabilities:
○ Layout mode
○ Ignore regions
○ And more...

@giltayar
Setup is difficult
● Use a good Storybook SDK
● And get a no-coding experience that tests your storybook
for free!

@giltayar
Scaling to hundreds of components
● Push the rendering itself to the cloud
● Send the DOM + resources (CSS, images, fonts, ...) to the cloud,
and have a grid of browsers render them concurrently
● [nice image]

@giltayar
Visual Testing
Application

@giltayar
1. Run the app
2. Run a browser
3. Drive the app along the flow we want
4. Take screenshots along the way
5. Compare against a baseline screenshot
We need to...

@giltayar
Drive the flow
it('should show the main flow correctly', async () => {
await registerUser()
await publishPost()
await logout()
await showBlogHomePage()
})

@giltayar
… and take screenshots along the way
async function registerUser() {
await driver.get('http://localhost:3000/register')
await setText('input[placeholder=Username]', 'ausername')
await setText('input[placeholder=Password]', 'aPassword')
await setText('input[placeholder=Email]', 'an@email.com')
await checkWindow('registration-page')
await click('button[type=submit]')
await waitFor('.article-preview')
await checkWindow('empty-user-home-page')
}

@giltayar
Similar Problems
● Scaling to hundreds of pages

@giltayar
With a similar solution...

@giltayar
How do we
overcome our
fear of
deployment?

@giltayar
E2E Tests
● Tests everything together:
● Frontend, with
● Backend, with
● All microservices together,
● In an environment that is as close to production as possible

@giltayar
The Theory
Visuals are covered by E2E tests, so no need to cover
them in E2E tests

@giltayar
The Reality
Most frontend testing is done by the QA Automation team

@giltayar
Similar Problems (but much much bigger)
● Scaling to hundreds of pages

@giltayar
integration
Frontend unit
tests
Frontend
components
Component
Visuals
Frontend
Application
Application
Visuals
E2E Tests
E2E Automation
Visual Tests

@giltayar
Fear of problems in
production

@giltayar
Backend Production Problems
● Hardware/Node problems
● Networking problem
● Database problems
● Scaling problems
● …

@giltayar
Solution: Observability
● Logging
● Monitoring
● Alerting

@giltayar
Frontend Problems
● What frontend problems could there be?
● It’s all been tested!

@giltayar
But how can we visually test this?

@giltayar
Visual layout testing:
Testing the layout while ignoring the data

@giltayar
A non-trivial problem
This is where machine-learning really shines

@giltayar
integration
Frontend unit
tests
Frontend
components
Component
Visuals
Frontend
Application
Application
Visuals
E2E Tests
E2E Automation
Visual Tests
Production
Observability
Production Visual
Layout

@giltayar
How do we
overcome our
fear of
hacking?

@giltayar
What can we do?
● Penetration Testing

@giltayar
integration
Frontend unit
tests
Frontend
components
Component
Visuals
Frontend
Application
Application
Visuals
E2E Tests
E2E Automation
Visual Tests
Production
Observability
Production Visual
Layout
Penetration Testing

@giltayar
But What About Site Defacing?

@giltayar
It’s the corporate sites that are usually defaced

@giltayar
integration
Frontend unit
tests
Frontend
components
Component
Visuals
Frontend
Application
Application
Visuals
E2E Tests
E2E Automation
Visual Tests
Production
Observability
Production Visual
Layout
Penetration Testing
Corporate Site
Visual Testing

@giltayar
● Fear of hacking
○ Web site defacement
● Fear of production problems
○ Layout problems
● Fear of deployment
○ E2E Automation tests
● Fear of changing code
○ Component visual bugs
○ Application visual bugs
Backend unit
tests
Backend Services Backend
database
integration
Frontend
unit tests
Frontend
components
Component
Visuals
Frontend
Application
Application
Visuals
E2E Tests
E2E Automation
Visual Tests
Production
Observability
Production Visual
Layout
Penetration
Testing
Corporate Site
Visual Testing

@giltayar
If you come out of this talk learning one thing,
it’s this...

@giltayar
MY TESTS WILL REMAIN.
TEST MY CODE.

@giltayar
Write tests and overcome your fear!

@giltayar
(and don’t forget to test the visual parts…)

@giltayar
Thank You!
Gil Tayar (@giltayar)

Visual Testing: The Missing Piece of the Puzzle -- presentation by Gil Tayar

More Related Content

What's hot (20)

Similar to Visual Testing: The Missing Piece of the Puzzle -- presentation by Gil Tayar (20)

More from Applitools (20)

Recently uploaded (20)

Visual Testing: The Missing Piece of the Puzzle -- presentation by Gil Tayar