Visual Testing: Turbo-Charge Your Functional Tests with Visual Powers in Just 8 Lines of Code -- by Gil Tayar

@giltayar
You’re almost there:
turbocharge your functional tests
with visual powers
Gil Tayar (@giltayar)
August 2018
This presentation: http://bit.ly/turbocharge-webinar
Github repo: https://github.com/giltayar/realworld-frontend-testing

@giltayar@giltayar
About Me ● My developer experience goes all the way
back to the ‘80s.
● Am, was, and always will be a developer
● Testing the code I write is my passion
● Currently evangelist and architect @
Applitools
● We deliver Visual Testing tools:
If you’re serious about testing, checkout
Applitools Eyes
● Sometimes my arms bend back
● But the gum I like is coming back in style
@giltayar

@giltayar
What Are Functional
Tests?

@giltayar
So many kinds of tests...

@giltayar
Tests, Tests, Glorious Tests!
● Unit
● Integration
● E2E
● Acceptance
● Visual
● Contract
● Browser
● Automation
● Functional
● And the list goes on…

@giltayar
Not to mention the
different interpretations

@giltayar
Functional tests are
automation tests that automate the browser
in order to test the functionality
of a web application*
* Or a mobile app

@giltayar
Let’s write a Functional Test

@giltayar
● Huge library (npm)
○ Biggest.
○ Very high quality
○ Lots of tests!
● DLL hell is gone (almost…)
● No bullshit libraries (readme-s,
not javadocs)
I love NodeJS
● Constantly improving in
performance
● Fast startup times
● ES2018 is a wonderful language
● Async programming is
wonderful
● Good FP language too
● Constantly evolving

@giltayar
● All the cool new automation frameworks are in
NodeJS/JavaScript
● It is the language of the frontend developers
○ Writing tests with the frontend developers
○ Code sharing!
I love NodeJS

@giltayar
Let’s give a look at the app we’re testing

@giltayar
Let’s write a Functional Test in JavaScript!
https://github.com/giltayar/realworld-frontend-testing, branch step1

@giltayar
All Functional Tests are Stories

@giltayar
All Functional Tests Look Like This
● Action
● Action
● Validation
● Action
● Validation
● Action
● Action
● Action
● Validation

@giltayar
Let’s do the following functional story

@giltayar
async function waitFor(selector) {
await driver.wait(
until.elementLocated(By.css(selector)))
}
async function setText(selector, text) {
const field = await driver.findElement(
By.css(selector))
await field.sendKeys(text)
}
Create some utility functions
async function click(selector) {
const element = await driver.findElement(
By.css(selector))
await element.click()
}
async function getText(selector) {
const element = await driver.findElement(
By.css(selector))
return await element.getText()
}

@giltayar
Register the User
action
action
action
action

@giltayar
// action: browse to registration page
await driver.get('http://localhost:3000/register' )
// action: set username, password, email to something
await setText('input[placeholder=Username]' , 'ausername')
await setText('input[placeholder=Password]' , 'aPassword')
await setText('input[placeholder=Email]' , 'an@email.com' )
// action: submit form
await click('button[type=submit]' )
Register the User

@giltayar
// action: browse to registration page
await driver.get('http://localhost:3000/register' )
// action: set username, password, email to something
await setText('input[placeholder=Username]' , 'ausername')
await setText('input[placeholder=Password]' , 'aPassword')
await setText('input[placeholder=Email]' , 'an@email.com' )
await click('button[type=submit]' )
Fragile Selector Problem
Fragile selectors

@giltayar
class RegisterPage {
async function registerUser(username, email, password) {
// ...
}
}
Example RegisterPage Object

@giltayar
Empty Home Page
validation
validation
action validation

@giltayar
await waitFor('img[alt=ausername]')
// validate username
expect(await getText('a[href="/@ausername"]')).to.equal('ausername')
// validate articles list
expect(await getText('.article-preview'))
.to.equal('No articles are here... yet.')
// validate active tab
expect(await getText('.nav-link.active')).to.equal('Your Feed')
Validate the Empty User Home Page

@giltayar
Publish an article
action
action
action
action
action

@giltayar
// action: click on new post
const newPost = await driver.findElement (By.partialLinkText ('New Post'))
await newPost.click()
await waitFor('input[placeholder="Article Title"]' )
// action: set the title, description, and article
await setText('input[placeholder="Article Title"]' , 'a title')
await setText('input[placeholder="What's this article about?"]' , 'something' )
await setText('textarea[placeholder*="Write your article"]' , 'wonderful' )
// action: set the tags
const tags = await driver.findElement (By.css('input[placeholder="Enter tags"]' ))
for (const tag of ['a', 'b', 'c']) await tags.sendKeys(tag, Key.ENTER)
await click('button[type=button]' )
Publish a Post

@giltayar
See that it’s there, and add a comment
validation
validation
validation
validation
action
action

@giltayar
await waitFor('h1')
// validate title
expect(await getText('h1')).to.equal('a title')
// validate article content
expect(await getText('.article-content')).to.include('wonderful')
// validate tags
expect(await getText('.tag-list')).to.equal('abc')
Validate Post

@giltayar
// action: set comment
await setText('textarea', 'a comment')
// action: submit comment
await click('button[type=submit]')
Add Comment

@giltayar
Validate the comment
validation
validation

@giltayar
await waitFor('div.card .card-block')
// validate comment text
expect(await getText('div.card .card-block')).to.equal('a comment')
// validate comment username
expect(await getText('div.card a.comment-author:nth-child(2)'))
.to.equal('ausername')
Validate Comment

@giltayar
// action: goto settings page
await driver.get('http://localhost:3000/settings')
await waitFor('button.btn-outline-danger')
// action: click logout button
await click('button.btn-outline-danger')
Logout

@giltayar
Check Blog with anonymous user
validation
validation
validation
validation
validation

@giltayar
// goto home page
await driver.get('http://localhost:3000/')
// validate post author
expect(await getText('a.author'))
// validate post title
expect(await getText('.article-preview h1'))
.to.equal('a title')
// validate post description
expect(await getText('.article-preview h1 + p'))
.to.equal('something')
// validate post tags
expect(await getText('.article-preview ul'))
.to.equal('abc')
Validate Home Page for Anonymous User
// validate popular tags
expect(await getText('.sidebar .tag-list'))
.to.equal('abc')
// action: goto post page
await click('.article-preview h1')
await waitFor('.article-page')
// validate article description
expect(await getText('div.article-content p'))
.to.equal('wonderful')
// validate article tags
expect(await getText('ul.tag-list')).to.equal('abc')

@giltayar
Problems with Actions
● Fragile Selectors
● Complex Code
● Solution: Page Objects

@giltayar
Problems with Validations
● Complex Code
● Too much to validate
● Only positive check
○ We don’t validate what we don’t know about
● Solution: Page Objects, partially

@giltayar
Actions are Focused
Validations are Holistic

@giltayar
Let’s see why validations are holistic
● Add the following to Register.js:
onSubmit: (username, email, password) => {
dispatch({ type: UPDATE_FIELD_AUTH, key: 'username', value: '' })
const payload = agent.Auth.register(username, email, password);
● This will clear the username after every submit

@giltayar
Let’s see what the effect of this bug will be

@giltayar
But does the test pass?
Yes, it does!

@giltayar
Actions are Focused
Validations are should be Holistic

@giltayar
Let’s revert the “bug”, rebuild,
and figure out how to solve the validation
problem

@giltayar
Solving the Validation Problem

@giltayar
What else is holistic?

@giltayar
Replace Humans with Image Diffing!

@giltayar
Let’s do some live coding for that!

@giltayar
So we’ve replace this...
expect(await getText('.error-messages' )).to.include("email can't be
blank")
With this…
expect(Buffer.from(await driver.takeScreenshot (),
'base64')).to.matchImage(
'registration-blank-email-error' ,
)

@giltayar
Why is this better?
It’s just one line replacing another

@giltayar
Solutions to problems with validations
○ No more selectors
● Complex Code
○ Same code for all validations
● Too much to validate
○ We validate holistically
● Only positive check
○ We validate things we didn’t even think of validating

@giltayar
Let’s try it out...
● Run the tests without the bug
● Copy baseline image to expected-screenshots folder
● Add the following to Register.js:
onSubmit: (username, email, password) => {
dispatch({ type: UPDATE_FIELD_AUTH, key: 'username', value: '' })
const payload = agent.Auth.register(username, email, password);
● This will clear the username after every submit

@giltayar
Functional vs Visual Validations
● Let’s test it in step2: functional validations
git checkout step2 && npm run build && npm test
● The test passes! (and it shouldn’t)
● Now let’s try step3: visual validations
git checkout step2 && npm test
● The test fails! (as it should)

@giltayar
Let’s revert the “bug”, rebuild,
and continue looking at
visual vs. functional valiation

@giltayar
Functional Validations are Focused
Visual Validations are Holistic

@giltayar
async function validatePost() {
await waitFor('.validate-content')
// validate title
// validate article content
expect(await getText('.article-content')).to.include('wonderful')
// validate tags
expect(await getText('.tag-list')).to.equal('abc')
}
And we’ve change this...

@giltayar
async function validatePost() {
await waitFor('.article-content')
await checkWindow('new-post')
}
Into this...

@giltayar
// goto home page
// validate post author
expect(await getText('a.author'))
expect(await getText('.article-preview h1'))
.to.equal('a title')
// validate post description
expect(await getText('.article-preview h1 + p'))
.to.equal('something')
// validate post tags
expect(await getText('.article-preview ul'))
.to.equal('abc')
And this long code...
// validate popular tags
expect(await getText('.sidebar .tag-list'))
.to.equal('abc')
// action: goto post page
// validate article description
expect(await getText('div.article-content p'))
.to.equal('wonderful')
// validate article tags
expect(await getText('ul.tag-list')).to.equal('abc')

@giltayar
async function validateBlog() {
await checkWindow('anonymous-home-page')
await checkWindow('anonymous-blog-post-view')
}
Into this...

@giltayar
Problems with Naive Image Diffing
● Small anti-aliasing differences
○ Mac vs. Linux vs. Windows
○ Different GPUs
● Date/time problems
○ regions that are different from run to run
● Full screen vs. Viewport
● Comparing is a pain
● “Accepting” a new baseline is a pain

@giltayar
Cloud-Based Solutions

@giltayar
Cloud-based solution
● Visual diffing algorithms see like a human sees
○ Ignoring the small differences
○ AI-level algorithms are now involved and keep getting better
● Screenshot management tools are available, enabling you to
○ See the diffs
○ Approve new baselines
○ Open bugs on diffs that are bugs

@giltayar
Let’s Run the Demo

@giltayar
Let’s change some code
to see some of the
baseline management tooling

@giltayar
Supercharging your
functional tests

@giltayar
Supercharging Your Test with Visual Validations
● Write code that issues action after action, to create a long story
○ You can use a recorder like Selenium IDE or one of the new cloud-based
ones
● Intersperse your code with visual validations
○ You can use the open-source solutions or a cloud-based option
● You get two for the price of one:
○ Much simpler functional validations
○ Visual validation of your page: catch those visual bugs that you couldn’t
catch till now!

@giltayar
Thank You
Gil Tayar (@giltayar)
This presentation: http://bit.ly/turbocharge-webinar
The Github repo: https://github.com/giltayar/realworld-frontend-testing

Visual Testing: Turbo-Charge Your Functional Tests with Visual Powers in Just 8 Lines of Code -- by Gil Tayar

More Related Content

What's hot (20)

Similar to Visual Testing: Turbo-Charge Your Functional Tests with Visual Powers in Just 8 Lines of Code -- by Gil Tayar (20)

More from Applitools (20)

Recently uploaded (20)

Visual Testing: Turbo-Charge Your Functional Tests with Visual Powers in Just 8 Lines of Code -- by Gil Tayar