You have Selenium... Now what?

You have Selenium... Now what?
Hans Buwalda
©2016 LogiGear Corporation
www.logigear.com
www.testarchitect.com
www.happytester.com
@hansbuwalda
hans@logigear.com
hans@happytester.com

• Topics:
– domain language approaches
– test design as driver for automation
– automation friendly test design techniques and examples
– special cases (data driven, multi-station, graphics, etc)
– drone perspective: overview with some selective details
• What this talk is not:
– an interactive workshop
– specific to Selenium only
– a technical class on automation
– an interactive workshop
– about component based development
– introduction to unit testing
Scope
Warning: there is an overlap with my talk yesterday!!

• A combination of open source products and standards, aimed at
web testing (including a Firefox based IDE)
• In the later versions it works with an interface called “WebDriver”,
implemented with drivers in browsers
• WebDriver has been implemented in open source libraries, but is
now also becoming implemented in browsers
• For example Microsoft Edge can only be accessed programmatically
via WebDriver
• Products like Appium try to implement WebDriver for mobile
browsing and even mobile apps
• Our product (TestArchitect) is moving to WebDriver, expect this to be
a common practice
Selenium

Relation to code Quality / depth Automation Scalability
Unit Testing
Close relationship
with the code
Singular test
scope, but deep
into the code
Fully automated
by nature
Scalable, grows
with the code,
easy to repeat
Functional/
Integration
Testing
Usually does not
have a one-on-one
relation with code
Quality and scope
depends on test
design
In particular UI
based automation
can be a challenge
Often a bottle-
neck in scalability
Exploratory
Testing
Human driven, not
seeking a relation
with code
Usually deep and
thorough, good at
finding problems
May or may not
be automated
afterwards
Not meant to be
repeatable. Rather
do a new session
Some test kinds and their scalability (simplified)

Test Pyramid
• Proposed by Mike Cohn (in his book: "Succeeding with Agile").
• My version would be less "pointy"
• In particular I like to separate technical and design considerations
• Also, I've seen API's that, for higher level operations, are quite complex. Trying to use
them can become like re-implementing the UI again
• Web UI's are getting richer and more complex to test (SPA's, AngularJS, etc)
UI
Service or
Component
Unit
UI
Unit
original my version (for now)
Service or
Component

Issues are not always obvious...
Downton Abbey

Brief test design exercise
• View this imaginary application
• It adds two numbers
• What tests would you create?
• Which tests would you automate?

The 5% Rules of Test Automation
• No more than 5% of all test cases should be
executed manually
• No more than 5% of all efforts around
testing should involve automating the tests

Don't just automate manual testing

Don't just automate manual testing
Good automated testing is not the same as automating good manual testing. . .

Example Scripting, Java + Selenium
WebElement element = driver.findElement(By.name(name));
element.sendKeys("mystery magic");
element.submit();
(new WebDriverWait(driver, 10)).until(
new ExpectedCondition<Boolean>() {
public Boolean apply(WebDriver d) {
return d.getTitle()
.toLowerCase().startsWith("mystery");
}
}
);
System.out.println(driver.getTitle());

• Frameworks, libraries, patterns
– libraries: sets of functions, you call them
– frameworks: also sets of functions, but the framework is typically in
charge and calls you
– patterns: standardized ways to do certain tasks
• Selenium is often used with frameworks
– to execute tests and collect results (like unit testing frameworks)
– to offer higher level API's
– to support BDD, data driven testing or keywords
• A common practice is the Page Object pattern
– shield from HTML
– provide an application specific API, with functions like "getSongTitle"
• Examples of keyword frameworks: Robot Framework,
and our own TestArchitect
Libraries, frameworks, patterns
Framework

Domain language approaches: Actions
4 actions, each with
an action keyword
and arguments
read from top
to bottom
fragment from a test with actions
acc nr first last
open account 123123 John Doe
acc nr amount
deposit 123123 10.11
deposit 123123 20.22
acc nr expected
check balance 123123 30.33
• Stays out of the programming language
• The test developer creates tests using actions with keywords and
arguments
• Checks are, as much as possible, explicit (specified expected values)
• The automation task focuses on automating the keywords, each
keyword is automated only once

• In "low level" tests interaction details are not hidden, since they are
the target of the test
• The right level of abstraction depends on the scope of the test, and is
an outcome of your test design process
TEST MODULE Screen Flow
user
start system john
TEST CASE TC 01 Order button
window button
click main create order
window
check window exists new order
FINAL
close application
Example of a low level test

Variables and expressions with keywords
• This test does not need an absolute number for the
available cars, just wants to see if a stock is updated
• As a convention we denote an assignment with ">>"
• The "#" indicates an expression
TEST CASE TC 02 Rent some more cars
car available
get quantity Chevy Volt >> volt
first name last name car
rent car John Green Chevy Volt
rent car Jane White Chevy Volt
car expected
check quantity Chevy Volt # volt - 2

Data driven testing with keywords
• The test lines will be repeated for each row in the data set
• The values represented by "car", "first" and "last" come
from the selected row of the data set
TEST CASE TC 03 Check stocks
data set
use data set /cars
car available
get quantity # car >> quantity
first name last name car
rent car # first # last # car
car expected
check quantity # car # quantity - 1
repeat for data set
DATA SET cars
first last car
John Doe Chevy Volt
Mary Kane Ford Escape
Jane Collins Chrysler 300
Tom Anderson Buick Verano
Henry Smyth BMW 750
Vivian Major Toyota Corolla

ACTION DEFINITION check balance
user
argument customer
argument amount
window control value
enter balance inquiry last name # customer
window control
click balance inquiry view balance
window control expected
check balance inquiry balance # amount
Re-use actions to make new actions
• In the below example we make a new action
• Existing actions are strung together to create new ones with a
broader scope
• Often steps in low level tests are re-used to create these action
definitions
:
customer amount
check balance Smith 223.45
check balance Jones 0.00
check balance James -330.45
:
use many times in tests:
define in one place:

Example action implementation in Python
This action script (in Python, Java or C#) verifies whether the rows in a table are sorted:
# get table object, column number and column count
windowName = LIBRARY.NamedArgument("window")
tableName = LIBRARY.NamedArgument("table")
columnName = LIBRARY.NamedArgument("column")
table = ABT.OpenElement(windowName, tableName)
column = table.GetColumnIndex(columnName)
rowCount = table.GetRowCount()
# check the sort order, row by row
previous = table.GetCellText(0, column)
for i in range(1, rowCount):
current = table.GetCellText(i, column)
if current < previous :
LIBRARY.AdministerCheck("order", "sorted", "fails " + str(i+1), 0)
return
previous = current
LIBRARY.AdministerCheck("order", "sorted", "all rows in order", 1)
find the table in the UI
if a value is smaller than before, fail the test
if all rows are ascending, pass the test
get arguments from the test line
def action_checkSortOrder():
Note that the approach resembles the Page Object pattern, but in my view is not the
same. The abstraction is “any table”, even in non-Web UI’s or non-UI’s altogether. The
Page Object pattern could also work on top of this.

Using the new action
• By keeping an action generic it can be applied for a
variety of situations
• Some examples of using "check sort order":
window table column
check sort order view orders orders table ID
window table column
check sort order annual results regions revenue
window table column
check sort order inventory cars price
window table column
check sort order registration students last name

Behavior Driven Development (BDD)
• Uses natural language scenarios
• Tools like JBehave and Cucumber map these to code
• Structure is "Given-When-Then" (GWT)
• Example:
Given a customer previously bought a black sweater from me
And I currently have three black sweaters left in stock
When he returns the sweater for a refund
Then I should have four black sweaters in stock
(source: Wikipedia)

BDD with keywords (example)
GIVEN A customer previously bought a black sweater from me
first last id
add client John Jones >> client
AND I currently have three black sweaters left in stock
article color quantity
get stock sweater black >> sweaters
WHEN He returns the sweater for a refund
customer article color
return article # client sweater black
THEN I should have four black sweaters in stock
article color quantity
check stock sweaters black # sweaters + 1

Equivalence, conversion
Given a customer previously bought a black sweater from me
And I currently have three black sweaters left in stock
When he returns the sweater for a refund
Then I should have four black sweaters in stock
customer buys, article, color
a customer previously bought a {color} {article} from me
set stock, article, color, amount
I currently have {amount} {color} {article} left in stock
my stock of {color} {sweater} is {amount}
return article, article, color
he returns the {color} {article} for a refund
check stock, article, color
I should have four {color} {article} in stock
article color
customer buys sweater black
article color amount
set stock sweater black 3
article color
return article sweater black
article color amount
check stock sweater black 4

Test Module Sections
Test Module
Objectives
Initial - setup
Test cases
Final - cleanup

High Level Test Design - Test Development Plan
Objectives
Test Module 1
Test Cases
Test Module 2 Test Module N
Actions
. . .
AUTOMATION
Objectives Objectives
interaction test business test
define the "chapters"
create the "chapters"
create the "words"
make the words work
Test Cases Test Cases
enter log in user name jdoe
enter log in password car guy
window control property expected
check property log in ok button enabled true
user password
log in jdoe car guy
first last brand model
enter rental Mary Renter Ford Escape
last total
check bill Renter 140.42
Action Based Testing

Organize the tests
• What tests would you create for this application?
• How would you organize them?
• What, if anything, would you avoid?

Why Better Test Design?
• Better test design can improve quality of test
– many tests are often quite "mechanical" now, no surprises
– one to one related to specifications, user stories or requirements,
which often is ok, but lacks aggression
– no combinations, no unexpected situations, lame and boring
– such tests have a hard time finding (interesting) bugs
• Better test design can give (much) better automation
– when unneeded details are left out of tests, those details don't have
to be maintained
– avoiding "over checking": creating checks that are not in the scope
of a test, but may fail after system changes
– limit the impact of system changes on tests, making such impact
more manageable
I have become to believe that successful automation is usually less of a
technical challenge as it is a test design challenge.

What's the trick...

• Business objects and business flows
– objects are like cars, invoices, locations, etc
– flows are like create, fulfill, pay and close an order
• Other tests
– functions and features, like premium calculation or PDF output
– administration, users, security, authorizations
– graphics
– technologies, protocols, ...
– customization, extensibility
– . . .
• Business versus interaction
– differentiate within business objects and other categories
– interaction can be further differentiated into: values, UI, keyboard,
etc
Example of considerations

Example Top Level Structure
Project create, update, delete/close
copy, move
categorize, enumerate, identify
convert, serialize, export/import, ...
UI, dialogs, forms, pages
input (validation, defaulting, dependencies)
flows (primary paths, alternate paths)
keyboard shortcuts, keyboard controls, ...
. . .
<business object 1>
Lifecycles, data operations
Interaction
Functions and Features
Technologies, protocols, controls
Data (handling, quality, ETL, ...)
Security, authorization, admin
Graphics, multi-media, charts, ...
Load, performance
Integration
Customizing, extensibility
Business Flows
Concurrency, race conditions, ...
Business Objects
processes, transactions, end-to-end, day in the life,
combinations of flows, ...
calculations, analyses, PDF output, ...
Repeat for each
component or sub-system

Example: E-commerce site
• Articles
• Categories
• Customers
• Promotions
• Orders
• Invoices
• Payments
• Credit cards
• Staff members
• Countries
• . . .

• Promotion kinds
– percentage discounts
– fixed dollar discounts
– comprehensive promotions
– regional promotions
– volume discounts
– prompt payment discounts
– article based, comprehensive
– . . .
• Life cycles
– creation
– modification
• various properties
• before or after activation
– expiration
– cancellation
Promotions -- business tests

Promotions -- interaction tests
• Screen by screen
• Discount percentage/fixed
– does the amount field change
• Regional checkbox
– does the region list show up
– are the regions correct
– can the regions be selected/deselected
• Start, duration, end date
– are the date checks working
– can either duration or end date be entered
• Articles, categories
– can categories be selected, do they show up
– do the right articles show up, can they be selected
• . . .

What tests could look like
Business tests for business object "Promotions"
Interaction tests for business object "Promotions"
window control
click main new promotion
window type
select promotion town
window list value
check list item exists promotion towns Tietjerksteradeel
name start finish percentage category
create promotion christmas 12/1/2016 12/25/2016 11 all
create promotion tablets 12/20/2016 1/1/2017 20 tablets
date
time travel 12/23/2016
article price
check nett price iPad Mini 4 284.79

Example 2: a testing company in
California
• The company sells services like test development and
automation
• It also has consultancy and training offerings
• And it has a product for test development and
automation
• These various offerings are often combined in the orders
they receive
• Invoicing for services as monthly
• They're working on an internal system for order handling
and invoicing

Example 2: a testing company in
California
• Example business objects:
– projects, orders ("SOW's"), customers, employees, invoices,
payments
• Example business flows:
– customer place orders, often multiple ones over time
– one order can have multiple items
– tools are invoiced (and delivered) immediately
– services are delivered over time
– staff members write time-sheets with hours spent
• has elaborate UI's and processes
– invoices can combine items from multiple orders
– payments can cover multiple invoices

What about existing tests?
• Compare to moving house:
– some effort can't be avoided
– be selective, edit your stuff,
• look at the future, not the past
– first decide where to put what, then put it there
– moving is an opportunity, you may not get such chance again soon
• Follow the module approach
– define the modules and their scope as if from scratch
– use the existing test cases in two ways:
• verify completeness
• harvest and re-use them for tests and for actions
– avoid porting over "step by step", in particular avoid over-checking

Eye on the ball, Scope
• Always know the scope of the test module
• The scope should be unambiguous
• The scope determines many things:
– what the test objectives are
– which test cases to expect
– what level of actions to use
– what the checks are about and which events should generate a
warning or error (if a “lower” functionality is wrong)

Use the right level actions
Low level of UI interaction detail makes sense only with the module
scope is to test the UI
window tree tree item path
click tree item main projects /Projects/Drill Assembly
window list item
check list item exists main tasks Plan of Approach
Better to have a business level action and hide the details in an action
definition:
But do show details when it matters. The example below is too high level,
requires drill down into action definition to understand what is tested.

Example of a Test
Step Description Expected
step 16 Open http://www.bigstore.com The "BIG Store" main page is displayed, with a "sign in" link
step 17 Click on "Sign In", upper right corner A sign in dialog shows, the "Sign in" button is disabled
step 18 Enter "johnd" in the user name field The "Sign In" button is still disabled
step 19 Enter "bigtester" in the password field Now the "Sign In" button is enabled
step 20 Click on the "Sign in" button The page now shows "Hello John" in the upper right corner
step 21 Enter "acme watch" in the search field The "Search" button is enabled
step 22 Click on the "Search" button 5 watches of Acme Corporation are displayed
step 23 Double click on "Acme Super Watch 2" The details page of the Acme Super Watch 2 is displayed
step 24 Verify the picture of the watch The picture should show a black Acme Super Watch 2
step 25 Select "red" in the "Color" dropdown list The picture now shows a black Acme Super Watch 2
step 26 Type 2 in the "Order quantity" textbox The price in the right shows "$79.00 + Free Shipping"
step 27 Click on "Add to cart" The status panel shows "Acme Super Watch 2" added
step 28 Click on "Check out" The Cart Check Out open, with the 2 Acme Super Watches
is this a good test? is it good for automation?

GWT scenario
Given User turns off Password required option for Drive Test
And User has logged in by Traffic Applicant account
And User is at the Assessments Take a Test page
And User clicks the Traffic Test link
And User clicks the Next button
And User clicks the Sheet radio button in Mode page if displayed
And User clicks the Start button
And User waits for test start
And User clicks the Stop Test button
When User clicks the Confirm Stop Test button
And User enters the correct applicant password
And User clicks the Confirm Stop Test button
Then The Test is Over should be displayed in the Message label
And the value of the Message label should be The test is over
And The Welcome to Traffic Testing page should be displayed

"Anti-patterns" (informal)
• Interaction Heavy – Not having many business tests
• Lame – No depth or variety, no testing techniques used
• Enter Enter Click Click – Test steps are too detailed
• No Life – Missing life cycle steps of business objects
• Clueless – No clear scope for the tests
• Cocktail – Interaction tests mixed with business tests
• Over-Checking – Checks not relevant for the scope
• Sneaky Checking – Checks hidden in actions
• Action Explosion – Many actions, hardly different
• Mystery Actions – Cryptic actions, unclear what they do
• Techno – Actions and tests that look like code
– often _NOts0EasY_2REad
– but great to impress non-technical people

Example of a test module
TEST MODULE Order processing
start system
TEST CASE TC 01 Order for tablets
user password
login jdoe doedoe
window
check window exists welcome
order id cust id article price quantity
create order AB123 W3454X tablet 198.95 5
order id total
check order total AB123 994.75
. . .

Low-level, high-level, mid-level actions
• Low-level (system level) actions
– detailed interaction with the UI (or API)
– generic, do not show any functional or business logic
– examples: "click", "expand tree node", "select menu"
• High-level (application and business level) actions
– represent a business function or event fitting the scope of the test
– hides the interaction
– examples: "enter customer", "rent car", "check balance"
• Mid-level actions: auxiliary actions that represent common
sequences of low level actions
– usually to wrap a page or dialog
– enhance maintainability
– example: "enter address fields“
– complex pages could be further split up
like with the Page Object pattern
enter customer
enter address fields
enter select set . . .. . .

Tip: Provide default values in actions
ACTION DEFINITION login
name default value
argument user tester
argument password testerpw
enter login window user name # user
enter login window password # password
window control
click login window login
user password
login tamaraj tj1234
text
check message Hello Tamara
login
date payee amount
make payment 1/1/12 Gas Co. 85.00
Use login action w/ arguments Use login default values

ABT in Agile
Test Module
Definition
(optional)
Test Module Development
Interface Definition
Action Automation
Test Execution
Sprint Products
Product
Backlog
Test re-use
Automation re-use
product
owner
team
prod owner
& team
User stories
Documentation
Domain understanding
Acceptance Criteria
PO Questions
Situations
Relations
Agile life cycle
Test development
Main Level Test Modules
Interaction Test Modules
Cross over Test Modules

Using ABT in Sprints (1)
• Try to keep the testers in the same sprint as the rest of
the team
– don't let automated tests clutter up and lack behind
– note that in many environments tests and their automation are not
highest priority
• Agree on the approach:
– is testability a requirement for the software?
– do we regard tests, or some of the tests, as products?
• would potentially become part of the backlog
• would not only be up to the team which tests to create
• Just like for development, use discussions with the team
and product owners
– deepen understanding, for the whole team
– help identify items like negative, alternate and unexpected
situations
– tests can work as driver for development (TDD and ATDD)

Using ABT in Sprints (2)
• Create good starting conditions for a sprint:
– automation technology available (UI, non-UI, custom controls, graphics, ...)
– know how you will deal with data and environments
– understanding of subject matter, testing, automation, etc
• Do interface mapping by hand, using developer provided
identifications
– saves time by not having to use the viewer or other spy tools
– recording of actions (not tests) will go better
To discuss an approach, consider daily "sit down" meetings with some or
all members to coach and evaluate
 an end-of-day counterpart to the early-morning "stand up" meetings
 short and friendly, not about progress and impediments, but about practices and
experiences with them (like "what actions did you use?")
 a few meetings may suffice

Multiple System Access
System
Under Test
Action Automation
Web service
access
command
line access
Web UI
access
database
access
Test Modules

Testing in, or near, production
• In an agile and DevOps approach continuous deployment is
becoming the norm
– in particular for service based system (like web, apps, SaaS, but also client-server)
– logic in services is easier to manage and updated than distributed and variable clients
• In a DevOps approach tight cooperation between development, test
and deployment pays off
– automation testability
– testable system designs
– good test design remains a success factor
– tight integration in the build, deployment and production processes
• A/B testing can help test vary complex systems
– used not only for testing, also for trying out new features
– dividing the incoming traffic into an A and B flow (B is the new situation)
– automated tests can use the B flow
– to do this, have it well integrated in your system designs
• Continuous testing with random regression testing or monkey testing
*see also: Ken Johnston's chapter in the book of Dorothy Graham and Mark Fewster, and his keynote at StarWest 2012

A/B testing with a reverse proxy
• A/B testing means part of traffic is routed through a different server
or component (see if it works, and/or how users react)
• B could be a real-life user, a test user or also an automated test
• The strategy can be done at any component level
• Watch your test design, easy to drown in technical only
A
A
B
Reverse
Proxy
Users
Servers
A
B
newcurrent
A
B

• Passive timing
– wait a set amount of time
– in large scale testing, try to avoid passive timing altogether:
• if wait too short, test will be interrupted
• if wait too long, time is wasted
• Active timing
– wait for a measurable event
– usually the wait is up to a, generous, maximum time
– common example: wait for a window or control to appear (usually the test tool will do this for
you)
• Even if not obvious, find something to wait for...
• Involve developers if needed
– relatively easy in an agile team, but also in traditional projects, give this priority
• If using a waiting loop
– make sure to use a "sleep" function in each cycle that frees up the processor (giving the AUT
time to respond)
– wait for an end time, rather then a set amount of cycles
Active Timing

Things to wait for...
• Wait for a last control or elements to load
– developers can help knowing which one that is
• Non-UI criteria
– API function
– existence of a file
• Criteria added in development specifically for this purpose, like:
– "disabling" big slow controls (like lists or trees) until they're done loading
– API functions or UI window or control properties
• Use a "delta" approach:
– every wait cycle, test if there was a change; if no change, assume that the
loading time is over:
– examples of changes:
• the controls on a window
• count of items in a list
• size a file (like a log file)

• Should be a "must have" requirement
– first question in a development project: "how do we test this?"
• Design of the system(s) under test:
– components, tiers, services,
• Hidden identifying properties
• Hooks for timing
• White-box access to anything relevant:
– input data (ability to emulate)
– output data (what is underlying data being displayed)
– random generators (can I set a seed?)
– states (like in a game)
– objects displayed (like monsters in a game)
• Emulation features, like time-travel and fake locations
Testability, key items

Versions, environments, configurations
• Many factors can influence details of automation
– language, localization
– hardware
– version of the system under test
– system components, like OS or browser
• Test design can reflect these
– certain test modules are more general
– others are specific, for example for a language
• But for tests that do not care about the differences, the
automation just needs to "deal" with them
– shield them from the tests
minimum safe distance
from a bear is 91 meters
LOCALIZATION
converting yards to meters

Capture variations of the system under test in the actions and interface
definitions, rather than in the tests (unless relevant there).
Variation Variation Variation
"Variations"
"Master Switch"
Actions, Interface Definitions
. . .

Possible set up of variations
linked variation
keyworded variation
Specify for example in a dialog when you start an execution:

• Not every test is straightforward UI based
• Always go back to the test: what is it I want
• Express that with actions
– make them up as you go
– assume everything is possible
• For the automation side, ask for advise
– you pattern might have been seen before
Special situations

"Lead Deputy" Testing
• For "multi station" testing, when multiple machines have to
participate in a test in real-time
• For example if a supervisor needs to approve a withdrawal in a bank
teller system
• Can be "synchronized" and "parallel" (with a rendezvous point)
lead machine
playing the
"bank teller"
deputy machine
playing the
"supervisor"
acc nr amount tx id
request 123123 10,000 >> tx
name
use deputy supervisor
tx id amount
approve # tx 10,000
use lead
tx id amount
pay out # tx 10,000
lead machine
playing the
"bank teller"
...
...

Device Testing
Software Under Test
Agent
ABT
Automation
Interface
Info
Testing Host
Device
Androi
d

• Approach applicable for pictures like graphics or icons
• The tester will add a line "check picture", that includes
a "question" as one of the arguments
• While the test is executed TA keeps the recorded
pictures
• After execution the pictures are shown to a manual
testing for approval
• Once approved unchanged same pictures won't be
presented to the manual tester again in future runs
Multimedia: The "check picture"
Approach

• Automation is not as much a technical challenge as it is a test
design challenge
• Domain language based approaches like BDD and Actions can
make tests are accessible, and easier to maintain
• However, some kind of systematic approach, like the ABT method, is
necessary to keep the tests manageable
• Testing is teamwork, including automation engineers, developers
and product owners
• From a test design perspective approaches for non-UI and UI driven
testing are very similar
Summary hans@logigear.com
That’s all folks for this talk. However, I have
much more stuff, please let me know any
questions.

• Automation is not as much a technical challenge as it is
a test design challenge
• Domain language based approaches like BDD and
Actions can make tests are accessible, and easier to
maintain
• However, some kind of systematic approach, like the
ABT method, is necessary to keep the tests manageable
• From a test design perspective approaches for non-UI
and UI driven testing are very similar
Summary hans@logigear.com
That’s all folks for this talk. However, I have
much more stuff, please let me know any
questions.

You have Selenium... Now what?

More Related Content

What's hot (17)

Viewers also liked (9)

Similar to You have Selenium... Now what? (20)

More from Great Wide Open (20)

Recently uploaded (20)

You have Selenium... Now what?