Acceptance Testing for Continuous Delivery by Dave Farley at #AgileIndia2019

Dave Farley
http://www.davefarley.net
@davefarley77
http://www.continuous-delivery.co.uk
(C)opyright Dave Farley 2015
Acceptance Testing for
Continuous Delivery

The Role of Acceptance Testing
Local Dev. Env.
Source
Repository

Artifact
Repository
Local Dev. Env.
Deployment Pipeline
Commit
Production Env.
Deployment
App.
Commit
Acceptance
Manual
Perf1
Perf2
Staged
Production
Source
Repository
Acceptance
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Manual Test Env.
Deployment
App.

Artifact
Repository
Local Dev. Env.
Deployment Pipeline
Commit
Production Env.
Deployment
App.
Commit
Acceptance
Manual
Perf1
Perf2
Staged
Production
Source
Repository
Acceptance
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Manual Test Env.
Deployment
App.
Staging Env.
Deployment
App.
Manual Test Env.
Deployment
App.
Component
Performance
System
Performance
Acceptance

What is Acceptance Testing?
Asserts that the code does what the users want.

An automated “deﬁnition of done”

Asserts that the code works in a “production-like”
test environment.

A test of the deployment and conﬁguration of a
whole system.

Provides timely feedback on stories - closes a
feedback loop.

Acceptance Testing, ATDD, BDD, Speciﬁcation by
Example, Executable Speciﬁcations.

A Good Acceptance Test is:
An Executable Specification of
the Behaviour of the System

Unit Test CodeIdea
Executable
spec.
Build Release

The Problem:
There is often a disconnect between what users
want of a system and what the software delivers.

The Problem:
Identifying user-need and designing a solution
are two VERY difﬁcult problems
Let’s Try to NOT solve both
at the same time!

The Problem:
In Most Orgs, Functional Testing is Manual

The Problem:
In Most Orgs, Functional Testing is Manual
Slow, Low-Quality,
Expensive, Unreliable,
Error Prone,
Fragile,
Results Hard to Understand,
…

The Problem:
Automated Functional Tests Often Tightly-
Coupled To SUT

The Problem:
Automated Functional Tests Often Tightly-
Coupled To SUT
Slow to Develop,
Low-Quality, Expensive,
Unreliable,
Error Prone,
Fragile,
…

The Problem:
So if there is a disconnect, how can we bridge
that gap?

The Solution:
Establish a common shared language for
expressing a user’s need

Technique:
Always capture any requirement from the
perspective of “an external user of the system”

Technique:
Avoid ANY reference to HOW the system works

Technique:
Focus on WHAT the user wants of the system

Technique:
Link Executable Specs to User Stories

123 Some Story S
As a user I want some behaviour
so that I can achieve some benefit
Acceptance
Criteria
Story Templates
▪ A result that will indicate that the benefit is achieved
▪ Another result to confirm the benefit

123 Some Story S
Acceptance
Criteria
Story Templates
Test Case 1a..n

123 Some Story S
Acceptance
Criteria
Story Templates
Test Case 1a..n
Test Case 2a..n

Technique:
Make the Team’s “Deﬁnition of Done” = Minimum
of 1 Acceptance Test for each Acceptance
Criteria

Technique:
Imagine the least technical person, who
understands the problem domain, reading the
spec - It should make sense to them!

Technique:
Avoid “Technical Stories”, Always ﬁnd the
fundamental User need

Technique:
Make each story, each speciﬁcation as small as
possible

Technique:
Make each story, each speciﬁcation as small as
possible
Small is Good!

So What’s So Hard?
• Tests break when the SUT changes (Particularly UI)
• This is a problem of design, the tests are too
tightly-coupled to the SUT!
• The history is littered with poor implementations:
• UI Record-and-playback Systems
• Record-and-playback of production data
• Dumps of production data to test systems
• Nasty automated testing products.

So What’s So Hard?
• Tests break when the SUT changes (Particularly UI)
• This is a problem of design, the tests are too
tightly-coupled to the SUT!
• The history is littered with poor implementations:
• UI Record-and-playback Systems
• Record-and-playback of production data
• Dumps of production data to test systems
• Nasty automated testing products.
Anti-Pattern!
Anti-Pattern!
Anti-Pattern!
Anti-Pattern!

Who Owns the Tests?
• Anyone can write a test
• Developers are the people that will break tests
• Therefore Developers own the responsibility to
keep them working
• Separate Testing/QA team owning automated
tests

Who Owns the Tests?
• Anyone can write a test
• Developers are the people that will break tests
• Therefore Developers own the responsibility to
keep them working
• Separate Testing/QA team owning automated
tests Anti-Pattern!

Who Owns the Tests?
Developers Own
Acceptance Tests!

Properties of Good Acceptance Tests
• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efﬁcient

Public API FIX API
Trade
Reporting
Gateway
…
“What” not “How”
API Traders
Clearing
Destination
Other
external
end-points
Market
Makers
UI
Traders

Public API FIX API
Trade
Reporting
Gateway
…
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case

Public API FIX API
Trade
Reporting
Gateway
…FIX API
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case

Public API FIX API
Trade
Reporting
Gateway
…
API Traders
Clearing
Destination
Other
external
end-points
Market
Makers
UI
Traders
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case

Public API FIX API
Trade
Reporting
Gateway
…
FIX API
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
API
External
Stubs
FIX-APIUI FIX-APIFIX-API
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case

Public API FIX API
Trade
Reporting
Gateway
…
FIX API
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
API
External
Stubs
FIX-APIUI FIX-API

Public API FIX API
Trade
Reporting
Gateway
…
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
API
External
Stubs
FIX-APIUI FIX-API

Public API FIX API
Trade
Reporting
Gateway
…
API
External
Stubs
FIX-APIUI FIX-API
Test infrastructure common to all acceptance tests

“What” not “How” - Separate Deployment from Testing
• Every Test should control its start conditions,
and so should start and init the app.
• Acceptance Test deployment should be a
rehearsal for Production Release
• This separation of concerns provides an
opportunity for optimisation
• Parallel tests in a shared environment
• Lower test start-up overhead

“What” not “How” - Separate Deployment from Testing
• Every Test should control its start conditions,
and so should start and init the app.
• Acceptance Test deployment should be a
rehearsal for Production Release
• This separation of concerns provides an
opportunity for optimisation
• Parallel tests in a shared environment
• Lower test start-up overhead
Anti-Pattern!

Test Isolation
• Any form of testing is about evaluating
something in controlled circumstances
• Isolation works on multiple levels
• Isolating the System under test
• Isolating test cases from each other
• Isolating test cases from themselves (temporal isolation)
• Isolation is a vital part of your Test Strategy

Test Isolation - Isolating the System Under Test

External System
‘A’
External System
‘C’
System Under Test
‘B’

External System
‘A’
External System
‘C’
System Under Test
‘B’
?

External System
‘A’
External System
‘C’
System Under Test
‘B’Anti-Pattern!

System Under Test
‘B’
Test Cases
Veriﬁable
Output

Test Isolation - Validating The Interfaces

External System
‘A’
External System
‘C’
System Under Test
‘B’

External System
‘A’
External System
‘C’
Test Cases
Verifiable
Output
System Under Test
‘B’
Test Cases
Verifiable
Output
Test Cases
Verifiable
Output

External System
‘A’
External System
‘C’
Test Cases
Verifiable
Output
System Under Test
‘B’
Test Cases
Verifiable
Output
Test Cases Verifiable
Output

Test Isolation - Isolating Test Cases
(Assuming multi-user systems…)
• Tests should be efﬁcient - We want to run LOTS!
• What we really want is to deploy once, and run LOTS of
tests
• So we must avoid ANY dependencies between tests…
• Use natural functional isolation e.g.
• If testing Amazon, create a new account and a new book/product for
every test-case
• If testing eBay create a new account and a new auction for every test-
case
• If testing GitHub, create a new account and a new repository for every
test-case
• …

• We want repeatable results
• If I run my test-case twice it should work both
times
Test Isolation - Temporal Isolation

times
def test_should_place_an_order(self):
self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)

times
Continuous Delivery

times
Continuous Delivery1234

times

times
• Alias your functional isolation entities
• In your test case create account ‘Dave’ in reality the in the
test infrastructure ask the application to create account
‘Dave2938472398472’ and alias.

Repeatability - Test Doubles
External System

External System
Local Interface
to External System

External System
Local Interface
to External System
Communications
to External System

External System
Local Interface
to External System
Communications
to External System
TestStub
Simulating External
System
Local Interface
to External System

External System
Local Interface
to External System
Communications
to External System
TestStub
Simulating External
System
Local Interface
to External System
Production
Test
Environm
ent
kjhaskjhdkjhkjh askjhl lkjasl dkjas lkajl ajsd
lkjalskjlakjsdlkajsld j
lkajsdlkajsldkj
lkjlakjsldkjlka laskj ljl akjl kajsldijupoqwiuepoq dlkjl iu
lkajsodiuqpwouoi la
]laksjdiuqoiwuoijds
oijasodiaosidjuoiasud
kjhaskjhdkjhkjh askjhl lkjasl dkjas lkajl ajsd
lkjalskjlakjsdlkajsld j
lkajsdlkajsldkj
lkjlakjsldkjlka laskj ljl akjl kajsldijupoqwiuepoq dlkjl iu
lkajsodiuqpwouoi la
]laksjdiuqoiwuoijds
oijasodiaosidjuoiasud
Configuration

Test Doubles As Part of Test Infrastructure
TestStub
Simulating External
System
Local Interface
to External System

TestStub
Simulating External
System
Local Interface
to External System
Public Interface

TestStub
Simulating External
System
Local Interface
to External System
Test Infrastructure
Test
Case
Test
Case
Test
Case
Test
Case
Test Infrastructure
Back-Channel
Public Interface
System
UnderTest

Language of the Problem Domain - DSL
• A Simple ‘DSL’ Solves many of our problems
• Ease of TestCase creation
• Readability
• Ease of Maintenance
• Separation of “What” from “How”
• Test Isolation
• The Chance to abstract complex set-up and scenarios
• …

• A Simple ‘DSL’ Solves many of our problems
• Ease of TestCase creation
• Readability
• Ease of Maintenance
• Separation of “What” from “How”
• Test Isolation
• The Chance to abstract complex set-up and scenarios
• …
BDD
Behaviour Driven Development

Language of the Problem Domain - External DSL
ThoughtWorks’ TWIST
|eg.Division|
|numerator|denominator|quotient?|
|10 |2 |5 |
|12.6 |3 |4.2 |
|100 |4 |33 |
FITnesse
Narrative:
In order to communicate effectively to the business some functionality
As a development team
I want to use Behaviour-Driven Development
Lifecycle:
Before:
Given a step that is executed before each scenario
After:
Outcome: ANY
Given a step that is executed after each scenario regardless of outcome
Outcome: SUCCESS
Given a step that is executed after each successful scenario
Outcome: FAILURE
Given a step that is executed after each failed scenario
Scenario: A scenario is a collection of executable steps of different type
Given step represents a precondition to an event
When step represents the occurrence of the event
Then step represents the outcome of the event
Examples:
|precondition|be-captured|
|abc|be captured |
|xyz|not be captured|
JBehave
Cucumber - Gherkin

Language of the Problem Domain - Internal DSL
EasyB - Groovy
given "an invalid zip code", {
invalidzipcode = "221o1"
}
and "given the zipcodevalidator is initialized", {
zipvalidate = new ZipCodeValidator()
}
when "validate is invoked with the invalid zip code", {
value = zipvalidate.validate(invalidzipcode)
}
then "the validator instance should return false", {
value.shouldBe false
}
My Homebrew - Java
@Channel(fixApi, dealTicket, publicApi)
@Test
public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder()
{
trading.placeOrder("instrument", "side: buy", “price: 123.45”, "quantity: 4", "goodUntil: Immediate”);
trading.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
"side: buy", "quantity: 4", "matched: 4", "remaining: 0",
"executionPrice: 123.45", "executionQuantity: 4");
}
My Homebrew - Pythonfrom acceptance_test_dsl.acc_test_dsl import AccTestDsl
class PlaceOrderTest(AccTestDsl):
def setUp(self):
AccTestDsl.setUp(self)
self.add_channel("TRADING")
self.add_end_point("MQ")
def test_should_successfully_place_market_order(self):
order = self.trading.placeOrder(symbol="ZVZZT",
orderType="Market",
qty=300, TimeInForce='IOC')
self.trading.assertOrderPlaced(order)

@Test
public void shouldSupportPlacingValidBuyAndSellLimitOrders()
{
trading.selectDealTicket("instrument");
trading.dealTicket.placeOrder("type: limit", ”bid: 4@10”);
trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0");
trading.dealTicket.dismissFeedbackMessage();
trading.dealTicket.placeOrder("type: limit", ”ask: 4@9”);
trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0");
}

@Test
{
}
@Test
{
fixAPIMarketMaker.placeMassOrder("instrument", "ask: 11@52", "ask: 10@51", "ask: 10@50", "bid: 10@49");
fixAPI.placeOrder("instrument", "side: buy", "quantity: 4", "goodUntil: Immediate", "allowUnmatched: true");
fixAPI.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
"executionPrice: 50", "executionQuantity: 4");
}

@Test
{
}
@Test
{
}
@Before
public void beforeEveryTest()
{
adminAPI.createInstrument("name: instrument");
registrationAPI.createUser("user");
registrationAPI.createUser("marketMaker", "accountType: MARKET_MAKER");
tradingUI.loginAsLive("user");
}

public void placeOrder(final String... args)
{
final DslParams params =
new DslParams(args,
new OptionalParam("type").setDefault("Limit").setAllowedValues("limit", "market", "StopMarket
new OptionalParam("side").setDefault("Buy").setAllowedValues("buy", "sell"),
new OptionalParam("price"),
new OptionalParam("triggerPrice"),
new OptionalParam("quantity"),
new OptionalParam("stopProfitOffset"),
new OptionalParam("stopLossOffset"),
new OptionalParam("confirmFeedback").setDefault("true"));
getDealTicketPageDriver().placeOrder(params.value("type"),
params.value("side"),
params.value("price"),
params.value("triggerPrice"),
params.value("quantity"),
params.value("stopProfitOffset"),
params.value("stopLossOffset"));
if (params.valueAsBoolean("confirmFeedback"))
{
getDealTicketPageDriver().clickOrderFeedbackConfirmationButton();
}
LOGGER.debug("placeOrder(" + Arrays.deepToString(args) + ")");
}

public void placeOrder(final String... args)
{
final DslParams params = new DslParams(args,
new RequiredParam("instrument"),
new OptionalParam("clientOrderId"),
new OptionalParam("order"),
new OptionalParam("side").setAllowedValues("buy", "sell"),
new OptionalParam("orderType").setAllowedValues("market", "limit"),
new OptionalParam("price"),
new OptionalParam("bid"),
new OptionalParam("ask"),
new OptionalParam("symbol").setDefault("BARC"),
new OptionalParam("quantity"),
new OptionalParam("goodUntil").setAllowedValues("Immediate", "Cancelled").setDefault("Cancelled"),
new OptionalParam("allowUnmatched").setAllowedValues("true", "false").setDefault("true"),
new OptionalParam("possibleResend").setAllowedValues("true", "false").setDefault("false"),
new OptionalParam("unauthorised").setAllowedValues("true"),
new OptionalParam("brokeredAccountId"),
new OptionalParam("msgSeqNum"),
new OptionalParam("orderCapacity").setAllowedValues("AGENCY", "PRINCIPAL", "").setDefault("PRINCIPA
new OptionalParam("accountType").setAllowedValues("CUSTOMER", "HOUSE", "").setDefault("HOUSE"),
new OptionalParam("accountClearingReference"),
new OptionalParam("expectedOrderRejectionStatus").setAllowedValues("TOO_LATE_TO_ENTER",
"BROKER_EXCHANGE_OPTION",
"UNKNOWN_SYMBOL",
"DUPLICATE_ORDER"),
new OptionalParam("expectedOrderRejectionReason").setAllowedValues("INSUFFICIENT_LIQUIDITY",
"INSTRUMENT_NOT_OPEN",
"INSTRUMENT_DOES_NOT_EXIST",
"DUPLICATE_ORDER",
"QUANTITY_NOT_VALID",
"PRICE_NOT_VALID",
"INVALID_ORDER_INSTRUCTION",
"OUTSIDE_VOLATILITY_BAND",
"INVALID_INSTRUMENT_SYMBOL",
"ACCESS_DENIED",
"INSTRUMENT_SUSPENDED"),
new OptionalParam("expectedSessionRejectionReason").setAllowedValues("INVALID_TAG_NUMBER",
“REQUIRED_TAG_MISSING",
…

@Test
{
tradingUI.showDealTicket("instrument");
tradingUI.dealTicket.placeOrder("type: limit", ”bid: 4@10”);
tradingUI.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at
tradingUI.dealTicket.dismissFeedbackMessage();
tradingUI.dealTicket.placeOrder("type: limit", ”ask: 4@9”);
tradingUI.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at
}
@Test
{
}

@Channel(fixApi, dealTicket, publicApi)
@Test
{
trading.placeOrder("instrument", "side: buy", “price: 123.45”, "quantity: 4", "goodUntil: Immediate”);
trading.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
"executionPrice: 123.45", "executionQuantity: 4");
}

Language of the Problem Domain - Evolving a DSL
• Write the test case first
• Whoever would normally write the case specifies the
language to express the idea
• Get a Developer to implement the DSL to
support the new test case
• It’s OK for the Dev to refine the language to make it
cleaner, more general - but keep it simple!
• Devs should keeping pushing the DSL to be clean of ANY
understanding of the “How”
• Think of this as designing a language. Have some rules
about what should be in and what should not.

DSL- Four Layer Structure

Test Case
(Executable
Specification)
Test Case
(Executable
Specification)
Test Case
(Executable
Specification)
Test Case
(Executable
Specification)

Domain Specific Language
Test Case
(Executable
Specification)
Test Case
(Executable
Specification)
Test Case
(Executable
Specification)
Test Case
(Executable
Specification)

Test Case
(Executable
Specification)
Test Case
(Executable
Specification)
Test Case
(Executable
Specification)
Test Case
(Executable
Specification)
Protocol
Driver
(e.g. UI)
Protocol
Driver
(e.g. API)
External
System Stub
External
System Stub

Test Case
(Executable
Specification)
Test Case
(Executable
Specification)
Test Case
(Executable
Specification)
Test Case
(Executable
Specification)
Protocol
Driver
(e.g. UI)
System Under Test
Protocol
Driver
(e.g. API)
External
System Stub
External
System Stub

Testing with Time
• Test Cases should be deterministic
• Time is a problem for determinism - There are
two options:
• Ignore time
• Control time

Testing With Time - Ignore Time
Mechanism
Filter out time-based values in your test
infrastructure so that they are ignored
Pros:
• Simple!
Cons:
• Can miss errors
• Prevents any hope of testing complex time-based
scenarios

Mechanism
Treat Time as an external dependency, like an
external system - and Fake it!
Pros:
• Very Flexible!
• Can simulate any time-based scenario, with time under the
control of the test case.
Cons:
• Slightly more complex infrastructure
Testing With Time - Controlling Time

@Test
public void shouldBeOverdueAfterOneMonth()
{
book = library.borrowBook(“Continuous Delivery”);
assertFalse(book.isOverdue());
time.travel(“+1 week”);
assertFalse(book.isOverdue());
time.travel(“+4 weeks”);
assertTrue(book.isOverdue());
}

Test Infrastructure
Test
Case
Test
Case
Test
Case
Test
Case
System
UnderTest
public void someTimeDependentMethod()
{
time = System.getTime();
}
System
UnderTest

Test Infrastructure
Test
Case
Test
Case
Test
Case
Test
Case
System
UnderTest
include Clock;
{
time = Clock.getTime();
}
System
UnderTest

Test Infrastructure
Test
Case
Test
Case
Test
Case
Test
Case
System
UnderTest
include Clock;
{
}
public class Clock {
public static clock = new SystemClock();
public static void setTime(long newTime) {
clock.setTime(newTime);
}
public static long getTime() {
return clock.getTime();
}
System
UnderTest

Test Infrastructure
Test
Case
Test
Case
Test
Case
Test
Case
System
UnderTest
include Clock;
{
}
public void onInit() {
// Remote Call - back-channel
systemUnderTest.setClock(new TestClock());
}
public void time-travel(String time) {
long newTime = parseTime(time);
// Remote Call - back-channel
systemUnderTest.setTime(newTime);
}
Test Infrastructure
Back-Channel
public class Clock {
public static clock = new SystemClock();
public static void setTime(long newTime) {
clock.setTime(newTime);
}
public static long getTime() {
return clock.getTime();
}
System
UnderTest

Test Environment Types
• Some Tests need special treatment.
• Tag Tests with properties and allocate them
dynamically:

• Some Tests need special treatment.
• Tag Tests with properties and allocate them
dynamically:
@TimeTravel
@Test
public void shouldDoSomethingThatNeedsFakeTime()
…
@Destructive
@Test
public void shouldDoSomethingThatKillsPartOfTheSystem()
…
@FPGA(version=1.3)
@Test
public void shouldDoSomethingThatRequiresSpecificHardware()
…

Production-like Test Environments

Make Test Cases Internally Synchronous

• Look for a “Concluding Event” listen for that in
your DSL to report an async call as complete
•

Example DSL level Implementation…
public String placeOrder(String params…)
{
orderSent = sendAsyncPlaceOrderMessage(parseOrderParams(params));
return waitForOrderConfirmedOrFailOnTimeOut(orderSent);
}
•

• If you really have to, implement a  
“poll-and-timeout” mechanism in your test-
infrastructure
• Never, Never, Never, put a “wait(xx)” and expect
your tests to be (a) Reliable or (b) Efﬁcient!
•

• If you really have to, implement a  
“poll-and-timeout” mechanism in your test-
infrastructure
• Never, Never, Never, put a “wait(xx)” and expect
your tests to be (a) Reliable or (b) Efﬁcient!
•
Anti-Pattern!

Scaling-Up
Artifact
Repository
Deployment Pipeline
Acceptance
Commit
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Production Env.
Deployment
App.
Source
Repository
Manual Test Env.
Deployment
App.

Scaling-Up
Artifact
Repository
Deployment Pipeline
Acceptance
Commit
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Production Env.
Deployment
App.
Source
Repository
Manual Test Env.
Deployment
App.
Deployment Pipeline
Commit
Manual Test Env.
Deployment
App.
Artifact
Repository
Acceptance Acceptance Test
Environment

Scaling-Up
Artifact
Repository
Deployment Pipeline
Acceptance
Commit
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Production Env.
Deployment
App.
Source
Repository
Manual Test Env.
Deployment
App.
Deployment Pipeline
Commit
Manual Test Env.
Deployment
App.
Artifact
Repository
Environment
AA

Scaling-Up
Artifact
Repository
Deployment Pipeline
Acceptance
Commit
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Production Env.
Deployment
App.
Source
Repository
Manual Test Env.
Deployment
App.
Deployment Pipeline
Commit
Manual Test Env.
Deployment
App.
Artifact
Repository
Environment
A
A

Scaling-Up
Artifact
Repository
Deployment Pipeline
Acceptance
Commit
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Production Env.
Deployment
App.
Source
Repository
Manual Test Env.
Deployment
App.
Deployment Pipeline
Commit
Manual Test Env.
Deployment
App.
Artifact
Repository
Environment
Test Host
Test Host
Test Host
Test Host
Test Host
A
A

Scaling-Up
Artifact
Repository
Deployment Pipeline
Acceptance
Commit
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Production Env.
Deployment
App.
Source
Repository
Manual Test Env.
Deployment
App.
Deployment Pipeline
Commit
Manual Test Env.
Deployment
App.
Artifact
Repository
Acceptance
Acceptance
Acceptance Test
Environment
Test Host
Test Host
Test Host
Test Host
Test Host
AA

Data - SUT State
• One More Desirable Property…
• We should be able to run our tests anywhere 
• Ideally we want SUT to be an a fairly ‘Neutral’
State
• Data Deﬁning SUT State Falls into 3 categories:
• Transactional Data
• Reference Data
• Conﬁguration Data

Data - SUT State
• One More Desirable Property…
• We should be able to run our tests anywhere 
• Ideally we want SUT to be an a fairly ‘Neutral’
State
• Data Deﬁning SUT State Falls into 3 categories:
• Transactional Data
• Reference Data
• Conﬁguration Data
Use Prod
Data
Generate
in Scope of
Test
Use
Versioned
Test Data

Anti-Patterns in Acceptance Testing

• Don’t use UI Record-and-playback Systems

• Don’t Record-and-playback production data. This has a role, but it is NOT
Acceptance Testing

Acceptance Testing
• Don’t dump production data to your test systems, instead deﬁne the absolute
minimum data that you need

Acceptance Testing
• Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very
sceptical about them. Start with YOUR strategy and evaluate tools against that.

Acceptance Testing
• Don’t have a separate Testing/QA team! Quality is down to everyone - Developers
own Acceptance Tests!!!

Acceptance Testing
• Don’t let every Test start and init the app. Optimise for Cycle-Time, be efﬁcient in
your use of test environments.

Acceptance Testing
• Don’t include Systems outside of your control in your Acceptance Test Scope

Acceptance Testing
• Don’t include Systems outside of your control in your Acceptance Test Scope
• Don’t Put ‘wait()’ instructions in your tests hoping it will solve intermittency

Tricks for Success

Tricks for Success
• Do Ensure That Developers Own the Tests

Tricks for Success
• Do Focus Your Tests on “What” not “How”

Tricks for Success
• Do Think of Your Tests as “Executable Speciﬁcations”

Tricks for Success
• Do Make Acceptance Testing Part of your “Deﬁnition of Done”

Tricks for Success
• Do Keep Tests Isolated from one-another

Tricks for Success
• Do Keep Your Tests Repeatable

Tricks for Success
• Do Use the Language of the Problem Domain - Do try the DSL approach, whatever
your tech.

Tricks for Success
your tech.
• Do Stub External Systems

Tricks for Success
your tech.
• Do Test in “Production-Like” Environments

Tricks for Success
your tech.
• Do Make Instructions Appear Synchronous at the Level of the Test Case

Tricks for Success
your tech.
• Do Test for ANY change

Tricks for Success
your tech.
• Do Test for ANY change
• Do Keep your Tests Efﬁcient

Further Exploration
• http://en.wikipedia.org/wiki/Behavior-driven_development
• http://www.thoughtworks.com/insights/blog/speciﬁcation-example
• https://cukes.info/
• http://jbehave.org/
• http://www.agiledata.org/essays/databaseTesting.html
• http://www.red-gate.com/
• http://www.dbmaestro.com/
• https://www.youtube.com/watch?v=pw_RBblVny4

Q&A
Dave Farley
http://www.davefarley.net
@davefarley77

Acceptance Testing for Continuous Delivery by Dave Farley at #AgileIndia2019

More Related Content

What's hot (20)

Similar to Acceptance Testing for Continuous Delivery by Dave Farley at #AgileIndia2019 (20)

More from Agile India (20)

Recently uploaded (20)

Acceptance Testing for Continuous Delivery by Dave Farley at #AgileIndia2019