Martin Gijsen - Effective Test Automation a la Carte

Effective Test Automation
à la Carte
Martin Gijsen
Test automation architect
Independent consultant
Martin@DeAnalist.nl
© 2009

Overview

Effective test automation

Domain Specific Test Languages

Three example solutions (free tools only)

Where to get the automator

Conclusion

Q & A

Test automation scenarios
2
1
time
total
cost /
effort
1. manual testing
2. effective automated testing
3. less effective automated testing
3
About 2/3 of all test
automation projects fail
sooner or later
– Brian LeSuer (testing
experience magazine,
December 2008 issue,
page 47)

Effective test automation

Record & Playback only works in case of (very)
little manual maintenance to the testware

An automated test is a program

Test automation requires software engineering

For continuity of benefits, focus on:

3 R's: Tests must be easy to write, review and revise

Maintainability and maintenance sensitivity


Domain Specific Test Languages

Used by testers to express automated tests

Instructions defined by (or with) testers

Only the essence of a test case should remain:

Natural, high abstraction level

No irrelevant interfacing details

No irrelevant tooling details

The result is low maintenance

No programming

Clear test report

A good DSTL based test case
account first name last name address
open account 1234567890 John Doe 11 Test street, Testburg
open account 1234567891 Jane Doe 27 Testability lane, Testville
account amount
deposit John 12.345,00
amount from account to account
transfer 1234 John Jane
account amount
check balance John 11.111,00
check balance Jane 1.234,00

Example instruction documentation
Name and aliases 'begin test case' and 'begin testcase'
Description Indicates the beginning of a (new) test case.
Parameters
1 The identification of the test case, preferably unique within the test.
Optional.
2 The test case description. Optional.
Pre-condition None.
Post-condition
If the previous input line was part of a test case, that test case is
closed. A new test case is opened and assigned the next
sequence number for the report(s), starting at one.
Error condition If in a procedure definition.
Example ...

Developer domain:

Scripting

Source code
Focus is on HOW to test it
Requires software engineering skills
Tester domain:

Instructions
Focus is on WHAT to test
Requires analysis skills
How does it work?
Test
automation
solution
Test
automation
solution
TestTest Test
report
Test
report
System
Under
Test
System
Under
Test
Instruction
definitions
Instruction
definitions

Three examples
Real systems and test solutions:
1. Web services based billing system
2. Web application for batch payments
3. Generic solution for payment systems
Different systems, interfaces & tooling

Web services based billing system

Interface: web services (SOAP / XML)

Interface type: synchronous messaging

Tooling: ETA Framework, Axis, Java

Multiple implementations of instructions
for phased test execution

A sample test case
url
select end point http://12.34.56.78:1234/billing
name
select wholesale customer Joy inc.
cust nr first name last name ...
create end customer 12345 John Doe ...
cust nr service ... ...
add subscription 12345 X ... ...
item nr cust nr ... ...
check billing item X12345 12345 ... ...

The design
Test engineTest engine
TestTest Test
report
Test
report
Billing
system
Billing
system
Instruction
logic
Instruction
logic
Web serv.
client
Web serv.
clientWSDLWSDL AxisAxis

Web app for payment batches

Interface: web application (J2EE)

Interface type: Graphical user interface

Tooling: ETA Framework, WebDriver, Java

A sample test case
name
set screen item name order letter account accountNr
set screen item name order letter amount amount
set screen item name order letter date desiredDate
user name
log on JohnDoe
account amount date ...
enter order letter 123456789 12345,67 today ...

The design
TestTest Test
report
Test
report
Web
application
Web
application
Instruction
logic
Instruction
logic
WebDriverWebDriver

Payment systems

Interfaces:

Iso15022 / SWIFT MT messages

Iso20022 / SWIFT MX / UNIFI messages

Interface type: asynchronous messaging

Tooling: ETA Framework, Java

Low abstraction level is raised using procedures
(not in example)

A sample (low level) test case
name
define structure msgOut
name value
set msgOut.field1 value1
set msgOut.field2 value2
connection type name
send message toSut mt103 msgOut
connection name period
receive message fromSut msgIn within 10 seconds
value 1 value 2

The design
TestTest Test
report
Test
report
Payment systemPayment system
Instruction logicInstruction logic
UNIFI protocolUNIFI protocol SWIFT protocolSWIFT protocol Incoming msgsIncoming msgs
Connection logicConnection logic

WS logicWS logic Web app logicWeb app logic Payment logicPayment logic
WS clientWS client WebDriverWebDriver
Bringing it all together
System Under TestSystem Under Test
Protocol XProtocol X Protocol YProtocol Y Incoming msgsIncoming msgs
Connection logicConnection logic

The test automation menu
1. DSTL
– Define your own
2. Test engines
– ETA Framework
– Fit / FitNesse
– STAF
– xUnit
– ...
3. Interfacing components
– Web services
– Web app
– Messaging
– Java Swing
– ...
4. Vendor lock-in (optional)

Where to get the automator

Main criteria:

Skills (programming ≠ software engineering)

Availability

Cost

Train someone from the test team

Borrow from development

Hire or reserve developer for test team

Hire external consultant

Conclusions

Test automation is software engineering
- apply the right skills to succeed

A DSTL enables testers and reduces
maintenance

Automated testing with free software is possible
for many systems and interfaces (even combined)

While every system and DSTL is different,
the basic design of test solutions can be the same

Non-programming testers and non-testing
developers can be effective at automated testing

See also

“Exploring no man's land with keyword driven
testing” - presentation at Free Test 2009

“Effective automated testing with a DSTL”-
white paper

www.DeAnalist.nl for the ETA Framework and
its documentation

Martin Gijsen - Effective Test Automation a la Carte

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Martin Gijsen - Effective Test Automation a la Carte (20)

More from TEST Huddle (20)

Recently uploaded (20)

Martin Gijsen - Effective Test Automation a la Carte