The Broken Promise of Test Automation: Why are we still hand-cranking tests?

© Curiosity Software Ireland Ltd. 2020
The Broken Promise of Test
Automation
Why are we still hand-cranking tests?
Moderator: Tom Pryce
Communication Manager, Curiosity Software Ireland
Thomas.Pryce@Curiosity.Software
@TomTestsToo
Daniel Howard
Senior Researcher, Bloor
DHoward@bloorresearch.com
Linkedin.com/in/DanielStLegerHoward
James Walker
Director of Technology, Curiosity.
James.Walker@Curiosity.Software
@CuriositySoft
Huw Price
Managing Director, Curiosity
Huw.Price@Curiosity.Software
@DataInventor

The Broken Promise of Test
Automation
1. Daniel Howard: Ensuring application quality with test automation
2. Huw Price: What next for Automation in Testing?
3. James Walker, Demo: Some next steps for test automation
4. Questions and discussion

Ensuring application quality
with test automation
Daniel Howard
Senior Analyst,
Information Management and DevOps
NOVEMBER 2020

Confidential © Bloor Research 2020 Evolution is Essential
Agenda
Why test automation?
How do you make test automation work?
Types of automation
– A note on AI and machine learning
– Test design automation
– Test data management and automation
– Service virtualisation
– Test execution automation
Testing in perpetuity
Conclusion

Why test automation?
Why test at all?
– Test coverage is not an end in itself
– We’re really aiming to create high quality applications
– Hence effective testing must work towards this end
With that in mind, a strictly manual testing solution
is not fit for purpose
– Slow, expensive, biased, difficult to maintain, etc.
Test automation can address these issues
– But only if it’s implemented effectively

Comprehensive and continuous testing: you must automate
from beginning to end
– Any gaps can (will) create bottlenecks
– This means that every part of your testing process must integrate together seamlessly
Continuous testing throughout the development life cycle (original image courtesy of Nevercode)

Maintainability:
your testing suite must be kept up-to-date
Measurability:
every test should measurably improve the quality of
your applications or system
– Don’t waste time on unnecessary tests

Types of test automation
Test automation can (and should) drive every part
of the testing process
Traditionally limited to test execution automation
But can be much more than that, for instance:
– Test design automation
– Test data management and automation
– Service virtualisation

A note on AI and machine learning
AI and machine learning can be important tools for building
automation capabilities, both for testing and in general
However!
– They don’t provide automation in and of themselves
– They are helpful, but not necessary for test automation
– They are becoming buzzwords
Don’t mistake AI for automation
Focus on how much automation is provided in practice,
not whether a given feature is AI-enabled

Test design automation
Automate testing at a high level using model based testing
Model your system, then leverage it to generate
an optimised suite of test cases and/or scripts
What counts as ‘optimised’ ?
How do you build your model?
Automated change management
Ease of use and collaboration
Test design automation in Curiosity SoftwareTest Modeller

Test data management and automation
You can’t test without test data
Production data sets are too large and often sensitive
Traditional approach: subset and mask
Newer approaches
– Generate synthetic data that imitates your real data
– Create and distribute virtualised copies of your production data
Slow or difficult test data management can cause a bottleneck
– hence test data automation

Service virtualisation
The systems you’re testing don’t exist in vacuums:
they interact with services both internal and external
These services will suffer from downtime
and may charge based on use
Service virtualisation allows you to test with virtual services
– Lightweight, durable, dependable

Service virtualisation
Service virtualisation infrastructure (original image courtesy ofTraffic Parrot)

Test execution automation
Manage and run your test scripts
Moreover, different types of testing
– API testing
– Performance and load testing
– Non-functional testing generally
Ideally, all types of test should be generated and executed
automatically

Testing in perpetuity
Maintainability
Monitoring
– Identify issues in real-time
– Measure impact of testing efforts
Ongoing development
– Monitor trends to drive improvement

Conclusion
Testing exists to improve the quality of your applications in a real,
measurable sense (rather than as a box-ticking exercise)
Manual testing is woefully inadequate for this
(although it’s still better than nothing)
Automated testing is far better if it lives up to its potential
This potential can only be realised if your testing is continuous,
comprehensive, maintainable and measurable

What next for automation in
testing?
Huw Price, Managing Director, Curiosity Software

Ask yourself: What are the problems
worth solving?
Changes to tools and techniques should be driven by value. What are the
most pressing challenges undermining the value of testing today?
1. Test creation remains manual.
2. Manual test case design cannot achieve sufficient coverage for complex systems.
3. Overtesting.
4. Test data destabilises automation and creates bottlenecks.
5. Brittle tests break when the system changes.
6. Automation code is not easily re-usable.

An algorithm is not AI:
Further questions to ask in the face of AI/ML
1. Does my organisation have sufficient data to ensure that AI/ML will deliver value?
2. Are our technologies connected with value flowing?
3. How can I add the new technology to my existing and evolving estate?
4. How are we targeting and measuring testing?
− Lots of newer technology is great at identifying what we could test. But how do we know what we should?
5. Are we delivering meaningful results?
− Is it really a bug? How ‘risky’ is it? How do we fix it?

A step forward?
We need to identify technologies that:
1) Will deliver clear and measurable value.
− Do they solve the key challenges in testing today?
2) Are implementable today.
− They must build on or existing processes and tech (not just idealized
green fields).

Technologies that you can introduce today
Extending automation beyond test execution:
1. Automation and optimisation either side of test execution – test case design, test data automation.
2. Traceability automates maintenance – update a single source of truth, changes ripple across linked assets.
Technologies to move towards “intelligence” in testing
3. Self-healing page objects – multiple dynamic identifiers run tests after UI changes, updating invalid
identifiers.
4. Fast Modelling – A catalogue of rules models equivalence classes for known elements, ‘learning’ over time.
5. Honing tests based on recent defects – Self-generating tests circle a defect, automating root cause analysis.
6. Dynamic Tests that are iterative, feature driven, exhaustive. Automatically change the level of testing on
components through development.

Demo

Carry on the conversation?
• We’d love to chat further. Book a web meeting at
calendly.com/curiosity-james
• Want to try the technologies you’ve seen today? Visit Testmodeller.io
for a free trial.
• Follow @CuriositySoft and find us on LinkedIn
• Email us on James.Walker@Curiosity.Software or

The Broken Promise of Test Automation: Why are we still hand-cranking tests?

More Related Content

What's hot (20)

Similar to The Broken Promise of Test Automation: Why are we still hand-cranking tests? (20)

More from Curiosity Software Ireland (12)

Recently uploaded (20)

The Broken Promise of Test Automation: Why are we still hand-cranking tests?

Editor's Notes