![info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
21
Stable application myth
• can’t start automating until the application (or
the GUI) is stable
– throw-back to testing attitudes 30 years ago?
• testing comes at the end?
• testing is (only) execution?
• testing is more than testing [execution]
• automation is more than execution
• design your automation early, ready to run
when anything is ready to test
22
• test automation pyramid
– Mike Cohn, Lisa Crispin (Ch 1)
– more unit/component tests
– fewer GUI tests
• instability an opportunity rather than a problem
– build flexibility & robustness in your automation
against common types of changes
• which aspects are stable? (GUI often late)
– execute tests for stable parts when ready
– build (GUI) tests now but with detail abstracted out
When to write automated tests](https://image.slidesharecdn.com/k1-graham-171205201555/85/Blunders-in-Test-Automation-13-320.jpg)
Blunders in Test Automation
- 1. K1 Keynote 10/18/2017 8:30:00 AM Blunders in Test Automation Presented by: Dorothy Graham Software Test Consultant Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐268-‐8770 ·∙ 904-‐278-‐0524 - info@techwell.com - https://www.techwell.com/
- 2. Dorothy Graham Software Test Consultant In software testing for more than forty years, Dorothy Graham is coauthor of four books – Software Inspection, Software Test Automation, Foundations of Software Testing, and Experiences of Test Automation – and is currently working with Seretta Gamba on a test automation patterns wiki. A popular and entertaining speaker at conferences and seminars worldwide, Dot has attended STAR conferences since the first one in 1992. She was a founding member of the ISEB Software Testing Board and a member of the working party that developed the ISTQB Foundation Syllabus. Dot was awarded the European Excellence Award in Software Testing in 1999 and the first ISTQB Excellence Award in 2012. Learn more about Dot at DorothyGraham.co.uk.
- 3. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 1 Test Automation Blunders Prepared and presented by Dorothy Graham email: info@dorothygraham.co.uk Twitter: @DorothyGraham www.DorothyGraham.co.uk www.TestAutomationPatterns.org © Dorothy Graham 2017 2 Blunder • from old Norse word “blundra” – meaning “to shut one’s eyes” • now means – mistake caused by ignorance, carelessness – or not thinking things through • people blunder when they don’t see or understand
- 4. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 3 Contents • Test automation blunders – Testing-Tools-Test – Who needs GPS? – Silver Bullet – The Wrong Thing – How hard can it be? – Stable Application Myth – Project / Not Project – Inside the Box – Isolationism • Conclusion Twitter: @DorothyGraham 4 Testing-Tools-Test • blunder: thinking that tools actually do testing – most important, at the root of other blunders • shouldn’t have been called “testing tools” • better names: – “tester assistance tools” – “check-running tools” – “test execution tools” – “test running tools” – “partial-support-for-some-aspects-of-test- execution-and-comparison tools”
- 5. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 5 People (testers) vs tools • what do people do? – think, evaluate, assess, decide, observe, interpret – recognize patterns, have new ideas, find bugs – make mistakes • what do tools do? – they just run stuff - whatever they’ve been programmed to execute (including bad tests) – intelligence level: zero Get tools to do what computers do best, get testers to do what people do best 6 Contents • Test automation blunders – Testing-Tools-Test – Who needs GPS? – Silver Bullet – The Wrong Thing – How hard can it be? – Stable Application Myth – Project / Not Project – Inside the Box – Isolationism • Conclusion Twitter: @DorothyGraham
- 6. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 7 Who needs GPS? • if you don’t know where you are going, any road will do (Lewis Caroll) • where are you going with your automation? – testing and automation are different activities – different activities require different objectives • good objectives for testing? – find bugs, gain confidence, investigate • good objectives for automation? – Hint: they shouldn’t be the same! 8 What finds most bugs? regression tests exploratory testing likelihood of finding bugs most often automated What is usually automated?
- 7. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 9 Automation success = find lots of bugs? • tests find bugs, not automation • automation is a mechanism for running tests • the bug-finding ability of a single test is not affected by the manner in which it is executed • “find bugs” can be a dangerous objective – especially for regression automation! Automated tests Manual Scripted Exploratory Fix Verification Experiences of Test Automation, Ch 27, p 503, Ed Allen & Brian Newman 10 fast testing slow testing Effectiveness Low High EfficiencyManual testing Automated Efficiency and effectiveness poor fast testing poor slow testing goodgood greatest benefit not good but common worst better
- 8. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 11 Contents • Test automation blunders – Testing-Tools-Test – Who needs GPS? – Silver Bullet – The Wrong Thing – How hard can it be? – Stable Application Myth – Project / Not Project – Inside the Box – Isolationism • Conclusion Twitter: @DorothyGraham 12 Silver bullet solution: get the right tool • no such thing as “the right tool” or “best tool” – what’s “the best car”? poor benefits low cost good benefits high cost good benefits low cost poor benefits high cost benefits cost budget investment in good automation good benefits moderate cost commercial tools? open source tools?
- 9. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 13 Silver bullet: success is automatic • automation is (much) more than just a tool • it takes time and effort to succeed – building good automation is a learning process • management support is critical – high level managers need to understand automation capability & limitations, and have realistic expectations and budget – “people issues” – people use the automation, people develop the automation 14 Contents • Test automation blunders – Testing-Tools-Test – Who needs GPS? – Silver Bullet – The Wrong Thing – How hard can it be? – Stable Application Myth – Project / Not Project – Inside the Box – Isolationism • Conclusion Twitter: @DorothyGraham
- 10. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 15 Automate x% of tests? manual tests automated tests new approaches, e.g. monkey testing, HiVAT* manual tests automated (% manual) tests (& verification) not possible to do manually tests not automated yet *High Volume Automated Testing See http://kaner.com tests that shouldn’t be automated (users, colours, captcha, too long) 16 Testware architecture – poor architecture gives high maintenance cost • most frequent cause of abandoned automation / shelfware – two layers of abstraction • technical: for long life • human: for wide use – using the tool’s architecture ties you to that tool (version) Testers Test Execution Tool runs scripts HL Keywords Structured Scripts testware architecture write tests (in DSTL) Ch 5
- 11. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 17 Contents • Test automation blunders – Testing-Tools-Test – Who needs GPS? – Silver Bullet – The Wrong Thing – How hard can it be? – Stable Application Myth – Project / Not Project – Inside the Box – Isolationism • Conclusion Twitter: @DorothyGraham 18 How hard can it be? • different activities require different skills • classic blunder: let the testers automate – automating without automation skills? • newer blunder: let the automators write tests – testing without testing skills? • if testers are automators è a conflict of interest – do you run tests or do you automate tests? – automation is better long-term, BUT – deadline pressure pushes you back into manual testing
- 12. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 19 Tools will replace testers? • “we can reduce the number of testers once we have the tool” – what are your testers like? • mindless morons, or • intelligent investigators? – need more skills, not fewer – automation can free testers to do more test design, exploratory testing • and find more bugs – tools don’t replace testers, they support them 20 Contents • Test automation blunders – Testing-Tools-Test – Who needs GPS? – Silver Bullet – The Wrong Thing – How hard can it be? – Stable Application Myth – Project / Not Project – Inside the Box – Isolationism • Conclusion Twitter: @DorothyGraham
- 13. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 21 Stable application myth • can’t start automating until the application (or the GUI) is stable – throw-back to testing attitudes 30 years ago? • testing comes at the end? • testing is (only) execution? • testing is more than testing [execution] • automation is more than execution • design your automation early, ready to run when anything is ready to test 22 • test automation pyramid – Mike Cohn, Lisa Crispin (Ch 1) – more unit/component tests – fewer GUI tests • instability an opportunity rather than a problem – build flexibility & robustness in your automation against common types of changes • which aspects are stable? (GUI often late) – execute tests for stable parts when ready – build (GUI) tests now but with detail abstracted out When to write automated tests
- 14. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 23 Contents • Test automation blunders – Testing-Tools-Test – Who needs GPS? – Silver Bullet – The Wrong Thing – How hard can it be? – Stable Application Myth – Project / Not Project – Inside the Box – Isolationism • Conclusion Twitter: @DorothyGraham 24 Project / not project • not thinking of automation as a project – doesn’t need funding, resourcing – just do it in your “spare time” • thinking that automation is (just) a project – when will automation be finished? (wrong question!) – needs on-going continuous improvement – refactoring at regular intervals • project to start automation, then continued support
- 15. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 25 Contents • Test automation blunders – Testing-Tools-Test – Who needs GPS? – Silver Bullet – The Wrong Thing – How hard can it be? – Stable Application Myth – Project / Not Project – Inside the Box – Isolationism • Conclusion Twitter: @DorothyGraham 26 Automated tests/automated testing Select / identify test cases to run Set-up test environment: • create test environment • load test data Repeat for each test case: • set-up test pre-requisites • execute • compare results • log results • analyse test failures • report defect(s) • clear-up after test case Clear-up test environment: • delete unwanted data • save important data Summarise results Automated tests Select / identify test cases to run Set-up test environment: • create test environment • load test data Repeat for each test case: • set-up test pre-requisites • execute • compare results • log results • clear-up after test case Clear-up test environment: • delete unwanted data • save important data Summarise results Analyse test failures Report defects Automated testing Automated processManual process
- 16. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 27 Contents • Test automation blunders – Testing-Tools-Test – Who needs GPS? – Silver Bullet – The Wrong Thing – How hard can it be? – Stable Application Myth – Project / Not Project – Inside the Box – Isolationism • Conclusion Twitter: @DorothyGraham 28 Isolationism: isolated from • change: encasing your first efforts in stone – automated tests benefit from review and refactoring • realism: optimism as a strategy for automation – automated tests need to be tested and have bugs – realistic expectations • managers: not making automation benefits visible – to the people who matter, in a way that communicates • developers: not collaborating – design for automated testability, help developers • the wider world: not seeking existing knowledge – books, wiki*, articles, blogs, discussions, events *TestAutomationPatterns.wikispaces.com
- 17. info@dorothygraham.co.uk © Dorothy Graham 2017 www.DorothyGraham.co.uk www.TestAutomationPatterns.org 29 Conclusion: see how these are wrong • Summary of test automation blunders – Testing-Tools- DON’T –Test they just run stuff – Who needs GPS? automation needs direction – NO Silver Bullet needs time and effort – The Wrong Thing no testware architecture, just manual tests – How hard can it be? needs different skills – Stable Application Myth opportunity for flexibility – Project at times / Not just a Project ongoing, refactor – Inside the Box automation is more than automation – Isolationism collaborate, experiment, read, learn 30 Thank you! • More information: • downloads www.DorothyGraham.co.uk – articles and papers • email info@DorothyGraham.co.uk • blog http://dorothygraham.blogspot.com – including automation, certification • twitter – @DorothyGraham • TestAutomationPatterns.org – free wiki of automation advice, with Seretta Gamba eBook available at informit.com/swtest ing Save 35% with discount code SWTESTING