Exploring No Mans Land with Keyword-Driven Testing
- 1. Exploring no man's land with keyword driven testing Martin Gijsen, M.Sc. Martin@DeAnalist.nl Free Test Conference March 23, 2009
- 2. Presentation overview ● Automated testing ● When (not) to use Record & Playback ● Automated testing as no man's land ● Keyword driven testing ● The process ● Where to get the automator
- 3. Automated (functional) testing ● A computer performs the test ● Fast and consistent ● Potential benefits: ● Time (to market) / cost / quality ● More effective use of (test) resources ● More challenged (= happier) testers ● Accurate and up2date test status ● Continuity ● Business goals are drivers
- 4. The Record & Playback approach ● Record test actions, replay as needed ● Checks are added to scripts ● Fundamental problem is maintenance ● Software evolves until it is decommissioned ● Maintainability requires structure and abstractions ● Recorded test cases are long, unstructured and low level programs ● Maintenance sensitivity is high ● Maintainability is low
- 5. So why is it still used? ● Automated testing is often the job of a tester ● Most testers are: ● willing to record, but ● unwilling or unable to create good automation solutions ● Vendor sales pitch is not always clear on this
- 6. Result and conclusion It is often initially unclear what effective automated testing requires 2 out of every 3 TA projects fails sooner or later – Brian LaSuer Use R&P only in case of little manual maintenance
- 7. What about the developers? ● Test automation is software engineering ● Developers have the skills to create a good automation solution ● Many developers are unwilling or unable to create good test sets ● … or automate test cases that testers wrote
- 8. Welcome to no man's land ● Most testers cannot or will not automate well ● Most developers cannot or will not test well ● Automated testing is like a no man's land in between testing and development ● Is automated testing doomed?
- 9. Exploring no man's land Testing Automated testing Development
- 10. Exploring no man's land Testing Automated testing = analysis (what to test) + automation (how to test) Development
- 11. What is in between? Testing Test analysis ? Test automation Development
- 12. No more no man's land? Testing Test analysis Test automation Development
- 13. Keywords – The DS(T)L ● Domain Specific (Test) Language ● Used to express test cases ● Defined by (or with) the testers ● Specific to the application under test ● No programming ● High level
- 14. Bad example select (“a box”); type (“some text”); click (“OK”); ...
- 15. Better example create account;1234567890;John;Doe create account;2345678901;Jane;Doe deposit;1234567890;12345,67 transfer;1234,56;1234567890;2345678901 check balance;1234567890;11111,11 check balance;2345678901;1234,56
- 16. Good example Account First name Last name create account 1234567890 John Doe create account 2345678901 Jane Doe Account Amount deposit John 12345,67 Amount From To transfer 1234,56 John Jane Account Amount check balance John 11111,11 check balance Jane 1234,56
- 17. Keywords / Instructions ● Often have arguments ● No (irrelevant) test execution details ● No (irrelevant) interfacing details ● No tooling details ● Only the essence of a test case remains ● Easy to read, write and maintain test cases ● Low maintenance sensitivity
- 18. Sample instruction documentation Name and aliases 'begin test case' and 'begin testcase' Description Indicates the beginning of a (new) test case. Parameters Two parameters, both optional 1 The identification of the test case, printable characters, preferably unique within the test. Optional. 2 The test case description, printable characters. Optional. Validity Anywhere except inside a procedure definition. Pre-condition None. Post-condition If the previous input line was part of a test case, that test case is closed. A new test case is opened and assigned the next sequence number for the report(s), starting at one. Error condition If in a procedure definition.
- 19. Automating (basic) Test Test Automation report Application under test
- 20. Automating (more detailed) Engine Test Test report Instructions Interfacing Application under test
- 21. The ETA Framework 'engine' ● Can read from and report to a spreadsheet ● Supports test cases, variables and procedures ● Implement keywords in Java ● Use any Java library (Abbot, WebDriver, MQ, generated web service client, ...) ● Documented, actively used and developed ● Freeware
- 22. The process (waterfall) Test cases Keyword defs Automate Test Requirements Product development
- 23. The process (agile) Test cases Keyword defs Requirements Automate Test Product development Test cases Keyword defs Requirements Automate Test Product development Test cases Keyword defs Requirements Automate Test Product development
- 24. Where to get the automator ● Main criteria (for continuity): ● Skills ● Availability ● Cost ● Educate someone from test team ● Borrow from development team ● Hire or reserve developer for test team ● Hire external consultant
- 25. Summary ● Automated testing can help achieve your goals ● Avoid Record & Playback ● Good keywords help: ● The tester can focus on the testing ● The developer can focus on developing ● Easy to write test cases and little maintenance ● A good test engine also helps ● Use skilled automators ● The automators must be available when needed
- 26. Questions ?
- 27. Automation design Engine Test Test report Instructions Interfacing Application under test
- 28. Design for a telephony switch Engine Instructions (C++) Existing library (iTCL) Telephony switch
- 29. Design for a voice portal Engine Instructions (C++) Tool library Tool Voice portal
- 30. Design for web services (1) Engine Generic web service instructions (Java) Generated web service client (Java) Web services
- 31. Design for web services (2) Engine Specific web service instructions (Java) Generated web service client (Java) Web services
- 32. Design for a payments system Engine Iso20022 protocol SWIFT MT protocol Web service Web instructions instructions Java) instructions (Java) instructions (Java) (Java) Generated web service Web driver Messaging library (Java) client (Java) library (Java) SEPA banking system