Model-based Testing: Taking BDD/ATDD to the Next Level
19 likes7,877 views
The document discusses model-based testing (MBT) and its relationship with behavior-driven development (BDD) and acceptance test-driven development (ATDD), emphasizing the need for more models and less testing to effectively manage the complexities of software development. It highlights the limitations of traditional testing methods and advocates for a systematic, model-driven approach that can generate extensive test suites more efficiently. Additionally, it outlines the processes and tools involved, including the use of SpecFlow for generating test code and exploring software behavior.
![1. Create Test
Model Actions
• [Rule] tags a
method for
exploration
• Requires defines
a exploration
precondition –
must be true to
explore
Model Driven Development: Taking BDD/ATDD to the Next Level
39](https://image.slidesharecdn.com/taking-bdd-to-the-next-level-140227131354-phpapp01/85/Model-based-Testing-Taking-BDD-ATDD-to-the-Next-Level-40-320.jpg)
Model-based Testing: Taking BDD/ATDD to the Next Level
- 1. Model-based Testing: Taking BDD/ATDD to the Next Level Robert V. Binder System Verification Associates rvbinder@sysverif.com #MoreModelsLessTests Chicago Quality Assurance Association February 25, 2014
- 2. The Incredible Vastness of Testing ModelBased Testing Eaten Alive by the Testing Backblob Tell Me A Story More Models, More Tests Less Models, Less Tests Model Driven Development: Taking BDD/ATDD to the Next Level 1
- 3. The Incredible Vastness of Testing
- 4. Saved by Revealing Sub-domains • Grains of sand to fill the Grand Canyon ~ 1.68 × 1025 ~ 10 septillion • Trivial Program – – – – Data space: 21024 Sequences: 21024 Execution space: 22048 1.34 × 10616 Large subsets of any program’s execution space are revealing: if a bug exists in any member of that space and it is executed, an observable failure results. But, we rarely know exactly which sub-domains are revealing until we test. Model Driven Development: Taking BDD/ATDD to the Next Level 3
- 5. It’s release time. Do you know where your bugs are? Aerospace Medical IT 0.7 0.8 Other Agile 1.9 1.8 4.4 5.0 6.0 6.0 Mobile Best In Class Average bugs per thousand equivalent source code lines, post-release 4.7 Capers Jones’ statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP . Model Driven Development: Taking BDD/ATDD to the Next Level 4
- 6. It’s release time. Do you know where your bugs are? Aerospace Medical IT Best In Class Other 325 375 Agile Predicted total latent bugs, 500 KLOC Java 925 875 2511 2202 2350 Capers Jones’ statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP Model Driven Development: Taking BDD/ATDD to the Next Level 5
- 7. Driven to Distraction • Test Driven Development – TDD • Behavior Driven Development – BDD • Acceptance Test Driven Development • What is it? • How does it work? • What can it do for me? – ATDD • Model-based Testing – MBT Model Driven Development: Taking BDD/ATDD to the Next Level 6
- 8. Eaten Alive by the Testing Backblob EATEN ALIVE BY THE TESTING BACKBLOB
- 9. Where does the time go? Total Number of Test Cases Sprint 1 Sprint 2 Sprint 3 Sprint 4 } Tests/testing for new features Existing manual Test Cases not executed Available Dev/Test Time Existing automated Test Cases not maintained Model Driven Development: Taking BDD/ATDD to the Next Level 8
- 10. The Attack of the BackBlob http://www.youtube.com/watch?v=GODDLgM1gKo
- 11. Tell Me a Story TELL ME A STORY
- 12. The Chat Server • • • • • Logon List active users Post a message Broadcast message Logoff !!! Client Endpoint Client Endpoint Model Driven Development: Taking BDD/ATDD to the Next Level Chat Server 11
- 13. TDD – BDD – ATDD • Test Driven Development • Behavior Driven Development • Acceptance Test Driven Development • Programming strategies – Test suites are scaffolding – Test each method/feature at least once – Rerun after any add/change • Not testing strategies Model Driven Development: Taking BDD/ATDD to the Next Level 12
- 14. The TDD Red-Green Cycle • Imagine a class and its responsibilities • Code its API only Logon Logon GetStatus GetStatus Logoff Logoff Model Driven Development: Taking BDD/ATDD to the Next Level 13
- 15. The Red-Green Cycle • Imagine a class and its responsibilities • Code its API only TestLogon • Write and run a simple test – it should fail Logon GetStatus Logoff Model Driven Development: Taking BDD/ATDD to the Next Level 14
- 16. The Red-Green Cycle • Imagine a class and its responsibilities • Code its API only TestLogon • Write and run a simple test – it should fail Logon GetStatus • Code implementation • Rerun all tests – expect pass Model Driven Development: Taking BDD/ATDD to the Next Level Logoff 15
- 17. The Red-Green Cycle • Imagine a class and its responsibilities • Code its API only • Write and run a simple test – it should fail TestLogon Logon TestGetStatus GetStatus • Code implementation • Rerun all tests – expect pass Logoff • Repeat until feature complete Model Driven Development: Taking BDD/ATDD to the Next Level 16
- 18. The Red-Green Cycle • Imagine a class and its responsibilities • Code its API only • Write and run a simple test – it should fail TestLogon Logon TestGetStatus GetStatus • Code implementation • Rerun all tests – expect pass TestLogoff Logoff • Repeat until feature complete Model Driven Development: Taking BDD/ATDD to the Next Level 17
- 19. TDD Limitations Process, Activities Components, Test Objects • No explicit linkage for user/customer needs, requirements, or stories • Gaps – UI automation – Feature testing Model Driven Development: Taking BDD/ATDD to the Next Level 18
- 20. Behavior Driven Development Process, Activities Components, Test Objects Features, Scenarios Feature Files Scenario A *DD Tool Scenario B Scenario C Test Objects *Unit Components Model Driven Development: Taking BDD/ATDD to the Next Level 19
- 21. Acceptance Test Driven Development Process, Activities Features, Scenarios UI Test Cases Feature Files Scenario A *DD Tool Scenario B Scenario C Test Scripts Test Objects Test Tooling *Unit System Under Test Model Driven Development: Taking BDD/ATDD to the Next Level 20
- 22. Features and Scenarios Feature: story Scenario: instance of a story Steps: Given: pre-condition When: trigger Then: verification condition Feature: Logon In order to use the chat server, As a validated chat client, I want to submit my credentials and establish a session Scenario: Submit Good Given A connection to When I send the Login Then I should see the Model Driven Development: Taking BDD/ATDD to the Next Level Credentials the server is open message "Hi Chatter" message 21
- 23. Overview Feature Files Scenario A SpecFlow Scenario B Scenario C Test Objects Nunit Visual Studio Components http://www.specflow.org • Enter features, scenarios • Generates skeleton test code for each step – Extracts step names – Can run immediately • Developer adds step code • Test run reports • CI integration, more Model Driven Development: Taking BDD/ATDD to the Next Level 22
- 24. 1. Create a Feature File • SpecFlow generates a stub file • Edit as necessary • One feature • 1:N scenarios Model Driven Development: Taking BDD/ATDD to the Next Level 23
- 25. 2. Enter Scenario Steps • Given – Precondition – Data State • When – Trigger – API call, GUI action • Then – Verification conditions (expected results) Model Driven Development: Taking BDD/ATDD to the Next Level 24
- 26. 3. Inspect Skeleton Test Code • SpecFlow parses feature file • Generates skeleton test code for each scenario and step Model Driven Development: Taking BDD/ATDD to the Next Level 25
- 27. 3. Add Test Case Code • User adds more code as needed to implement each test case Model Driven Development: Taking BDD/ATDD to the Next Level 26
- 28. 4. Run Test Suite • The generated test code can be run immediately • Supports the redgreen cycle Model Driven Development: Taking BDD/ATDD to the Next Level 27
- 29. SpecFlow • BDD for Visual Studio • Link Feature Files to test code • Generates partial test code Feature File contains statements in Gherkin – Keywords + natural language • • Generates test code skeleton from steps • Feature File 1:N Steps per Scenario • Feature 1:N Scenarios per Feature File Steps may be parameterized Steps Rows • Variables and variable bindings in Rows – Scenarios – Test code generation using regular expression patterns Integration with TFS, CI, … Model Driven Development: Taking BDD/ATDD to the Next Level 28
- 30. Modelbased Testing THE VERY MODEL OF A MODERN …
- 31. What is a Model? • Focus • Smaller, simpler • In IT systems – Behavior – Structure – Content Model Driven Development: Taking BDD/ATDD to the Next Level 30
- 32. IT Models Process, Activities Behavior, Structure, Content Model Driven Development: Taking BDD/ATDD to the Next Level Technology 31
- 33. IT Models • Notional (design) • As built (documentation) • Test Models – Similar to notional and as-built – Unique aspects
- 34. Test Models • Focus on aspects that support testing goals • Behavior – Sequence and time • Transformation – Input, output function • Verification (oracle) • Coverage • Risk Management Model Driven Development: Taking BDD/ATDD to the Next Level 33
- 35. Chat Server • Logon Chat Server – Must be first – Can’t be repeated Logon GetStatus Logoff • GetStatus – Only after logon – Returns number of calls since logon • Logoff – Only after logon • Typical RFC status codes Model Driven Development: Taking BDD/ATDD to the Next Level 34
- 36. Simple Test Model Logoff nc = 0 GetStatus System Under Test (SUT) Logon Logon Logon GetStatus assert (200) assert (401) assert (401) assert (401) Logoff nc++ GetStatus Logoff assert (200) Logoff assert (200) Model Driven Development: Taking BDD/ATDD to the Next Level assert (200) assert (n == nc) 35
- 37. Chat Server Message Sequence Login a 1234 Hi Chatter a Login b secret Hi Chatter b List Users Users: a, b !!! Client Endpoint Client Endpoint Model Driven Development: Taking BDD/ATDD to the Next Level Chat Server 36
- 38. Chat Server Message Sequence List Users Users: a, b Post “Pumpkin Latte #mypics” Ack a1 “Pumpkin Latte #mypics” Bcast a1 “Pumpkin Latte #mypics” Logoff b Ack Logoff b !!! Client Endpoint Client Endpoint Model Driven Development: Taking BDD/ATDD to the Next Level Chat Server 37
- 39. Spec Explorer Demo • Model-based testing tool – Microsoft Research – Used to test all MSFT APIs – Robust and stable – Visual Studio “power tool” • C# code, not cartoons • Generates standalone executable test suite Model Driven Development: Taking BDD/ATDD to the Next Level 38
- 40. 1. Create Test Model Actions • [Rule] tags a method for exploration • Requires defines a exploration precondition – must be true to explore Model Driven Development: Taking BDD/ATDD to the Next Level 39
- 41. 2. Define Accepting State(s) • Tag condition(s) that terminate an exploration Model Driven Development: Taking BDD/ATDD to the Next Level 40
- 42. 3. Create Cord File • Set options • Define machines – Limits Actions used in an exploration • Similar to regular expression • Composable • Very powerful! Model Driven Development: Taking BDD/ATDD to the Next Level 41
- 43. 4. Explore the Model • Select any explorable machine – Explore – Generate test suite – Run on-the-fly test Model Driven Development: Taking BDD/ATDD to the Next Level 42
- 44. 5. Review Exploration Graph • Shows action/event sequence • Ellipse: model state computed by model program machine • Diamond: observation “event” – get something from SUT (input to the model program) • Right click to inspect any item Model Driven Development: Taking BDD/ATDD to the Next Level 43
- 45. 6. Define machine(s) to generate test suites • Set options to generate test code • Use any machine or composite machine Model Driven Development: Taking BDD/ATDD to the Next Level 44
- 46. 7. Explore the Test Suite machine • Each root to leaf path becomes a test suite Model Driven Development: Taking BDD/ATDD to the Next Level 45
- 47. 8. Generate Test Code • Explores the test machine • Writes out test code using VS testing framework Model Driven Development: Taking BDD/ATDD to the Next Level 46
- 48. 8. Generate Test Code • Not intended to be maintained • Re-generate any time • Add helper code in the model to extend/customize Model Driven Development: Taking BDD/ATDD to the Next Level 47
- 49. 9. Run the Test Suite • Uses built-in Visual Studio test runner • Adapters – Start/stop chat clients and chat server – Other SUT setup/cleanup • Model Driven Development: Taking BDD/ATDD to the Next Level Test code sends messages, checks replies 48
- 50. Model State (test case inputs) 10. Review Test Results • Detailed run log – – – – Event Handler (pass/fail) Requirement Trace VS test runner States Events Requirements • Add any other programmable function Model Driven Development: Taking BDD/ATDD to the Next Level 49
- 51. Spec Explorer Exploration • Interactive model development • Generate test suites • Machines Cord Declarations Model Program Options Machines define composable Action sequences • Options control exploration and data generation • Actions are explored – Rules define exploration conditions Model Data Structures Rules Actions Events Adapters – Events define expected SUT response • Data structures support exploration • Adapters abstract, control, and observe any interface, any platform Model Driven Development: Taking BDD/ATDD to the Next Level 50
- 52. MBT Driven Development Requirements, Stories, Scenarios... Explore Ambiguous, missing, contradictory, incorrect, redundant, incomplete Test Model Run Generate Model error, omission Test Suite Inputs (Test Sequences) Control Missing, incorrect Coverage Requirements Model Evaluate Expected Outputs (Test Oracle) SUT Observe Model Driven Development: Taking BDD/ATDD to the Next Level Bug 51
- 53. More Models, Less Tests MORE MODELS, LESS TESTS
- 54. Model-based Testing in the Real World • Windows protocol compliance (Web services, RPC, DCOM) • Total effort: 250 person years (mostly junior SDETs) • Saved 50 person years with model-based testing Model-based Testing 1.4 Days/Requirement 42% Less Time Per Requirement Traditional Testing 2.4 Days/Requirement Requirements Study Modeling Test Coding Adapter Coding Model Driven Development: Taking BDD/ATDD to the Next Level Test Execution 53
- 55. Scope of Test Asset Maintenance Model-based Testing Behavior Driven Development Model How much MBTgenerated test code must be maintained? Test Code None! Adapters Model Driven Development: Taking BDD/ATDD to the Next Level 54
- 56. More Models, Less Tests • More Better Testing • Icing the BackBlob – Rule-based – Maintain model – Systematic and deep exploration – Regenerate test suites – Generate 1000s of scenarios in seconds – Models grow much more slowly than test code Model Driven Development: Taking BDD/ATDD to the Next Level 55
- 58. Q&A #MoreModelsLessTests rvbinder@sysverif.com Model Driven Development: Taking BDD/ATDD to the Next Level 57
- 59. Learn More - MBT • UCCAT (User Conference on Advanced Automated Testing) Presentations http://ucaat.etsi.org/2013/program_conf.html • 2012 Model-based Testing User Survey http://robertvbinder.com/wp-content/uploads/rvb-pdf/arts/MBT-User-Survey.pdf • Open Source MBT tools http://www.robertvbinder.com/open-source-tools-for-model-based-testing/ • Overview of MBT, list of commercial MBT tools http://www.slideshare.net/robertvbinder/model-basedtestingignite • Spec Explorer http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745 http://msdn.microsoft.com/library/ee620411.aspx • Microsoft Open Protocol Project http://cacm.acm.org/magazines/2011/7/109908-microsofts-protocol-documentation-program/fulltext#! Model Driven Development: Taking BDD/ATDD to the Next Level 58
- 60. Learn More: BDD, etc. • SpecFlow http://www.specflow.org/ • Using SpecFlow with WatiN http://msdn.microsoft.com/en-us/magazine/gg490346.aspx • SpecFlow Quick Start http://www.youtube.com/watch?v=SVwBz-ZTliw • Writing Cucumber Feature Files https://github.com/cucumber/cucumber/wiki • Using Lava Lamps to indicate Build Status https://wiki.jenkins-ci.org/display/JENKINS/Hudson+Build+Status+Lava+Lamps Model Driven Development: Taking BDD/ATDD to the Next Level 59
- 61. Content and Image Sources Grand Canyon Time Lapse. Ezra Kamer http://www.youtube.com/watch?v=NF6O6HuNJ0k Contrast (Order and Chaos). M. C. Escher, 1950. National Gallery of Canada http://www.gallery.ca/en/see/collections/artwork.php?mkey=29270 The Blob, Theatre Scene. Fairview Productions, 1958. http://www.youtube.com/watch?v=GODDLgM1gKo http://www.imdb.com/title/tt0051418/ Order and Chaos II. M. C. Escher, 1955. National Gallery of Canada http://www.gallery.ca/en/see/collections/artwork.php?mkey=41980 Defect densities: • Capers Jones, Software Quality: Analysis and Guidelines for Success • Robert E. Park. Software Size Measurement: A Framework for Counting Source Statements. Technical Report CMU/SEI-92-TR-020. Software Engineering Institute, Carnegie Mellon University Spec Explorer. Microsoft http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9956b-d9bfa4902745 SpecFlow http://www.specflow.org/ The Blob, Final Scene. Fairview Productions, 1958. http://www.youtube.com/watch?v=0GEJgR-bX0w Spec Explorer Chat Model. Microsoft. All other content Copyright © Robert V. Binder, 2014. Model Driven Development: Taking BDD/ATDD to the Next Level 60