Requirements driven Model-based Testing
2 likes330 views
The document discusses the implementation of requirement-driven model-based testing (MBT) for the software bus service of a complex system, focusing on the challenges of traditional testing methods. It highlights the use of a parent-child architecture for testing communication between tasks and the benefits of MBT in identifying off-nominal behaviors and improving requirement traceability. Ultimately, the findings demonstrate MBT's effectiveness in detecting functional errors and revealing missing or unclear requirements.
Requirements driven Model-based Testing
- 1. 1© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Requirement-driven Model-based Testing of the cFS’ Software Bus Service Dharma Ganesan, Mikael Lindvall, Asa Valdimarsdottir, Stefan Hafsteinsson In collaboration with the cFS team: Susanne Strege,Walter Moleski, and Dave McComas, and Alan Cudmore
- 2. 2© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Agenda Context Motivation System UnderTest (SUT) Challenges with traditional testing Model-based testing (MBT) Model-based testing applied on Software Bus Sample Issues detected Conclusion Possible future work
- 3. 3© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Context The SARP project Software Assurance Research Program “An Assurance Approach to Identify,Test and Cover Off-nominal Behavior” Off-nominal behavior Unexpected sequences of events Out-of-range data values Environment issues
- 4. 4© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Motivation Testing is always important Even more important when Safety is critical Failures are expensive
- 5. 5© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering cFE/cFS Context Diagram Inter-task Message Router (SW Bus) Transponders Commands Real-time Telemetry (UDP) Comm Cards File downlink (CFDP) Summit Chip Mass Storage System CFDP File Transfer File Manager Local Storage Data Storage Event Services Executive Services Time Services 1553 Bus Support Software Bus Command Ingest Telemetry Output Table Services EDAC Memory Scrubber Self Test Memory Dwell Instrument Manager Checksum Memory Manager GN&C Applications (4) Mission Apps cFE core App CFS Applications Stored Commanding Software Scheduler Health & Safety Manager House- keeping Limit Checker
- 6. 6© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Challenges with traditional testing Why do we need more testing? Previous testing Unit tests Good coverage Only test execution is automated Test cases are manually constructed Tests are too low-level detailed Not testing multi-tasking nor communication between tasks
- 7. 7© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Solution Model-basedTesting (MBT) Relatively new technology Black box approach A model that describes desired behavior of the SUT Automatic generation of innumerable test cases Triggers off-nominal behaviors
- 8. 8© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Model-based Testing (MBT) The tester develops a model Instead of writing test cases Based on functional requirements and API documentation In our case: A model program Rules describe the SUT´s desired behaviors Becomes the “test oracle”
- 9. 9© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering MBT applied on the cFS‘ Software Bus Software Bus Service Handles communication between tasks/applications Publish-Subscribe architectural style Main features Pipes Create, Delete, Subscribe through, Receive from Messages Send, Receive Subscriptions Subscribe, Unsubscribe
- 10. 10© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Challenges of testing a SB Multi-tasking Communication between tasks/applications Need more than one task/application An application can not decide on the correctness of it‘s own execution Need an architecture for coordination Overview Control
- 11. 11© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Solution – The Parent Child Architecture Each test case acts as a Parent Application Generated by our model The Parent spans multiple Child Tasks The Parent sends commands to it‘s Child Tasks The Parent “knows“ which behaviors should succeed and which should fail Parent Child Task Child Task Child Task
- 12. 12© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Solution – The Parent Child Architecture Parent Child Task Result pipe Command pipe Command
- 13. 13© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Solution – The Parent Child Architecture Parent Child Task Result pipe Command pipeResult
- 14. 14© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Example: Command sequence How to test whether publish works between two tasks? Our test case generates a sequence as below Parent send a sequence of commands: Parent: “Child no. 1 create a pipe“ Child1creates a pipe and sends the return code to the parent Parent: “Child no. 1subscribe to message nr 5“ Child 1 subscribes to message nr 5 and sends the return code Parent:“Child 2 publish message nr 5“ Child 2 publishes message nr 5 and sends the return code Parent:“Child 1receive message nr 5” Child 1 receives message nr 5 and sends the return code
- 15. 15© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Our model Spec Explorer A plug-in toVisual Studio Tester develops a model program Written in C# Simplified and abstract version of the SUT Simple structures and data types Messages modeled as integers Pipes modeled as a set of integers No complex structures, threads, pointers or semaphores
- 16. 16© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Model Program (Sample) Model We capture requirement numbers in our model Helps in detecting missing requirements Will be used for establishing traceability links
- 17. 17© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Our model
- 18. 18© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Generated from our model program Spec Explorer generates state machines from our model In regular MBT models are either manually created and/or derived from test cases (Fraunhofers tool FAST) Create Pipe is telling child 0 to create a pipe with name 0 of depth 1.
- 19. 19© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Generated test sequences - sample Each chain is a test case A Parent Application Parent creates a child (id is 0 in InitTask) Also get test code Adapter converts the test code to SUT syntax From C# to C
- 20. 20© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Generated test sequences - sample Two ChildTasks Two ChildTasks and looser restrictions Each test case (i.e. a parent) creates two child task and send different commands such as Publish, subscribe, unsubscribe, ReadMsg, etc.
- 21. 21© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Zoom-in
- 22. 22© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Requirement tracability Future work: Create visualization
- 23. 23© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Cross running Parent Child Task Child Task Child Task Parent Child Task Child Task Parent Child Task Child Task Allows for testing of more complex concurrency situations
- 24. 24© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Issues detected These issues were unknown to the CFS team Off-nominal behaviors Trying to create an already existing pipe caused data corruption Cross running multiple Parent Applications Method called to delete ChildTasks does not clean up the task data properly Caused memory issue Requirement issues Missing or/and unclear requirements Off-nominal scenarios often not discussed Duplicate requirements
- 25. 25© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering A sample issue detected
- 26. 26© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Another issue (sample) Off-nominal pipe creation Found early Trying to create an already existing pipe caused data corruption Not accounted for in the SUT‘s code Caused failures in later steps Delete Pipe failed
- 27. 27© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Conclusion and future work MBT is promising Detecting off-nominal issues is the core of MBT MBT Helps in detecting Missing requirements (we can improve requirements) Functional errors in the SUT
- 28. 28© 2015 Fraunhofer USA, Inc. Center for Experimental Software Engineering Acknowledgements NASA SARP Martha Wetherholt Kenneth D. Rehm Steven Hard Markland Benson cFS team JonathanWilmot Contact: dganesan@fc-md.umd.edu mlindvall@fc-md.umd.edu