Automatic Generation of System Test Cases from Use Case Specifications
6 likes1,343 views
This document describes research on automatically generating system test cases from use case specifications. The researchers developed a technique called Use Case Modeling for Test Generation (UMTG) that involves: 1. Eliciting use cases and modeling the problem domain. 2. Analyzing use case specifications and domain models for consistency. 3. Generating test scenarios, inputs, and path conditions from use cases and constraints. 4. Mapping the abstract test cases to executable test code to facilitate automated testing. The technique was evaluated on a case study of an embedded system for detecting vehicle occupancy, demonstrating its ability to generate test cases with less additional modeling effort compared to current manual practices.
![RUCM Use Case Example
8
Use Case Name: Identify Occupancy Status
Actors: AirbagControlUnit
Precondition: The system has been initialized
. . .
"Basic Flow
1. The seat SENDS occupancy status TO the system.
2. INCLUDE USE CASE Classify occupancy status.
3. The system VALIDATES THAT the occupant class for airbag control is valid.
4. The system SENDS the occupant class for airbag control TO AirbagControlUnit.
Specific Alternative Flow
RFS 3
1. IF the occupant class for airbag control is not valid THEN
Postcondition: The occupant class for airbag control has been sent.
Postcondition: The previous occupant class for airbag control has been sent.
[Yue et al. TOSEM’13]](https://image.slidesharecdn.com/chunhui-issta-slides-final-150717220511-lva1-app6892/85/Automatic-Generation-of-System-Test-Cases-from-Use-Case-Specifications-8-320.jpg)
Automatic Generation of System Test Cases from Use Case Specifications
- 1. .lusoftware verification & validation VVS Automatic Generation of System Test Cases from Use Case Specifications Chunhui Wang, Fabrizio Pastore, Arda Goknil, Lionel Briand, Zohaib Iqbal University of Luxembourg,! Luxembourg! {chunhui.wang,fabrizio.pastore,arda.goknil,lionel.briand, muhammad.zohaib}@uni.lu ISSTA’15, Baltimore, July 17th, 2015
- 2. Problem • Verifying the compliance of a system with its requirements in a cost-effective way • Traceability between requirements and system test cases – This cannot be manual • E.g., automotive industry which must comply to ISO 26262 2
- 3. Objectives • Automatically generate test cases from requirements • Capture and create traceability information between test cases and requirements 3
- 4. Context Matters • Requirements are captured through use cases • Use cases are used to communicate with customers and the system test team • Complete and precise behavioral models are not an option: too expensive 4
- 5. Strategy • Analyzable use case specifications • Automatically extract test model from the use case specifications • Minimize modeling • No behavioral modeling 5
- 6. Use Case Modeling for System Tests Generation (UMTG) 6
- 7. Elicit Use Cases 1 THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND RUCM Use Cases 7
- 8. RUCM Use Case Example 8 Use Case Name: Identify Occupancy Status Actors: AirbagControlUnit Precondition: The system has been initialized . . . "Basic Flow 1. The seat SENDS occupancy status TO the system. 2. INCLUDE USE CASE Classify occupancy status. 3. The system VALIDATES THAT the occupant class for airbag control is valid. 4. The system SENDS the occupant class for airbag control TO AirbagControlUnit. Specific Alternative Flow RFS 3 1. IF the occupant class for airbag control is not valid THEN Postcondition: The occupant class for airbag control has been sent. Postcondition: The previous occupant class for airbag control has been sent. [Yue et al. TOSEM’13]
- 9. Elicit Use Cases 1 THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND RUCM Use Cases Model the Domain 2 Domain Model 9 1 System AirbagControl Sensor OccupantStatus - OccupantClassForAirbagControl - OccupantClassForSeatBeltReminder 1 1 1 1 1 1 1..*
- 10. Elicit Use Cases 1 Model the Domain 2 THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND RUCM Use Cases Generate Scenarios and Inputs 6 ERRORS ARE ABSENT TEMPERATURE IS LOW STATUS IS VALID Identify Constraints 4 Constraint descriptions Evaluate Consistency 3 Domain Model Based on Natural Language Processing 10
- 11. 11 Basic Flow 1. The seat SENDS occupancy status TO the system. 2. INCLUDE USE CASE Classify occupancy status. 3. The system VALIDATES THAT the occupant class for airbag control is valid and the occupant class for seat belt reminder is valid. 4. The system SENDS the occupant class for airbag control TO AirbagControlUnit. 5. The system SENDS the occupant class for seat belt reminder TO SeatBeltControlUnit. 6. The System Waits for next execution cycle. Postcondition: The occupant class for airbag control and the occupant class for seat belt reminder have been sent. INPUT STEP INCLUDE STEP CONDITIONAL STEP OUTPUT STEP OUTPUT STEP INTERNAL STEP POSTCONDITION DOMAIN ENTITY CONSTRAINT CONSTRAINT DOMAIN ENTITY DOMAIN ENTITY
- 12. ERRORS ARE ABSENT TEMPERATURE IS LOW STATUS IS VALID Identify Constraints 4 Generate Scenarios and Inputs 6 Elicit Use Cases 1 Constraint descriptions THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND Model the Domain 2 12 RUCM Use Cases Evaluate Consistency 3 Domain Model Missing Entities
- 13. Evaluate Model Consistency Occupant Class for Airbag Control Occupant Class for Seat Belt Reminder Domain Entities Airbag Control Classification Filter Sensor Tagged Use Case
- 14. Evaluate Model Consistency 14 Occupant Class for Airbag Control Occupant Class for Seat Belt Reminder Domain Entities AirbagControl System Sensor OccupantStatus - OccupantClassForAirbagControl - OccupantClassForSeatBeltReminder Airbag Control Classification Filter Sensor ClassificationFilter Tagged Use Case 1 1 1 1 1 1 1 1..* 1 1
- 15. Generate Scenarios and Inputs 6 Elicit Use Cases 1 Missing Entities THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND Model the Domain 2 Evaluate Consistency 3 Domain Model 15 ERRORS ARE ABSENT TEMPERATURE IS LOW STATUS IS VALID Identify Constraints 4 Constraint descriptions RUCM Use Cases Errors.size() == 0 Status != null t > 0 && t < 50 Specify Constraints 5 OCL constraints
- 16. Identify Constraints 16 The occupant class for airbag control is valid Voltage error is detected List of Constraints The occupant class for seat belt reminder is valid Tagged Use Case
- 17. Voltage error is detected List of Constraint Descriptions The occupant class for seat belt reminder is valid Tagged Use Case Engineers Specify Constraints System.allInstances() ->forAll(b | b.occupancyStatus.classForAirbag <> Status::Error) Software Engineers specify corresponding OCL constraints The occupant class for airbag control is valid 17
- 18. Errors.size() == 0 Status != null t > 0 && t < 50 Elicit Use Cases 1 Missing Entities Specify Constraints 5 THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND Model the Domain 2 Evaluate Consistency 3 Domain Model ERRORS ARE ABSENT TEMPERATURE IS LOW STATUS IS VALID Identify Constraints 4 Constraint descriptions 18 RUCM Use Cases Generate Scenarios and Inputs 6 Object Diagrams Test Scenarios OCL constraints
- 19. 19 Basic Flow 1. The seat SENDS occupancy status TO the system. 2. INCLUDE USE CASE Classify occupancy status. 3. The system VALIDATES THAT the occupant class for airbag control is valid and the occupant class for seat belt reminder is valid. 4. The system SENDS the occupant class for airbag control TO AirbagControlUnit. 5. The system SENDS the occupant class for seat belt reminder TO SeatBeltControlUnit. 6. The System Waits for next execution cycle. Postcondition: The occupant class for airbag control and the occupant class for seat belt reminder have been sent. INPUT STEP INCLUDE STEP CONDITIONAL STEP OUTPUT STEP OUTPUT STEP INTERNAL STEP POSTCONDITION DOMAIN ENTITY CONSTRAINT CONSTRAINT DOMAIN ENTITY DOMAIN ENTITY
- 20. INCLUDE STEP CONDITIONAL STEP OUTPUT STEP OUTPUT STEP INTERNAL STEP POSTCONDITION 20 Basic Flow 1. The seat SENDS occupancy status TO the system. 2. INCLUDE USE CASE Classify occupancy status. 3. The system VALIDATES THAT the occupant class for airbag control is valid and the occupant class for seat belt reminder is valid. 4. The system SENDS the occupant class for airbag control TO AirbagControlUnit. 5. The system SENDS the occupant class for seat belt reminder TO SeatBeltControlUnit. 6. The System Waits for next execution cycle. Postcondition: The occupant class for airbag control and the occupant class for seat belt reminder have been sent. UseCaseStart INPUT STEP DOMAIN ENTITY CONSTRAINT CONSTRAINT DOMAIN ENTITY DOMAIN ENTITY Precondition
- 21. CONDITIONAL STEP OUTPUT STEP OUTPUT STEP INTERNAL STEP POSTCONDITION 21 Basic Flow 1. The seat SENDS occupancy status TO the system. 2. INCLUDE USE CASE Classify occupancy status. 3. The system VALIDATES THAT the occupant class for airbag control is valid and the occupant class for seat belt reminder is valid. 4. The system SENDS the occupant class for airbag control TO AirbagControlUnit. 5. The system SENDS the occupant class for seat belt reminder TO SeatBeltControlUnit. 6. The System Waits for next execution cycle. Postcondition: The occupant class for airbag control and the occupant class for seat belt reminder have been sent. UseCaseStart Input INCLUDE STEP DOMAIN ENTITY CONSTRAINT CONSTRAINT DOMAIN ENTITY DOMAIN ENTITY Precondition OccupancyStatus DomainEntity DOMAIN ENTITY
- 22. 22 UseCaseStart Input Include Condition Output Exit Condition Exit Output Exit OccupancyStatus DomainEntity System.initialized = true OCL Constraint AirbagControlUnit.AllI… OCL Constraint … OCL Constraint … DomainEntity OCL Constraint OccupancyStatus.ClassForSeat <> Error AND OccupancyStatus.ClassForAirbag <> Error
- 23. 23 UseCaseStart Input Include Condition Output Exit Condition Exit Output Exit System.initialized = true AND OccupancyStatus.ClassForSeat <> Error AND OccupancyStatus.ClassForAirbag <> Error Path condition: OCL Constraint OccupancyStatus.ClassForSeat <> Error AND OccupancyStatus.ClassForAirbag <> Error OccupancyStatus DomainEntity System.initialized = true OCL Constraint AirbagControlUnit.AllI… OCL Constraint OCL Path Condition: System.allInstances() -> forAll(b | b.initialized = true ) AND OccupancyStatus.allInstances() -> forAll(o| o.ClassForSeat <> Status:: Error) AND OccupancyStatus.allInstances() -> forAll(o| o.ClassForAirbag <> Status::Error)
- 24. 24 System.initialized = true AND OccupancyStatus.ClassForSeat <> Error AND OccupancyStatus.ClassForAirbag <> Error Path condition: OCL Solver :System Initialized = true Version = 3 :OccupancyStatus ClassForSeat = Adult ClassForAirbag = Adult Input Values OccupancyStatus.ClassForAirbag = Adult OccupancyStatus.ClassForSeat = Adult System.initialized = true Object Diagram 1 System AirbagControl Sensor OccupantStatus - ClassForAirbag - ClassForSeat 1 1 1 1 1 1 1..* Domain Model:
- 25. THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND ERRORS ARE ABSENT TEMPERATURE IS LOW STATUS IS VALID Identify Constraints 4 Constraint descriptions Errors.size() == 0 Status != null t > 0 && t < 50 Generate Scenarios and Inputs 6 Elicit Use Cases 1 Missing Entities Specify Constraints 5 OCL constraints Model the Domain 2 Evaluate Consistency 3 Domain Model RUCM Use Cases Generate Test Cases 7 Test Cases Object Diagrams Test Scenarios 25
- 26. 26 UseCaseStart Input Include Condition Output Exit Test Case occupancyStatus.ClassForAirbag = Adult OccupancyStatus DomainEntity Scenario Under Test System.initialized = true <INPUT> <CHECK> <INPUT> AirbagControlUnit.AllInstances() -> forAll(a|a.status = a.bodySense.occupancyStatus) System.initialized = true OCL Constraint Input Values occupancyStatus.ClassForAirbag = Adult occupancyStatus.ClassForSeat = Adult System.initialized = true occupancyStatus.ClassForSeat = Adult AirbagControlUnit.AllI… OCL Constraint Abstract Test Case
- 27. THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND ERRORS ARE ABSENT TEMPERATURE IS LOW STATUS IS VALID Identify Constraints 4 Constraint descriptions Errors.size() == 0 Status != null t > 0 && t < 50 Generate Scenarios and Inputs 6 Elicit Use Cases 1 Missing Entities Specify Constraints 5 OCL constraints Model the Domain 2 Evaluate Consistency 3 Domain Model RUCM Use Cases Generate Test Cases 7 Test Cases Object Diagrams Test Scenarios 27 Mapping Table
- 28. Generation of Executable Test Cases 28 ResetPower (Time = INIT_TIME) SetBus (Chanel = RELAY_CLASS, Status = CLASS1) ReadAndCheckBus ( D0 = OCCUPIED, D1 = OCCUPIED) Abstract test steps Resulting Test Driver Functions Matching Patterns Function Name Parameters <INPUT> Classfor* = Adult SetBus <INPUT> System.initialized = true ResetPower <CHECK> AirBagControlUnit.allInstant… ReadAndCheckBus Chanel = RELAY_CLASS Status = CLASS1 Time = INIT_TIME Mapping Table: occupancyStatus.ClassForAirbag = Adult System.initialized = true <INPUT> <CHECK> <INPUT> AirbagControlUnit.AllInstances() -> forAll(a|a.status = a.bodySense.occupancyStatus) occupancyStatus.ClassForSeat = Adult Abstract Test Case: D0=OCCUPIED … Executable Test Case:
- 29. THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND ERRORS ARE ABSENT TEMPERATURE IS LOW STATUS IS VALID Identify Constraints 4 Constraint descriptions Errors.size() == 0 Status != null t > 0 && t < 50 Generate Scenarios and Inputs 6 Elicit Use Cases 1 Missing Entities Specify Constraints 5 OCL constraints Model the Domain 2 Evaluate Consistency 3 Domain Model RUCM Use Cases Generate Test Cases 7 Test Cases Object Diagrams Test Scenarios 29 Mapping Table
- 30. Case Study • BodySense, embedded system for detecting occupancy status in a car • Evaluation: • Cost of additional modelling • Effectiveness in terms of covered scenarios compared to current practice at IEE 30
- 31. Complexity of the Case Study 31 Use Case Steps Use Case Flows UC1 50 8 UC2 44 13 UC3 35 8 UC4 59 11 UC5 30 8 UC6 25 6
- 32. Costs of Additional Modeling 32 Use Case Steps Use Case Flows OCL Constraints UC1 50 8 9 UC2 44 13 7 UC3 35 8 8 UC4 59 11 12 UC5 30 8 5 UC6 25 6 12 5 to 10 minutes to write each constraints
- 33. Effectiveness: scenarios covered 33 0 5 10 15 20 25 30 35 40 UC1 UC2 UC3 UC4 UC5 UC6 Scenarios Covered By Engineer Scenarios Covered By UMTG 100% 100% 100% 100% 100% 100% 81% 77% 100% 86% 50% 67% It is hard for engineers to capture all the possible scenarios involving error conditions.
- 34. Problem • Requirements-driven test automation • Traceability between requirements and system test cases – This cannot be manual • E.g., automotive industry which must comply to ISO 26262 34 ERRORS ARE ABSENT TEMPERATURE IS LOW STATUS IS VALID Errors.size() == 0 Status = null > 0 && t < 50 Identify Constraints 4 Generate Scenarios and Inputs 6 Generate Test Cases 7 Evaluate Consistency 3 Elicit Use Cases 1 Model the Domain 2 Specify Constraints 5 Domain Model Missing Entities Constraint descriptions OCL constraints THE ACTOR SEND HE SYSTEM VALI HE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND HE SYSTEM VALI HE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND HE SYSTEM VALI HE SYSTEM DIS THE ACTOR SEND RUCM Use Cases Test Cases Object Diagrams Test Scenarios0 5 10 15 20 25 30 35 40 UC1 UC2 UC3 UC4 UC5 UC6 Scenarios Covered By Engineer Scenarios Covered By UMTG 100% 100% 100% 100% 100% 100% 81% 77% 100% 86% 50% 67% Acceptable modelling effort Full scenario coverage
- 35. 35 NLP Pipeline Tokenizer UMTG Gazetteer Part of Speech Tagger UMTG Transducers The seat SENDS occupancy status TO the system.
- 36. 36 NLP Pipeline Tokenizer UMTG Gazetteer Part of Speech Tagger UMTG Transducers The seat SENDS occupancy status TO the system. ACTOR SEND NOUN PHRASE SYSTEMTO
- 37. 37 NLP Pipeline Tokenizer UMTG Gazetteer Part of Speech Tagger UMTG Transducers The seat SENDS occupancy status TO the system. ACTOR SEND NOUN PHRASE SYSTEMTO ACTOR SEND TONOUN PHRSE SYSTEM INPUT STEP PATTERN