Documented Requirements are not Useless After All!

Documented Requirements !
are not Useless After All! !
!
The Case for Supporting Change, !
Conﬁguration, and Testing
Lionel Briand

XP 2016, May 27th, 2016

Interdisciplinary Centre for ICT Security, Reliability, and Trust (SnT)
University of Luxembourg, Luxembourg

Fundamentals of Agile Methods
2
•  Individuals and Interactions over Process and
Tools
•  Customer Collaboration over Contracts
•  Working Software over Documentation
•  Responding to Change over Planning

Requirements in Agile Methods
3
•  “Customer” on site
•  Whole development team involved in eliciting
requirements
•  Use of a “common” language, e.g., user stories
•  Test cases as “executable requirements” (TDD)
•  Minimize traceability
•  Prioritize requirements
•  Accommodate requirements changes
•  Like all principles, these are meant to be used
thoughtfully and adapted in context

These principles are often used !
in a simplistic and dogmatic
fashion
4

Agile development, in practice,
tend to underplay the role of
documented requirements
5

Objectives of the Talk
6
•  In reality, whether and how you write
requirements is a trade-off
•  When are documented requirements
important?
•  What are their main applications?
•  What form do they take in practice?
•  What are the challenges?
•  Project examples and lessons learned

Natural Language
8
•  Most requirements in practice are expressed
using natural language
•  Ambiguity and incompleteness
•  Trade-off between precision and cost
•  Current support is insufﬁcient, especially for
large sets of requirements

Domain Knowledge
9
•  All requirements depend, more or less
explicitly, on domain knowledge
•  Common concepts and terminology
•  Not always consistent among all stakeholders
•  Software engineers often have a superﬁcial
understanding of the application domain they
target

Change
10
•  Requirements change frequently
•  Changes have side-effects on other
requirements, design decisions, test cases …
•  How do we support such changes in ways
that scale to hundreds of requirements?

Conﬁguring Requirements
11
•  Many software systems are part of product
families targeting varying needs among
multiple customers
•  Requirements typically need to be tailored or
conﬁgured for each customer
•  Because of interdependencies among such
decisions, this is often error-prone and
complex.

Challenges
12
•  Dealing effectively with natural language
•  Capturing domain knowledge
•  Dealing with frequent changes
•  Conﬁguring requirements with customers in
product families
•  These challenges also apply to agile
development

14
Analyzing and Handling Natural
Language Requirements

Challenges
16
•  Large projects in satellite domain (e.g., ESA)
•  Hundreds of natural language requirements
•  Three tiers of requirements
•  Many stakeholders
•  Requirements capture a contract
•  Requirements change

Research Objectives
Checking
compliance
to
boilerplates
Domain
Model
extraction
Change
Impact
analysis in
requirements
17

Research Objectives
Checking
compliance
to
boilerplates
Domain
Model
extraction
Change
Impact
analysis in
requirements
18

Compliance with Boilerplates
19
•  Rupp’s boilerplate:

•  As soon as an unplanned outage is detected
the Surveillance and Tracking Component
shall notify the SOT operator at the local
station.
•  Automation required for compliance checking
•  No glossary, scalable parsing
[<When?>
<under what
conditions?>]
THE SYSTEM
<system name>
SHALL
SHOULD
WILL
PROVIDE <whom?>
WITH THE ABILITY TO
<process>
<process>
BE ABLE TO
<process>
<object>
[<additional details
about the object>]

Approach and Results
20
•  Natural Language Processing (GATE)
•  Text chunking
•  Boilerplates: RUPP and EARS
•  Boilerplates are expressed as BNF grammars
and then pattern matching rules
•  Four industrial case studies
•  Low number of false positives and negatives
•  Hundreds of requirements in a few minutes

Tool: RETA
21
GATE NLP
Workbench
Conformance
Diagnostics
(within GATE)Requirements
Lists of modals,
conditional words,
ambiguous terms, etc.
LSTLSTLSTLST
Rules for checking
template
conformance
JAPEJAPEJAPEJAPE
Rules for checking
best practices
Glossary
(optional)
JAPEJAPEJAPEJAPE
Requirements Analyst
Requirements Authoring &
Management
http://sites.google.com/site/retanlp/

Research Objectives
Checking
compliance
to
boilerplates
Domain
Model
extraction
Change
Impact
analysis in
requirements
22

Change Impact Analysis
23
•  A change in requirements may lead to
changes in other requirements
•  Hundreds of requirements
•  No traceability
•  We propose an approach based on: (1) Natural
Language Processing, (2) Phrase syntactic
and semantic similarity measures
•  Results: We can accurately pinpoint which
requirements should be inspected for
potential changes

Example
• R1: The mission operation controller shall transmit satellite
status reports to the user help desk.
• R2: The satellite management system shall provide users with
the ability to transfer maintenance and service plans to the
user help desk.
• R3: The mission operation controller shall transmit any
detected anomalies with the user help desk.
24

Change
status reports to the user help desk document repository.
user help desk.
25

Challenge #1
Capture Changes Precisely
user help desk.
26

Challenge #2 !
Capture Change Rationale
user help desk.
27

•  R1: The mission operation controller shall transmit satellite status reports to the user help desk
document repository.
•  R2: The satellite management system shall provide users with the ability to transfer
maintenance and service plans to the user help desk.
•  R3: The mission operation controller shall transmit any detected anomalies with the user help
desk.
Challenge #2 !
Change Rationale
Rationale:

R1: We want to globally rename “user help desk”
R2: Avoid communication between “mission
operation controller” and “user help desk”
R3: We no longer want to “transmit satellite
status reports” to “user help desk” but instead to
“user document repository”

28

Solution Characteristics
• Account for the phrasal structure of requirements
The mission operation controller shall transmit satellite
user help desk, Deleted
user document repository, Added
• Consider semantically-related phrases that are not exact
matches and close syntactic variations
29

Tool: Narcia
30
https://sites.google.com/site/svvnarcia/

Research Objectives
Checking
compliance
to
boilerplates
Domain
Model
extraction
Change
Impact
analysis in
requirements
31

Domain Models
A domain model is a representation of conceptual entities or
real-world objects in a domain of interest.
32

Motivation
•  In practice, domain models are not preceding the elicitation and writing of
requirements
•  Representation of important domain concepts and their relations
•  Facilitate communication between stakeholders from different
backgrounds
•  Help identify inconsistencies in terminology, etc.
•  Support is required for building domain models
•  A lot of information required to build a domain model is automatically
extractable
33

Approach
mulator”, via a ref to dependency.
ACH
NPs, VBs Dependencies
Process
Requirements
Statements
NL
Requirements
Lift
Dependencies to
Semantic Units
nsubjnn dobj
Domain
Model
Construct
Domain
Model
0..1 1..*
relation
Extraction
Rules
1
23
Figure 7: Approach Overview.
ents
our
ex-
ach.
the
NL
doc-
out-
lass
ow,
the 34

Requirements to Design IA
Requirements
Design
Objective: Identifying
impact of requirements
changes on SysML
design
35

Motivating Scenario!

Example Change Requests:
•  Change to R11: Change over temperature detection level to 147 C from 110 C
•  Change to R12: Temperature range should be extended to -40/150 C from -20/120 C
text = "The CP controller shall
provide temperature diagnostics."
id = "R1"
«requirement»
Temperature Diagnostics
text = "The CP controller shall
detect temperatures exceeding
110 ºC."
id = "R11"
«requirement»
Over-Temperature Detection
text = "The CP controller shall be
able to measure temperatures
between -20 ºC and 120 ºC."
id= "R12"
«requirement»
Operational Temperature Range
36

:Digital to Analog
Converter
:DC Motor
Controller
«satisfy»
:Over-Temperature
Monitor
:Diagnostics
Manager
Over-
Temperature
Motor
drive
mode
Error
: Diagnostics
and Status
Signal Generation
…
Motor
position
B1
B2
B3
B4
«requirement»
Over-Temperature
Detection
(R11)
…
…
«requirement»
Operational
Temperature Range
(R12)
…
:Temperature
Processor
«satisfy»
B5
B6
Temperature
Voltage
(digitized)
Temperature
Motor
speed
…
Motor
torque
…
Block Diagram
37

Actually Impacted Elements
• Both changes are related to ``Temperature Diagnostics’’, but
they impact two drastically different parts of the system
• Change to R11 impacts a decision statement in software
• Change to R12 impacts electronic voltage divider circuit
(hardware) and a lookup table (software)
38

:Digital to Analog
Converter
:DC Motor
Controller
«satisfy»
:Over-Temperature
Monitor
:Diagnostics
Manager
Over-
Temperature
Motor
drive
mode
Error
: Diagnostics
and Status
Signal Generation
…
Motor
position
B1
B2
B3
B4
«requirement»
Over-Temperature
Detection
(R11)
…
…
«requirement»
Operational
Temperature Range
(R12)
…
:Temperature
Processor
«satisfy»
B5
B6
Temperature
Voltage
(digitized)
Temperature
Motor
speed
…
Motor
torque
…
Impacted by R12
Impacted by R12
Impacted by R12
Impacted by R11
Impacted Blocks …!

39

Approach
• Algorithm to compute estimated impacted elements in SysML designs
for a requirements change
• Combination of structural analysis, behavioral analysis, and natural
language processing (model labels)
• Number of design elements to inspect (average): 4.8% of design
elements that can possibly be impacted
• Only one missed impacted element across all changes
• Frequent changes: Signiﬁcant savings and better quality assurance

40

41
Supporting Product Lines and
Requirements Conﬁguration

Context

International Electronics !
& Engineering (IEE)
IEE develops real-time embedded systems:
•  Automotive safety sensing systems
•  Automotive comfort & convenience systems,
e.g., Smart Trunk Opener

42

Smart Trunk Opener (STO)

STO Provides automatic and hands-free access to a vehicle’s
trunk (based on a keyless entry system)

43

IEE Requirements Engineering
Use Case Driven
Development
Use Case !
Diagram
Use Case!
Speciﬁcations
Domain !
Model
44

Dealing with Multiple Customers
45
STO Requirements
from Customer A
(Use Case Diagram
and Speciﬁcations,
and Domain Model)
Customer A
for STO
modify modify
modify modify
STO Test Cases for
Customer A
evolves to
(clone-and-own)
STO Requirements
from Customer B
(Use Case Diagram
and Domain Model)
Customer B
for STO
evolves to
(clone-and-own)
STO Test Cases for
Customer B
evolves to
(clone-and-own)
STO Requirements
from Customer C
(Use Case Diagram
and Domain Model)
Customer C
for STO
evolves to
(clone-and-own)
STO Test Cases for
Customer C

Product Line Approach
46
•  A Product Line approach was clearly needed
•  Restricted and analyzable use case specifications
(NLP)
•  Variability modeling in use case diagrams and
specifications
•  Automated configuration guidance for configuring
requirements with each customer
•  Automated generation of product-specific use case
models based on decisions

Use Cases And
Domain Model
Customer A
for Product X
Product-Line
Use Cases And
Domain Model
Identify
Commonalities and
Variabilities
Conﬁgurator
Customer B
for Product X
Use Cases And
Domain Model
Customer C
for Product X
Use Cases And
Domain Model
configure
evolves
reconfigure
evolves
reconfigure
reconfigure
reconfigure
47

Product Line Use Case Diagram for
STO (Partial)
48

• RUCM is based on a (1) template, (2) restriction rules,
and (3) speciﬁc keywords constraining the use of
natural language in use case speciﬁcations
• RUCM reduces ambiguity and facilitates automated
analysis of use cases

Restricted Use Case Modeling: !
RUCM
49

• Flow of events is described in restricted natural language
RUCM
Basic Flow
1. INCLUDE USE CASE Identify System Operating Status.
2. The system VALIDATES THAT the operating status is OK.
3. The system REQUESTS the move capacitance FROM the UpperSensor.
4. The system REQUESTS the move capacitance FROM the LowerSensor.
5. The system VALIDATES THAT the movement is a valid kick.
6. The system VALIDATES THAT the overuse protection feature is enabled.
7. The system VALIDATES THAT the Overuse protection status is inactive.
8. The system SENDS the valid kick status TO the STOController.
Post condition: The gesture has been recognised and the STO Controller has
been informed.
50

• Keyword: INCLUDE VARIATION POINT: ...
• Inclusion of variation points in basic or alternative ﬂows of
use cases:

Use Case: Identify System Operating Status
Basic Flow
1. The system VALIDATES THAT the watchdog reset is valid.
2. The system VALIDATES THAT the RAM is valid.
3. The system VALIDATES THAT the sensors are valid.
4. The system VALIDATES THAT there is no error detected.
Speciﬁc Alternative Flow
RFS 4
1. INCLUDE VARIATION POINT: Storing Error Status.
2. ABORT.
!
Example Variability Extension
51

•  Tool Support (PUMConf): https://sites.google.com/site/pumconf/
•  Positive feedback from engineers, both about the modeling
approach and configuration tool
•  They confirmed they benefited from:
•  Understanding the commonalities and differences across
product requirements
•  Automated guidance in a configuration that is often complex,
i.e., many (interdependent) decisions
Results
52

Regulations - Requirements
•  Many (government) systems need to be
compliant with the law and remain so
over time
•  E.g., Tax systems
•  How can we help with making sure IT
engineers implement systems that
comply with the law?
•  How can we verify they do?
54

Solution Overview
Test cases
Actual
software
system
Traces to
Traces to
Analyzable
interpretation
of the law
Generates
Results match?
Impact of legal
changes
Simulates
55

Domain Model
56
Art. 105bis […]The commuting
expenses deduction (FD) is
defined as a function over the
distance between the principal
town of the municipality on
whose territory the taxpayer's
home is located and the place of
taxpayer’s work. The distance is
measured in units of distance
expressing the kilometric
distance between [principal]
towns. A ministerial regulation
provides these distances.
Interpretation + Traces

Procedures as Activity Diagrams
57
The amount of the deduction is
calculated as follows:
If the distance exceeds 4 units but is
less than 30 units, the deduction is €
99 per unit of distance.
The first 4 units does not trigger any
deduction and the deduction for a
distance exceeding 30 units is limited
to € 2,574.
Interpretation + Traces

Discussion
•  Models understandable by both legal experts and IT
specialists
•  We addressed the gap between legal experts and IT
specialists
•  Modeling effort was considered reasonable given the life span
of such eGovernment systems
•  Traceability to the law was considered a signiﬁcant asset
given frequent and complex changes in the law
58

59
Requirements-Driven Testing

Context
•  Context: Automotive, sensor systems
•  Traceability between system requirements and test cases
•  Mandatory when software must comply with ISO 26262
•  Customers also require such compliance
•  Use-case-centric development
TC4
TC3
TC2
Requirements!
!
Test cases
TC1
60

Objectives
• Automatically generate test cases from requirements
• Capture and create traceability information between test
cases and requirements
• Requirements are captured through use cases
• Use cases are used to communicate with customers and the
system test team
• Complete and precise behavioral models are not an option:
too expensive (Model-based testing)
61

Strategy
• Analyzable use case speciﬁcations
• Automatically extract test model from the use case
speciﬁcations
• Minimize modeling, domain modeling only
• No behavioral modeling
62

THE ACTOR SEND
THE SYSTEM VALI
THE SYSTEM DIS
THE ACTOR SEND

THE ACTOR SEND
THE SYSTEM VALI
THE SYSTEM DIS
THE ACTOR SEND

THE ACTOR SEND
THE SYSTEM VALI
THE SYSTEM DIS
THE ACTOR SEND

ERRORS ARE ABSENT
TEMPERATURE IS LOW
STATUS IS VALID
Identify Constraints
4
Constraint descriptions
Errors.size() == 0
Status != null
t > 0 && t < 50
Generate
Scenarios and
Inputs
6
Elicit Use Cases
1
Missing Entities
Specify Constraints
5
OCL constraints
Model the Domain
2
Evaluate
Consistency
3
Domain Model
RUCM
Use Cases
Generate
Test Cases
7
Test Cases
Object
Diagrams
Test
Scenarios Mapping Table

64
https://sites.google.com/site/umtgTestGen/
Toolset integrated with IBM
DOORS and Rhapsody

Applications
66
•  Requirements to support a shared
understanding among many stakeholders in
large projects
•  Requirements as contract with customers
•  Requirements to support communication
between software engineers and domain
experts
•  Requirements to support compliance with
standards, e.g., traceability to tests
•  Requirements to support change

Forms of Requirements
67
•  Natural language statements, complying or
not with boilerplates
•  Use case speciﬁcations, possibly structured
and restricted
•  Models, e.g., class and activity diagrams
•  Agile practice: User stories are a basis for
communicating requirements but are not
analyzable or complete – This is not always
acceptable

The form of requirements depends on how
one intends to apply them and many
contextual factors
68

Contextual Factors
69
•  Regulatory compliance, e.g., standards
•  Project size, team distribution, and number of
stakeholders
•  Background of stakeholders and communication
challenges
•  Domain complexity
•  Presence of product lines with multiple customers
•  Importance of early agreement on and full compliance
with requirements (e.g., Contract, safety certiﬁcation)
•  Frequency and consequences of changes in
requirements

Conclusions
70
•  Documented requirements are useful after all!
•  The extent to which requirements and their traceability are
documented depends on multiple factors
•  They also come in different forms, mostly depending on
analysis requirements
•  Many challenges to be addressed by research:

(1) Ambiguity in Natural Language requirements

(2) Automation of time consuming tasks: Scalability

(3) Change impact
•  NLP technology is now mature and provides many
opportunities for automation and lowering the documentation
overhead

The challenge is to ﬁnd the right trade-off!
in a given context
71

Acknowledgements
72
•  Mehrdad Sabetzadeh
•  Arda Goknil
•  Shiva Nejati
•  Chetan Aurora
•  Ines Hajri
•  Ghanem Soltana
•  Chunhui Wang

Natural Language Requirements
•  [RE 2015] C. Arora et al., Change Impact Analysis for Natural Language Requirements: An NLP
Approach
•  [TSE 2015] C. Arora et al., Automated Checking of Conformance to Requirements Templates using
Natural Language Processing
•  [ESEM 2014] C. Arora et al., Improving Requirements Glossary Construction via Clustering
Requirements-Driven Testing
•  [ISSTA 2015] C. Wang et al., Automatic Generation of System Test Cases from Use Case
Speciﬁcations
74

SysML Traceability and Safety Analysis
•  [TOSEM 2014] L. Briand et al., Traceability and SysML Design Slices to Support Safety
Inspections: A Controlled Experiment
•  [IST 2012] S. Nejati et al., A SysML-Based Approach to Traceability Management and Design
Slicing in Support of Safety Certiﬁcation: Framework, Tool Support, and Case Studies
Legal Modeling and Analysis
•  [MODELS 2014] G. Soltana et al., UML for Modeling Procedural Legal Rule
•  [SoSYM 2016] G. Soltana et al., Model-Based Simulation of Legal Policies: Framework, Tool
Support, and Validation
75

Product Families and Configuration
•  [MODELS 2015] I. Hajri et al., Applying Product Line Use Case Modeling in an Industrial
Automotive Embedded System: Lessons Learned and a Refined Approach
•  [SoSYM 2016] I. Hajri et al., A Requirements Configuration Approach and Tool for Use Case-Driven
Development
Impact Analysis
•  [FSE 2016] S. Nejati et al., Automated Change Impact Analysis between SysML Models of
Requirements and Design
•  [RE 2015] C. Arora et al., Change Impact Analysis for Natural Language Requirements: An NLP
Approach
76

Documented Requirements are not Useless After All!

More Related Content

What's hot (20)

Similar to Documented Requirements are not Useless After All! (20)

More from Lionel Briand (20)

Recently uploaded (20)

Documented Requirements are not Useless After All!