Requirements Based Testing

Requirements Based
Testing
PhD. Maryna Didkovska

• Disney's Lion King, 1994-1995
• Intel Pentium Floating-Point Division Bug, 1994
• 1999 Patriot Missile Defense System, 1991
• The Y2K (Year 2000) Bug, circa 1974 The Explosion of
the Ariane 5, 1996
• The Explosion of the Ariane 5, 1996

Infamous Software Error
Case Studies

• The Disney company released its first multimedia CD-
ROM game for children, The Lion King Animated
Storybook
• Sales were huge
• Disney failed to properly test the software on the many
different PC models available on the market. The
software worked on a few systems, but not on the most
common systems that the general public had

Disney's Lion King, 1994-
1995

• (4195835 / 3145727) * 3145727 – 4195835 = ???
• the way Intel handled the situation:
 The problem was found before the chip was released.
Intel's management decided that it wasn't severe
enough to warrant fixing it, or even publicizing it.
 Once the bug was found, Intel attempted to diminish its
perceived severity through press releases and public
statements.
 When pressured, Intel offered to replace the faulty
chips, but only if a user could prove that he was
affected by the bug.

Intel Pentium Floating-
Point Division Bug, 1994

• On December 3, 1999, NASA's Mars Polar
Lander disappeared during its landing attempt on
the Martian surface.
• Reason for the malfunction was the unexpected
setting of a single data bit.
• The lander was tested by multiple teams. One
team tested the leg fold-down procedure and
another the landing process from that point on.
Both pieces worked perfectly individually, but
not when put together.

NASA Mars Polar Lander,
1999

• fail to defend against several missiles, including
one that killed 28 U.S. soldiers in Dhahran, Saudi
Arabia
• Timing error in the system's clock accumulated to
the point that after 14 hours, the tracking system
was no longer accurate
• In the Dhahran attack, the system had been
operating for more than 100 hours

Patriot Missile Defense
System, 1991

• little memory for storage
• method: to shorten dates from 4-digit format,
such as 1973, to a 2-digit format, such as 73.
• Several hundred billion dollars were spent, to
replace or update computer programs to fix
potential Year 2000 failures

The Y2K (Year 2000) Bug,
circa 1974

• On June 4, 1996 Ariane 5 rocket launched exploded
forty seconds after its lift-off from Kourou, French
Guiana.
• Development costed $7 billion. The destroyed rocket
and its cargo were valued at $500 million.
• Failure: a 64 bit floating point number relating to the
horizontal velocity of the rocket was converted to a 16
bit signed integer. The number was larger than 32,767,
the largest integer storable in a 16 bit signed integer, and
thus the conversion failed.

The Explosion of the
Ariane 5, 1996

• - to demonstrate quality or proper behavior;
• - to detect and fix problems.

Primary goal of testing is to increase the
probability that the application-under-test will
behave correctly under all circumstances and will
meet defined requirements.

Testing: primary purposes

1. Given valid input, the program produces the correct
output.
2. Given invalid input, the program correctly and
gracefully rejects the input.
3. The program doesn't hang or crash, given either valid
or invalid input.
4. The program keeps running correctly for as long as
expected.
5. The program behaves as specified.

The program works
correctly if:

Software Testing – is a process of software analysis and defect
detecting. Actions of defect detecting are directed to defining as many
as possible defects, which should be fixed.
IEEE Standard Glossary of Software Engineering Terminology says
that testing is «The process of exercising software to verify that it
satisfies specified requirements and to detect errors».

Why is testing a process?
Because testing is a planned and well-ordered activity. This moment is
important because under time shorted conditions well planned and
systematic approach leads to defects detecting more rapidly than poorly
planned testing activity.

Software Testing Definition

Defect (bug) – non-conformance to requirements or functional
specification. It is something that does not correspond to valid Customer’s
expectations that are assumed but may be not described in product
requirements. A defect may have a place in requirements or any other
documentation, in design or architecture. A bug can be result of incorrect
environment, configuration or data.
Expected Result – the behaviour of the system (software product) we
expect to see in result of our actions (inputs). In other words – this is
correct behaviour of the software product.

If the actual behaviour of the program we see does not correspond to that
one which we expect (not the same), such behaviour may be considered as
wrong and we may say that we have found the defect.

• The number of possible inputs is very large.
• The number of possible outputs is very large.
• The number of paths through the software is very large.
• The software specification is subjective. You might say
that a bug is in the eye of the beholder.

It's Impossible to Test a
Program Completely

Amount of Testing:

Software Testing Is a
Risk-Based Exercise

• It can show that bugs exist, but it can't show that bugs
don't exist.
• You can perform your tests, find and report bugs, but at
no point can you guarantee that there are no longer any
bugs to find

Testing Can't Show That
Bugs Don't Exist

• Programmers have bad days
• Programmers often make the same mistake.
• Some bugs are really just the tip of the iceberg.

The More Bugs You Find,
the More Bugs There Are

• the more you test software, the more immune it
becomes to your tests

The Pesticide Paradox

• There's not enough time.
• It's really not a bug.
• It's too risky to fix.
• It's just not worth it.

Not All the Bugs You Find
Will Be Fixed

• If there's a problem in the software but no one ever
discovers it—not programmers, not testers, and not even
a single customer—is it a bug?
• Since you can't report on what you didn't see, you can't
claim that a bug exists if you didn't see it. Make a bug a
bug only if it's observed.

When a Bug's a Bug Is
Difficult to Say

• The specification will change:
• Features will be added that you didn't plan to test.
• Features will be changed or even deleted that you had already
tested and reported bugs on.

A product specification is an agreement among the software development team. It
defines the product they are creating, detailing what it will be, how it will act, what it
will do, and what it won't do.

Product Specifications
Are Never Final

THE PLACE OF SOFTWARE TESTING IN
SOFTWARE DEVELOPMENT CYCLE
The software must be tested …
but when?!

Distribution of bugs
7%
10%
56%
Requirements
27% Design
Other
Code

Importance 23

Phase in Which Found Cost Ratio
Requirements 1
Design 3-6
Coding 10
Unit/Integration Testing 15-40
System/Acceptance
Testing 30-70
Production 40-1000

Relative cost to fix an error 24

• Requirements are a specification of what
should be implemented.

• Requirement describes the need without
describing any implementation details of a
solution to that need
• what, but not how paradigm

Requirement. Definition 25

• Requirements and specifications are incomplete.
• Requirements and specifications change too
often.
• There is a lack of user input (to requirements).

Why software projects fail? 26

1. What the stakeholders expect will be delivered?
2. What the developers are expected to deliver?
3. What the testers are expected to test?

Main questions 27

Important Primarily to Users Important Primarily to
Developers
Availability Maintainability
Efficiency Portability
Flexibility Reusability
Integrity Testability
Interoperability
Reliability
Robustness
Usability

Quality attributes 29

• Unambiguous
• Complete
• Correct
• Feasible
• Necessary
• Prioritized
• Verifiable

Requirement Statement
Characteristics 30

• Complete
• Consistent
• Modifiable
• Traceable

Requirements Specification
Characteristics 31

Relative 2 1 1 0.5
Weights
Feature Relative Relative Total Value % Relative Cost % Relative Risk % Priority
Benefit Penalty Value Cost Risk

Some 2 4 8 5.2 1 2.7 1 3.0 1.22
feature
…
Totals 53 49 155 100.0 37 100.0 33 100.0

priority = value %
(cost % * cost weight) + (risk % * risk weight)

Setting Requirement
Priorities 32

User Functional Design Code Module Test Case
Requirement Requirement Element
UC-28 catalog.query.sort Class catalog catalog.sort() search.7
search.8
UC-29 catalog.query. Class catalog catalog. search.12
import import() search.13
catalog. search.14
validate()

Changes are not so dangerous
when you can manage them

Requirements Traceability 33

1. Validate requirements against objectives
2. Apply scenarios against requirements
3. Perform initial ambiguity review
4. Perform domain expert reviews
5. Create cause-effect graph
6. Logical consistency check by RBT Tool
7. Review of test cases by specification writers
8. Review of test cases by users
9. Review of test cases by developers
10. Walk test cases through design
11. Walk test cases through code
12. Execute test cases against code

RBT Process 35

• 1. Review requirements documents
• 2. Test the requirements
• 3. Define acceptance criteria

Validating the requirements 36

• The author of the work product and perhaps
peers of the author
• The author of any predecessor work product
or specification for the item being inspected
• People who will do work based on the item
being inspected
• People who are responsible for work products
that interface with the item being inspected

Review: Participants 37

• Author
• Moderator
• Reader
• Recorder

Review: Roles 38

• The document conforms to the standard template.
• The document has been spell-checked.
• The author has visually examined the document for any layout
errors.
• Any predecessor or reference documents that the inspectors will
require to examine the document are available.
• Line numbers are printed on the document to facilitate referring
to specific locations during the inspection meeting.
• All open issues are marked as TBD (to be determined).
• The moderator didn't find more than three major defects in a
ten-minute examination of a representative sample of the
document.

Review: Entry Criteria 40

• All issues raised during the review have been addressed.
• Any changes made in the document and related work
products were made correctly.
• The revised document has been spell-checked.
• All TBDs have been resolved, or each TBD's resolution
process, target date, and owner has been documented.
• The document has been checked into the project's
configuration management system

Review: Exit Criteria 41

• Do any requirements conflict with or duplicate other
requirements?
• Is each requirement written in clear, concise, and unambiguous
language?
• Is each requirement verifiable by testing, demonstration,
review, or analysis?
• Is each requirement in scope for the project?
• Is each requirement free from content and grammatical errors?
• Can all the requirements be implemented within known
constraints?
• Are all specified error messages unique and meaningful?

Check List Example for Review
(focused on Correctness) 42

• Decompose high-level requirements into enough
detail to reveal exactly what is being requested
• Make sure that all user classes have provided
input
• Check boundary values for missing requirements.
• Represent requirements information in multiple
ways. Model requirements
• Sets of requirements with complex Boolean logic
(ANDs, ORs, and NOTs) often are incomplete

Missed requirements 43

• Interviews and discussions with potential users
• Documents that describe current or competing products
• System requirements specifications
• Problem reports and enhancement requests for a current
system
• Marketing surveys and user questionnaires
• Observing users at work
• Scenario analysis of user tasks
• Events and responses

Sources 44

• Lets you define the data items included in a change
request
• Lets you define a state-transition model for the change-
request life cycle
• Enforces the state-transition model so that only
authorized users can make the permitted status changes
• Records the date of every status change and the identity
of the person who made it
• Lets you define who receives automatic e-mail
notification when an originator submits a new request or
when a request's status is updated

Change-Control Tools 46

1. Context free questions. The traditional questions are:
Who will use this feature? What does this user want to do with it?
What do they lose? What else does this user want to do in
conjunction with this feature? Who is not allowed to use this
product or feature and what security is in place to prevent them?
2. Writing test cases
begin writing check lists and test cases early in requirements
development.
3. Models, Diagrams
Requirements document is often long and hard to understand.
Requirements described on page 1 can contradict the
requirement on page 100. It’s very hard to remember everything.

Testing Requirements: Basic
Techniques of Analyzing 47

• Be Deterministic: Given an initial system state and a set
of inputs, you must be able to predict exactly what the
outputs will be.
• Be Unambiguous: All project members must get the
same meaning from the requirements; otherwise they are
ambiguous.
• Be Terse: Requirements should be written in a brief
manner, with as few words as possible (500 words).
• Be Explicit: Requirements must never be implied.

Anatomy of a good
requirement 48

• is a test of the requirements to insure that they are written
in a clear, concise and unambiguous manner.
• occurs prior to the review of requirements for content by
the domain experts.
• is intended to provide the domain experts with a better
quality set of requirements to work from, in order to
identify missing requirements, and improve the content of
all requirements.

Ambiguity Review 49

• The dangling Else • Built-in assumptions
• Ambiguity of reference • Ambiguous precedence
• Scope of action relationships
• Omissions • Implicit cases
• Ambiguous logical • Etc.
operators • I.E. versus E.G.
• Negation • Temporal ambiguity
• Ambiguous statements • Boundary Ambiguity
• Random organization

Ambiguity review check list 50

Ambiguous Terms Ways to Improve Them
acceptable, adequate Define what constitutes acceptability and how the system can
judge this.
as much as practicableDon't leave it up to the developers to determine what's practicable.
Make it a TBD and set a date to find out.
at least, at a Specify the minimum and maximum acceptable values.
minimum, not more
than, not to exceed
between Define whether the end points are included in the range.
depends on Describe the nature of the dependency.
efficient Define how efficiently the system uses resources, how quickly it
performs specific operations, or how easy it is for people to use.
fast, rapid Specify the minimum acceptable speed at which the system
performs some action.
flexible Describe the ways in which the system must change in response to
changing conditions or business needs.

51

improved, better, Quantify how much better or faster constitutes adequate
faster, superior improvement in a specific functional area.
including, including The list of items should include all possibilities. Otherwise, it can't
but not limited to, and be used for design or testing.
so on, etc., such as
maximize, minimize, State the maximum and minimum acceptable values of some
optimize parameter.

normally, ideally Also describe the system's behavior under abnormal or non-ideal
conditions.
optionally Clarify whether this means a system choice, a user choice, or a
developer choice.

reasonable, when Explain how to make this judgment.
necessary, where
appropriate 52

robust Define how the system is to handle exceptions and respond to
unexpected operating conditions.
seamless, transparent, Translate the user's expectations into specific observable product
graceful characteristics.
several State how many, or provide the minimum and maximum bounds
of a range.
shouldn't Try to state requirements as positives, describing what the system
will do.
state-of-the-art Define what this means.

sufficient Specify how much of something constitutes sufficiency.

support, enable Define exactly what functions the system will perform that
constitute supporting some capability.
user-friendly, simple, Describe system characteristics that will achieve the customer's
easy usage needs and usability expectations.
53

• SIEMENS AG
• Friedrich-Alexander University Erlangen-Nuremberg,
Software Engineering Department

• Critical Software
• Intelligent Process Control (IPC) System for the board
production technological process, based on the expert
system

IPC System 54

V.1 The expert system should be able to recognize and
identify potential critical situations.

It is too general.
• What does it mean “potential critical
situations”?
• How can this be done?
• How can the system recognize and identify
critical situations?

It’s unverifiable.

Ambiguity Review : Example 55

V.2 The expert system should keep a history of the values
V.2 The expert system should keep a history of the values
that ititis reading at specific time intervals. If the values has
that is reading at specific time intervals. If the values has
risen considerably without any obvious reason during the
risen considerably without any obvious reason during the
last ten measurements, the expert system will inform it.
last ten measurements, the expert system will inform it.

• What values should be gathered by the system?
• How deep should be the history?
• What does ―specific time interval‖ mean exactly?
• What is ―risen considerably‖?
• What is ―obvious reason‖?
• Will? The word ―will‖ identifies an ambiguity called a ―Dangling Else‖. The
sentence with ―will‖ in it tells us what is normally expected to happen
• It? Whom exactly and in what form?


V.3 The expert system should keep monthly history of the
pressure and temperature, that are read every 60 seconds. The
values and time of the measurement should to be stored in DB.
If pressure has risen on more then 10 pts and variation of
temperature is less then 3 pts, then the IPC system should
display to the operator message ―Warning: critical rising of
pressure‖ and produce alarm signal. The warning message
should be displayed until an operator presses button ―close‖
(Sketch Warning.CritialPressure)


• The most of the configuration settings of IPS system shall be
easily upgradeable in future versions
• It is ambiguous: How could we determine that it is ―easily‖
• It’s incomplete:
• What do ―upgradeable‖ mean? How (in what way) should it be
done?
• Which exactly settings do ―the most of configuration settings‖
include?
• In what number of version this functionality will be necessary?
• It’s unverifiable.

IPC. Problems in
requirements 58

• The IPC system should not allow an operator functions , the
consequences of which might be disastrous.
of which might be disastrous.
• It’s incomplete and unverifiable:
• What does ―disastrous‖ mean?
• ―may be‖ is very fuzzy, ―may be‖ but ―may be not‖. How can
be verified the functions the results of which might be
disastrous?
• What does ―should not allow‖ mean? Someone can assume
that buttons for some functions will be disabled, another - that
an operator will not see such functions, another one that the
IPC system should produce alarm message.

IPC. Problems in
requirements 59

• The error message shall appear when the measurement on
operator demand is not done, and there are problems with
network or invalid sensor ID.

• Ambiguous logical operators
• (when the measurement is not done, and there are problems
with network) or (invalid sensor ID)
• (when the measurement is not done), and (there are problems
with network or invalid sensor ID)
• What does ―problems with network‖ mean?
• What is ―invalid sensor ID‖?

IPC. Problems in
requirements 60

• The error message shall appear when the measurement on
operator demand is not done, and there are problems
with network or invalid sensor ID
• Cause
• A - when the measurement on operator demand is not done
• B - there are problems with network
• C - or invalid sensor ID

• Effect
• E - The error message shall appear

Cause-Effect graph 61

Cause Test #1 Test #2 Test #3 Test #4 Test #5

A T T T F F
B T T F F T
C T F T T F
Effect
D T T F F F
E T T T T F

E =AB+C

Only necessary tests

Conditions coverage

Decision table 63

• Use-Case Diagram
• Data Flow Diagram
• Entity Relationships Diagram
• State-Transition Diagram
• Activity diagram
• Class diagram
• Decision Tables and Decision Trees

Analysis models 64

• Product name, Version, Date, File, Revision history

1. Use-Case Name
• Brief Description
2. Flow of Events A Picture Is Worth
1024 Words
• Basic Flow
• Alternative Flows
3. Special Requirements
4. Preconditions
5. Postconditions
6. Extension Points

Use-Case Specification 65

• Actors
• Use cases
• Constraints
• Relationships
• Association
• Inclusion
• Extension
• Generalization

USE-CASE UML 66

• Ask users to describe how they will determine
whether the product meets their needs and is fit
for use.
• Base acceptance tests on usage scenarios

Define acceptance criteria 68

• Schedule and cost overruns
• A product that doesn't satisfy user needs or doesn't meet user expectations
• A product that requires corrections and patches immediately following
release
• Team member frustration, loss of morale, demotivation, and staff turnover
• Extensive rework
• Duplication of effort
• Missed market window or delayed business benefit
• Loss of market share or revenue
• Product returns or market rejection of the product
• Reduced feature set delivered
• Software that isn't testable

Common Symptoms of
Requirements Problems 69

• Fewer requirements defects
• Reduced development rework
• Fewer unnecessary features
• Lower enhancement costs
• Faster development
• Fewer miscommunications
• Reduced scope creep
• Reduced project chaos
• More accurate system-testing estimates
• Higher customer and team member satisfaction

Benefits of good
requirements 70

Requirements Based Testing

More Related Content

What's hot (20)

Viewers also liked (9)

Similar to Requirements Based Testing (20)

More from SSA KPI (20)

Recently uploaded (20)

Requirements Based Testing