Pairwise testing

www.kanoah.com
ABOUT US
Kanoah is an innovative company
providing ground-breaking solutions to
software testing professionals on the
Atlassian JIRA platform

About Kanoah Tests
Kanoah Tests is a full featured test management, integrated seamless into
JIRA with the same look-n-feel. No need to learn or switch between different
applications
Coordinate all test management activities including test planning, authoring,
execution, tracking and reporting from a central location
Kanoah Tests enables you to track testing progress and quality to foster
collaboration and visibility across traditional and agile teams
Get real-time insights into your testing progress with out of the box
reports
Easily integrate your automated tests and submit test results with Kanoah
Tests’ powerful REST API or use the API to automate many areas of the
application

After looking for several years at plugins for test management we finally found
Kanoah Tests. The other solutions were either too complex, didn't integrate well
with Jira, or were focused on a single project. Kanoah Tests proved to be an
elegant solution that allowed linking between any project. Kanoah has been
very responsive to feedback, requests, suggestions as well bugs. The customer
service is awesome. I'd highly recommend Kanoah Tests to teams of any size
looking to simplify test management and consolidate tools.
Don
Pierce
Robert
Murhamer
Liked Kanoah from the moment I discovered it. Integrates nicely with JIRA and
especially with Agile. Test cases can be authored right from the story level, but has
all other functionality a Test Case Management solution would need to have,
ranging from creating test plans, executing test cases, importing test cases, API for
automation, a.s.o. Additionally the team at Kanoah is amazing and responding to
any question very quickly. It wasn't hard to sell to my management to purchase
Kanoah. Will highly recommend Kanoah to anybody.
Zour
Brosh
I just start working with Kanoah and I am impressed how it's simple to manage tests
without unlimited non-used features like in most of the test management tools and
still to get the needed functionality and results. The integration with Jira is a great
working solution that enable to share testing and development in simple way on
one system. I recommend to use Kanoah for testing management. It will help to do
Kanoah as much as possible customizable like Jira to match each group
methodology
For more reviews, visit: https://marketplace.atlassian.com/plugins/com.kanoah.test-manager/server/reviews
Reviews

Key Features
Native seamless integration with JIRA
No need to learn or switch between
different applications
Perfect for agile & traditional testing
approaches
Manage, organize and track all your testing
efforts in a central place
Reuse test cases across your projects
Powerful REST API
Establish clear traceability between
requirements, test cases, and defects
Execute test cases and track results that
matter
Get real-time insights into your testing
progress with out of the box reports
Live statistics accessible to your entire
team

Benefits for the testers
No need to lear or switch between
different applications
Reuse test cases across projects for
regression
Link test cases to requirements and
defects
API support for automated efforts
Benefits for the teams
Informed decisions based on real-time
insights
End-to-end traceability and impact
analysis
Centralized Test Management
Save time and increase productivity

Why choose Kanoah Tests
Coordinate all test
management activities right
inside JIRA
1Testing right
inside JIRA
Take advantage of the built-in
reports to track the results and
measure progress
2Make informed
decisions
Kanoah Tests users receive
priority support, even
during trials
3Stelar
support

What is Pairwise Testing?
In computer science, all-pairs testing or pairwise testing is a combinatorial
method of software testing that, for each pair of input parameters to a system
(typically, a software algorithm), tests all possible discrete combinations of
those parameters.
Using carefully chosen test vectors, this can be done much faster than an
exhaustive search of all combinations of all parameters, by “parallelizing” the
tests of parameter pairs.
The most common bugs in a program are generally triggered by either a
single input parameter or an interactions between pairs of parameters.
Bugs involving interactions between three or more parameters are both
progressively less common and also progressively more expensive to find,
such testing has as its limit the testing of all possible inputs.
Thus, a combinatorial technique for picking test cases like all-pairs testing is a
useful cost-benefit compromise that enables a significant reduction in the
number of test cases without drastically compromising functional coverage.
[1]

Combinatorial test data generators generate data tables for testing. The most
basic, commonly used combinatorial data generation strategy is what is
known as pairwise testing, all-pairs testing, covering arrays or Taguchi
designs.
Pairwise testing tries to alleviate the following practical problem: your system
has ten configuration parameters, and every configuration parameter has ten
different, interesting values. How can you test that your system behaves
correctly with all the different configurations?
Because there are 10 billion different configurations available, you cannot test
for all of them. The proposal of pairwise testing is that it is enough to test for
all the possible value combinations for anytwo of the ten variables.
A rude calculation shows that there are 45 configuration variable pairs, and
for every pair there are one hundred (ten times ten) different pairwise set
values available, so 4,500 tests should suffice. This is obviously an
improvement over 10 billion, but actually all the pairs can be covered in a
much smaller test set, because every single test can cover multiple pairs at
the same time. [2]

In computer systems, defects usually involve a single condition, independent
of any other condition in the system. If there’s a problem with a device or a
variable or a setting, the problem is usually in that device or variable or
setting alone.
In testing, we want to be sure that we don’t miss problems based on conflicts
between two or more conditions, variables, or configurations, so we often test
in combinations in order to find defects most efficiently.
Suppose that a defect depends upon every variable in the system to be in
one specific state. If only we could try all of the possible combinations by
checking each aspect of the system in each of its states, combined with all of
the other aspects in each of their possible states, we would thereby test every
state and every combination of states in the system. [3]

All Pairs Technique
Suppose that there are five variables,
each represented by a letter of the
alphabet. And suppose that each
variable can contain a value from one
to five.
Let’s set variables A, B, C, and D all
equal to 1. If those values are fixed,
variable E can have a value from one to
five. We’ll keep track of the total
number of combinations in our leftmost
column; our first five combinations will
be numbered from 1 to 5. [3]

All Pairs Technique
Let’s set variable D to two instead of
one, and keep A, B, and C at one. Five
more combinations, numbered 6-10.
Then we’ll set D to 3, then to 4, and then
to 5, setting E to values from 1 to 5 each
time. We’ll have to go through 25
combinations before we exhaust all of
the combinations for D and E, and only
then can we change C to from 1 to 2.
The columns on the right will roll over
relatively quickly, but as we move
leftwards, columns A, B, and C will
change much less often. [3]

All Pairs Technique
In fact, if there are only five variables in the program, each with five possible
states, the program would be absurdly simple, yet we’d have to test 3125
combinations.
If we could test one combination per minute, complete testing would take seven
and a half days of continuous, seven-hour-a-day tests. There’s got to be a way to
reduce the number of tests into something that we can handle. [3]

All Pairs Tables
Imagine that our defect depends upon
checkbox A being cleared (that is,
unchecked) and checkbox B being set
(checked). If we try all of the possible
settings in combination with one
another, we’ll find the defect.
Combinations 1, 3, and 4 work all right,
but Combination 2 exhibits the
problem. We’d require four tests to
make sure that we had covered all of
the combinations in which A and B
could be found. Two variables, and two
settings for each; four combinations. [3]

All Pairs Tables
Now consider something only a little
more complex: a black-box test of a
dialog with three sets of radio buttons
on it, representing travel needs in North
America.
The first set of radio buttons specifies a
destination, by country; the second
notes the choice of an aisle or window
seat; and the third affords the choice of
coach, business, or economy class.
When the user makes her selections
and presses “OK” to close the dialog,
the application will take on a different
path of execution for each combination
of given options in given states.
Assume now that there is a defect that
depends upon two conditions. This is
known as a double-mode fault. [3]

All Pairs Tables
The most difficult kind of problem to
find by black-box testing is one in
which several variables are involved,
and each must be in a specific, unique
state to trigger the problem.
For example, when the country field is
set to “USA”, the seat choice to “aisle”,
and the service class to “First”, then
and only then will the bug be triggered.
This is known as a triple-mode fault, or
more generally as a multi-mode fault,
which describes faults associated with
three or more parameters.[3]

All Pairs Tables
This table makes sure that Canada
is tested at least once with each of
Coach, Business Class, and First
Class, and with the checkbox in the
Aisle state and the Window state.
Similarly, every option is tested
with every other option.
We capture the double-mode fault
exposed in combination 3; we need
not test combination 12, since its
pairs are covered by combination 1,
combination 11, and combination 3
(which tests Destination=USA and
Class=Coach, and exposes the
conflict between them). In this case
we cut in half the number of test
requirements, from 18 to nine. [3]

Orthogonal Array
Orthogonal array testing is a systematic way of testing all-pair combinations
of variables using orthogonal arrays.
It significantly reduces the number of all combinations of variables to test all
pair combinations.
By definition, it is a two-dimensional array constructed with special
mathematical properties such that choosing any two columns in the array
provides every pair combination of each number in the array. [3]

Orthogonal Array
There are three rules for selecting an orthogonal array:
There must be at least as many columns as factors (i.e. number of columns is
equal to number of factors).
There must be at least enough numbers in the columns to hold the options for
each factor. i.e. If you have two options for a factor, then test cases should be
there for both the values and if there are three options for a factor, then test
cases should be there for all the three values. If we have spare numbers that
don’t map to any option for that factor can be represented by a tilde ( ~ ).
There must be at least as many rows or test cases as the product of the two
largest numbers of options. For example, if one factor has 4 options and
another has 3 options and yet another has 2 options, then 4 x 3 = 12 rows. [3]

Orthogonal Array
1) Consider two factors X
& Y. Now list down all
possible combinations for
the pair which will be 0 0,
0 1, 1 0 and 1 1.
2) Consider the factors X
& Z and list down all
possible combinations for
the pair and here notice
to shuffle the order while
writing the pairs i.e. 0 0,
0 1, 1 1 and 1 0. [3]

Orthogonal Array
Note that here we changed
the order of values of Z for
the 3rd and 4th rows.
Write down all the values of X
similar to how it is written for
the pairs X & Y (2). While
writing down the values of Z,
initially all the values were
written as same as Y in pairs
X & Y. But then later on to
cover all pairs of
combinations for the factors
Y & Z, the values of Z in 3rd
and 4th row is interchanged
as highlighted. [3]

Orthogonal Array
3) Now consider the factors Y & Z, all
the combinations for the pairs is now
listed.
Here notice that if we had not
changed the order for the pairs X &
Z, all combinations would not have
been covered for pairs Y & Z Or else
in another way, write down the
values of Y from X & Y factor and
values of Z from X & Z factor.
If we combine this we will get the
order of combinations for Y & Z. [3]

Pairwise Testing Techniques - Orthogonal Array
Now the table shown next are
the test cases or test
combinations developed using
orthogonal array.
Note that there are only four
test cases after applying this
technique, half compared to
conventional approaches. [3]

Pairwise Testing Tools
PICT – ‘Pairwise Independent Combinatorial Testing’, provided by Microsoft Corp.
IBM FoCuS – ‘Functional Coverage Unified Solution’, provided by IBM.
ACTS – ‘Advanced Combinatorial Testing System’, provided by NIST, an agency of
the US Government.
Hexawise
Jenny
Pairwiser by Inductive AS.
VPTag free Pairwise Testing Tool. [4]

Conclusion
Pairwise testing technique can dramatically reduce the number of
combinations to be covered but remains very effective in terms of fault
detection.
It is indeed a smart test design technique that guarantees a win-win situation
for both test effort and test effectiveness.
During the Test planning phase of software testing, Pairwise testing technique
should always be taken into consideration. Either we are doing it manually or
using any tool to generate test cases, it becomes a necessary component of
the test plan because it in turn affects Test estimation. [4]

References
[1] https://en.wikipedia.org/wiki/All-pairs_testing
[2] https://www.conformiq.com/2012/01/understanding-pairwise-test-generation/
[3] http://www.developsense.com/pairwiseTesting.html
[4] http://www.softwaretestinghelp.com/what-is-pairwise-testing/

Next Steps
Getting Started Guide
GETTING STARTED RESOURCES FREE TRIAL
Documentation
Support
Tutorials
What we're reading this week
Blog
Start you 30-day free trial now

Pairwise testing

More Related Content

What's hot (20)

Viewers also liked (13)

Similar to Pairwise testing (20)

More from Kanoah (7)

Recently uploaded (20)

Pairwise testing