Choosing an alm tool set

1
Ian McDonaldHND Dip BSc PGCE MBCS CITP CSci MInstP CPhys EurPhys MIET IEng ISEB Test Practitioner
Choosing an Application Lifecycle Management (ALM) Tool Set
© 2013 Ian McDonald

2
Introduction
I frequently get asked – which tool do
you recommend?
There is no short simple answer and a lot
will depend upon specific needs.
These notes are to help individuals consider
the wider implications of tool choice.
This is a first draft and this will be updated
from time to time.

3
Overview
 There is no golden solution for the choice of
Application Lifecycle Management (ALM) tool sets.
 The wrong choice can adversely impact the ability for
a company to:
 Grow
 Be competitive
 Deliver successfully.
 Too often key attributes for tool selection are forgotten,
either due to the lack of experience of those choosing
a tool set, or because they allow themselves to be led
by tool vendors, tool partners or choices made in other
companies for meeting different needs.
 These notes are written as an independent consultant
with no vested interests in any individual tool supplier.

4
Application Life Cycle Management
With ALM tool sets we are focusing on 5
specific areas that support the
development lifecycle. These areas are:
Management of Requirements.
Management of Test Cases and Scripts.
Management of Test Results.
Management of Defect / Fault Reports.
Creation of reports for managers, QA and
process improvement.

5
Lifecycle Types
 In managing the application lifecycle, we need to
remember that there are different types of lifecycle and
some tooling may be less or more appropriate than
others. These life cycles include among others:
 Agile including lean, Scrum, RUP, ORUP and many others.
 V-Model.
 Waterfall.
 Spiral (continuous development and deployment).
 However many tools are not tied to one specific
lifecycle and through correct use can be equally
effective across many lifecycle types.

6
First Things First
 Before starting to look at tools, there are a number of
key things that need to be addressed first:
 What is the health status of the lifecycle?
 Who are the lifecycle stakeholders?
 What are the lifecycle strengths?
 What are the lifecycle weaknesses?
 How does the lifecycle benefit the business?
 What are the strengths and weaknesses of the current
support tooling?
 How is the tooling used by stakeholders?
 What are the problems in use?
 What are the limitations?
 Is the tooling fit for the business growth?
 Is the tooling being used consistently across projects?

7
Stakeholders
 In looking at ALM tooling, there are a number
of stakeholders – including, but not limited to:
 Project Managers including Programme Managers.
 Business Analysts
 Developers
 Testers including Quality Assurance Staff.
 3rd
parties who may need to review reports, submit
defects, etc.
 For successful tooling adoption, all
stakeholders need to be fully engaged and
involved.

8
Strengths and Weaknesses
 In choosing a replacement tool it is first important to
understand why a replacement is required. There are
many reasons for change, some possibilities are to:
 Reduce licence costs?
 Reduce tool maintenance costs?
 Improve reliability of delivery?
 Improve production costs?
 Provide greater security in delivery?
 Provide for process improvement?
 In supporting your goal, where are the strengths and
weaknesses for your current tool set?
 Are there any other opportunities in changing tools?

9
Cost Opportunity
 Tooling must ultimately benefit the business.
It is not there to make life simpler for a group
of individuals. There must be a financial return
for the business to justify the upfront
investment.
 The best approach to identifying the
opportunity is to use metrics. However for
companies who have as yet to collect metrics,
the best they can do is to either refer to typical
industry metrics or to employ a consultant who
can independently assess the situation.

10
Hidden Opportunities
 While in some cases it can quickly be pointed
out that by using tool X, the manufacturing
cost will fall by Y% and so the tool will pay for
itself in Z years, in most situations the
reasoning can be more subtle. Some
examples include among other points:
 Opportunities to cut defect injection – so reducing
production time and costs.
 Ability to deliver more reliable testing, cutting defect
delivery into products, so improving customer
satisfaction and so increasing sales.
 Allowing management greater insight into the
health of a project, so improving delivery control.

11
Stakeholder Benefit
 While the business will need to change,
people often do not like change. They have
been working with a known and understood
process and change will impact their daily
work life. Sometimes it may mean for some
individuals extra work in certain parts of the
lifecycle. It may mean that an initial task takes
longer. So it is important that stakeholders are
engaged and that they see there is a clear
overwhelming benefit to them personally in
addition to the business argument.

12
The case of the Business Analyst
 The Business Analyst (BA) will typically create requirements. These will get passed over the
fence to the developers, who often will reinterpret requirements, try to plug gaps and ignore
requirements that cannot be tested and even over-engineer the solution, adding cost.
 The Developer throws the product over the wall to the test team, who reinterpret some
requirements, others they may get guidance from the developers and eventually they sign off
a product and deliver it with defects still present and perhaps not doing exactly what the BA
envisaged originally. Of course it will all go wrong and who is at fault – the authors of the
requirements!
 So if the developers are actually rewriting the tests – why have a BA?
 The point is that this is happening usually because the lifecycle is waterfall and BA’s do not have the
early support of testers and developers.
 Review of requirements is poor, so the team do not appear to have joint responsibility for requirements.
 The test team are supposed to champion the BA vision and the link between the BA and tester is
probably non-existent in many companies.
 Tests are to reflect the BA requirements. However these requirements have been recreated and
changed with no audit trail, no notification of change and the tests at best may only partially test a given
requirement.
 So for the BA to be a winner – they need:
 Greater control over interpretation and change of requirements.
 Early buy in for requirements – so if there is a fault, it is in the requirement review. The team are
responsible.
 Ability to audit test cases – to ensure that what is being tested is what is required.
 Maintaining the initial vision through to delivery.
 Greater partnership with the test team.

13
The case of the Developer
 The developer typically faces a number of problems:
 Incomplete requirements – needing gaps filled.
 Changing requirements – with no process control. Leading to late detection
of problems in delivery.
 No prioritisation of requirements, making it difficult to make the right choices
in an Agile delivery.
 Multiple repeat defect reports.
 No way of easily grouping defects to create work packages and so reduce
defect fix time overall.
 Defects that are incorrectly reported, due to interpretation of requirements.
 If the developer can gain benefit in solving the above problems then they
will want to buy-in. However usually developers want to maintain:
 Tooling that supports unit testing and static analysis.
 Use of Agile – however can be seen as being Fragile, if necessary processes
and monitoring controls were incorrectly discarded. Agile uses no
Unnecessary documentation – however some items and processes are
necessary, for Agile to be successful.

14
The case of the Project Manager
 The project manager will try and estimate the project delivery time.
depending upon experience, will often underestimate delivery and be
very optimistic. Risk analysis may vary and usually testing is based upon
a percentage of development effort, which can be very unrealistic and
risky if existing code is being migrated as test time can be greater than
development time.
 The project manager will be in receipt of mixed messages, from the
developers claiming delivery of untested code, from the testers, who will
claim that test have failed or not run as functionality is missing. In
practice no code should be counted as delivered unless it has been
successfully tested. Often this type of problem arises due to company
bonus incentives that are driving the wrong behaviour.
 With delivery timescales sliding, the project manager wants to know
quickly in real time:
 What is happening with requirements creep – how is this being managed?
 What code has passed testing and so can be claimed to be delivered?
 Which prioritised requirements are delivered?
 Which defects are holding up delivery?
 When will testing stop – i.e. If the number of defects has been realistically
estimated, when is it expected to reach that target figure, within +/- X days?

15
The case of the Test Manager
 The test manager usually needs far less convincing. They will
want to:
 Bring requirements under configuration control.
 See requirements entered as single logical requirements and have
the ability to link sub-requirements to the parent requirement.
 Have well structured test cases (test vector values).
 Have the ability to create work packages for the creation of test
scripts.
 Manage up issue of test scripts and cases and link results to the
relevant version of scripts and cases.
 Manage the running of test scripts (manual and automation).
 Generate test reports for specific builds and configurations.
 Manage defects and ensure that they are mapped to test results,
scripts, cases and requirements.
 Good reporting, so the test manager knows what is happening at any
time.

16
Other Stakeholders
Other stakeholders outside the company
may have specific needs – especially if
you are working in a partnership.
Do you need a web interface?
Will 3rd
parties submit defects?
Will 3rd
parties access reports?
Are there wider licence implications?
Are there interfaces that need to be
considered?

17
What Else is Happening?
 It is very easy to get bogged down with functional
tests, defects and requirements and forget about other
needs within the lifecycle. These may include:
 Use of unit test supporting tools for developers.
 Risk analysis.
 Automation needs – including GUI and Performance.
 Reporting to higher management and QA.
 Financial monitoring.
 Support policies for tools and environments.
 Database licences and standardisation across company.
 User support issues.
 Training and recruitment.
 These will all need careful consideration.

18
Capability Maturity Model Integration (CMMI)
 CMMI focuses upon a number of core process
areas and ranks these at specific levels. The
levels are:
 1 – Initial – Unpredictable, reactive, poorly controlled.
 2 – Managed – Reactive.
 3 – Defined – Proactive, meets organisation standards.
 4 – Quantitatively managed – Measured and controlled.
 5 – Optimising – Focuses upon process improvement.
 Decide:
 Where are you now on the CMMI scale?
 Where on the CMMI scale do you want to be?
 What is needed to get to the CMMI level you require?
 What is your timetable to reach the CMMI level required?
 Are there budget or cash flow limitations limiting your progress?

19
CMMI Progress
 For a company to progress, it really needs to be within the zone 3
to 5:
 3 – Proactive, meets organisation standards.
 4 – Measured and controlled.
 5 – Process improvement.
 Managers need to move away from having to react to problems
and move towards predicting problems and proactively
eliminating them.
 This means that metrics need to be collected and used to help
that process.
 Standards will be put in place to facilitate the smooth transition of
projects through the lifecycle. However the standards and
processes have to be correctly focused to bring the right level of
success.
 The tooling deployed will (if correctly chosen and set up) assist to
facilitate that success. However tooling is no substitute for good
management.

20
Tool Properties
 Tool Properties need to be examined at
various levels. Among others, the properties
will include:
 Requirements Management
 Test Management
 Defect Management
 Reporting
 Tool Structure
 Tool Ownership
 Environment

21
Requirements Mapping
For Requirements Management the
following features are worth considering:
The ability to link tests to single logical
requirements, will mean that requirements
will have to be broken down to Sub
Requirements as single logical statements.
So the tool needs to support the structure.
Parent – A Level
Requirement
Daughter – B Level
Requirement
Daughter – B Level
Requirement
Daughter – C Level
Requirement
Daughter – C Level
Requirement

22
User Stories are No Substitute
Some tools refer to user stories, this
however is not a substitute for
requirement nesting. Usually user
stories can be handled by separate
parent requirements. The important
point is the ability to have multiple
nesting, down to a single logical testable
statement.

23
Requirement Control
 Controls to manage requirements are
important. Key points to watch are:
 Ability to update requirements and maintain copies
of the old versions. This is important so developers
and testers can view changes.
 A rollback may mean that the previous version of
the test and requirement needs to be restored.
 For project managers they need to understand the
impact of changes. Tests might need changing,
additional tests may be required, tests regrouped
and more importantly risk re-assessed.

24
Requirements Change
Changing a requirement under
configuration control, will mean that both
the development and test team will need
notification. This can be displayed as:
Warning - Requirement has been updated.
Warning - linked test may require attention
as a change in requirement was carried out.
Automated email notification to relevant
stakeholders.

25
Test Case Management
 Test cases are vector sets (inputs and outputs) used within
test scripts. As data can be input in different ways, there may
be many test scripts for a specific test case. There may also
be many vector sets to test a single requirement (e.g.
boundary values).
 Test vector sets may potentially be generated with tooling
such as the Classification Tree Editor (CTE – See separate
presentation). Hence it could be valuable if the ALM tool had
an interface to such tooling as CTE.
 Be able to group test cases into work packages – for running /
creating associated scripts.
 It should be possible to report which test cases have been run
and what the outcome is in terms of success and script
coverage.
 A requirement change should flag any associated test case /
script that may be impacted.

26
Test Script Running
Can you run tests and make appropriate
records to show the results for a test run
against a specific:
Browser
Operating system
Hardware platform
Application version
Work package (i.e. tests carried out by a
specific tester on a specific day).

27
Test Script Management
 As with test cases, any change in requirement should flag the test script.
 The test script matches the requirement and the test results. So if the
requirement or test script changes it is vital that the associated data for the
previous version is not lost. A test result should be traceable to the version of the
test script, test case and requirement at that time. Many tool sets fail to do this
and will incorrectly map old results to updated scripts.
 It should be possible to roll back software builds, so the test data should also have
the ability to roll back and not be subjected to rework. This can be achieved with
good configuration control, however some ALM tools manage this far better than
others.
 You should be ale to filter results for specific test runs containing details of the
configuration used for that set of tests. E.g. List tests run on FireFox 16.0.2,
Winsows 7 SP1, AMD 4 core 300 MHz processor, Test Application version 0.71,
on 05 Sep 2013.
 Automated scripts often require maintenance and so it should be possible to
quickly flip between an automated and manual version of a script.
 Often automated testing is run as part of an extended tool set. The results from
these tests are often required o be reported upon and so need to be interfaced to
the ALM tool set.

28
Test Result Management
 Test results need to reflect the version of test
scripts and test cases used at that time. This is
important as it may be used for:
 Further analysis to identify hidden defects.
 Reporting of the exact testing carried out.
 Analysis of gaps in testing.
 Test results may have associated data and
this needs to be managed. Data can include:
 Screen captures.
 Video playback of any data recorded.
 Attached documents and reports.

29
Video Capture
Some companies may require video
capture of test scripts this could be to:
Provide evidence in case of legal
proceedings. While this is not used widely,
it may become the norm in future years and
so the ability to respond to this can be
important.
Provide records to demonstrate a pattern in
uncovering the fault behind a defect report.

30
Defect Management
 Key properties to ensure are included:
 Ability to map defects to tests and requirements.
 The defect lifecycle can be adjusted to suit your needs – you should
have a defect lifecycle defined.
 What process control does the tool offer? Are roles well defined and
maintained within the tool?
 If a defect references a specific test, does it reference the current
updated version (incorrect) or the version that detected the fault
(correct)? Or better still, does the defect report reference the correct
test version and also provide information on other test versions?
 Attachment of screen captures, video and other documentary
evidence.
 Ability to group defects into work packages.
 Customisation of fields:
 How easy is it to discover if a defect was previously reported?
 Can you obtain process improvement information such as Root cause
Analysis and test Escape Analysis?
 Can you filter data to provide short lists of defects?

31
Reporting
 This is important – you will need to be able to report
on processes and obtain metrics for process
improvement.
 Does the tool allow for reporting across multiple requirements.
For example can you list a defect against a test script, parent
and daughter requirement? Some major tools do not allow for
this simple task.
 Can you produce the appropriate metrics, tables, graphs and
charts you require. How configurable is this?
 How responsive is reporting? Some reports when run can
slow the application considerably, so take care and check out
the reports you want to use, under the load you expect.
 If your company is using an Agile and/or V-Model lifecycle(s),
how appropriate is the reporting for your needs with the
lifecycle of choice?
 Can you apply a consistent tool solution across your
organisation?

32
Licences
 Tool licences can vary considerably. Key
questions to ask are:
 Is the licence for a network, site, company (country
or global)?
 A site licence may not be transferable between networks
at the same site. For companies developing software on
small team networks, some licence structures are not
affordable.
 Is the licence fixed to a specific user or a floating
(concurrent) licence for any user who is logged on?
 Some companies have licences that follow users.
 Some licences are floating (concurrent), so when a user
logs out, the licence is freed up for another individual.

33
Licence Composition
Some products will split licences by
function. For example the defect
management capability may require
different licences.
Decide what type of licences are
required by which users and for how
long – if floating (concurrent) licences
are being used.

34
ALM Licence Quantity
A simple method of estimating concurrent license requirements:
 Each tester needs a full time license. This includes the Test Lead/Manager if they are
also doing full time hands on testing.
 If the Test Lead/Manager is not carrying out full time testing, then they may only need 0.2
of a licence. This allows them to review, control processes and create reports. However
if they are also carrying out additional tasks such as: configuration control, quality
assurance and audits, they may require at least 0.5 of a licence and potentially a full time
licence.
 Each BA needs a minimum of 0.25 of a license. Assuming business analysts (BA’s), do
not generate requirements continuously. This could be higher depending upon the number
of BA’s and the number of parallel projects in the requirements gathering stage. Or it can
be lower, if BA’s generate requirements on another tool then transfer data across to the
ALM tooling. They may still however need licences for audits and requirement
maintenance.
 Developers who are working on defects need ¼ of a licence. They’re generally only
reviewing defects and updating them and therefore would only require the Defect Manager
element – although they may want to see the test, this can have the disadvantage of
making the code pass the test as opposed to being rugged code. However licence
estimates depend upon the number of projects in the testing/defect fixing phase at any
one time. If there is a team configuration controller and a central point where defects are
assigned from, then those users may require 0.5 of a licence.
 Others users will monitor a project or view occasional detail. To support for this use, only
0.25 of a licence is required. Potentially this could reduce to 0.1 of a licence.
 If you have a large team, say over 20, then additional licences may be required to avoid
occasions where use demand overlaps.
 To manage meetings where licence demand might outstrip supply, use a projector for
presentation or create a Webex session and so avoid hogging licences from others.
 You may want to have a policy for inactive use and automated tool logout.

35
Database structure and compatibility
 Your tooling will need to interface with a
database.
 Your company may have a database policy and an
official database upgrade path – is the tooling
compliant with that policy policy?
 Do you already have an appropriate licence on the
relevant server to support the tool database
requirements?
 The structure of the tool database needs to be
maintainable, robust and fit for purpose.
There can be surprisingly different qualities
between even well known tool sets.

36
Tool System Architecture
 Make a list of the tooling that you will need to
integrate. Consider:
 Database requirements.
 Business Analysis requirements.
 Development tool requirements.
 Manual testing requirements.
 Test automation requirements.
 Defect management requirements.
 Reporting – including cross project reporting and
metrics collection.
 Decide which server(s) the tools will reside on
and what the relationship mapping is.

37
Operating in the Cloud
For some companies a cloud solution –
where the tool licence is hosted on a
network outside of the control of the
company – may be the right solution.
However:
Is there financial risk in publishing your
systems vulnerabilities on an external
network?
Are there other commercial or security
risks?

38
Environment Needs
Are there any specific mobile
environments that need to be
considered?
Automation needs
Reporting needs
Are there other environment needs. E.g:
Mainframe / Green Screen
Windows, Unix (different flavours), Linux, etc.

39
Support
 What support does your tool vender offer?
 A free tool may have little support and so will have specific
risks:
 The tool may not support new technology – denying you an
upgrade path.
 There may be a slow response to major errors – denying you
access to the tooling.
 What is included within the maintenance contract?
 There may be phone support? What is the response time –
some large companies can be instant to several hours of
premium number time to access a first tier support person?
 What training is offered?
 Are there user groups, blogs and web help pages?
 How good are the help pages?
 What problems have been recorded on blogs?

40
Access to Experience
Some companies have tremendous
difficulty in recruiting staff. They choose
tools that are not popular in the market
and so they cannot find individuals with
the appropriate skills or who want to gain
those skills. If you are choosing an
uncommon tool, then your training
budget needs to account for new staff.

41
What are the Risks?
 Risks should be assessed.
 Will the tooling be there in 5 years time? Does the company have an
upgrade path and long term development plan?
 How does the company respond to changes in technology?
 How certain are you that the tool will do what you want? Have you
undertaken a trial?
 List the properties you need and ask for at least a demonstration where
those properties are witnessed.
 If the tool became obsolete or stopped working – how quickly can you
get a response and what would the commercial impact be on your
company?
 If the company stops trading – is the tool likely to become unsupported
or taken over by a rival company? E.g. Popular tools like Mercury and
rational were taken over by HP and IBM. Other smaller vendors have
stopped trading.
 What size team supports the tool? Will your concerns ever get
answered?
 If you need consultancy to create a specific plug-in or interface, does
the company have the capability to offer that service?

42
Open Source
 If using Open Source code, then consider:
 There will be commercial implications if you are
selling a product. So make certain you pay for the
right licence types.
 Are there code security and performance issues.
Do not assume the same rigour as with other
commercial tooling. However it is also worth
investigating with other commercial providers –
especially if less experienced.

43
Pick ‘n Mix Solution
 A single tool may not always be the solution.
 Alternatives might include:
 A major ALM tool plugged into the database of
another. E.g. you may want to use one tool for
developers, but use alongside that a different ALM
tool to manage requirements and tests. However
you want to create reports from data collected by
both tool sets.
 Use an ALM tool, but enhance requirement
management by interfacing a separate tool.
 Use a range of separate tools interfaced together.

44
Cost Benefit
 What is the benefit of the tooling solution?
 Who benefits and how?
 What new capability does the tooling offer?
 Is there any loss of capability?
 How is efficiency changed?
 What are the direct costs of ownership – licences,
training, maintenance, etc.
 What are the indirect costs of ownership –
additional development costs, including short term
and long term?
 What is the impact upon recruitment?

45
Try It Out
 Do not take for granted that the tooling does
everything it says on the label. Set up a project and
try it out. Find out what works well and what does not.
 Have a shopping list of key features and check them
off to ensure they work as you would want them to.
 This is a good time to now ask questions of people
who have used a specific tool. Find out what their
specific experience has been in their specific situation.
Remember you are looking for problems. It might be
good for them, it may not be for you. Here you are
trying to identify risk so you can mitigate it.

46
Finally
 These slides only provide a glimpse at the wider
questions.
 Do not assume that large companies tools are best – it
may be that their marketing is just better.
 Do not assume free tools are cheap – there may be
wider cost implications.
 Do not assume that if a tool is good for a developer it
will be good for everyone else. If you are trapped in
this influencing decision you may be disappointed by
what you thought was an ultimate solution. However
you may want to consider interfacing tools to give the
best of both worlds.
 Do not assume that if a tool is right for one company, it
will be right for you. They may have a different market
place, different environments and different risks.

47
END
Slides created by:
Ian R. McDonald
HND Dip BSc PGCE MBCS CITP CSci MInstP CPhys EurPhys MIET IEng
ISEB Test Practitioner
uk.linkedin.com/in/islandsystems

Available for consultancy.

Choosing an alm tool set

More Related Content

What's hot (20)

Similar to Choosing an alm tool set (20)

More from Ian McDonald (9)

Recently uploaded (20)

Choosing an alm tool set