To requirements and beyond...
0 likes339 views
The document outlines a presentation on requirements and testing. It discusses the need for good requirements, how to structure requirements using a Test Requirements Hierarchy (TRH), and provides examples of how to construct a TRH for a train ticket application. It also discusses measuring and tracking test progress and coverage using the TRH, and adjusting test strategies based on measurements to meet deadlines.
To requirements and beyond...
- 2. 2 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me…what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
- 3. 3 WHO’S WHO? Mais que diable allait-il faire dans cette galère? - Molière
- 4. 4 A Schizophrenic View on Me, Myself and I, … I’m 20 ! Olivier Denoo I’m Belgian
- 5. 5 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me…what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
- 7. 7 Functional Defect Project : Louvois Late payments, wrong salary calculation… The French Ministry was unable to fix the software issue… Families are appalled 346 Mio € Loss – no usage French Servicemen hit by a payroll software issue
- 8. 8 Security Defect Orange TelCo blamed – Privacy rules over hacking
- 9. 9 User Satisfaction issue Windows 8 – Even Vista was better!
- 10. 10 User / Customer Issue Technical Issue Functional Issue Ow! Let’s go fishing! Production Maintenance Poor Non- Functional Testing Poor Code quality Poor Technical architecture Bug - Defect Failure - Issue Insufficient involvement Poor business processes Communication Needs Market Poor Requirements Poor Functional Architecture Poor functional Testing Insufficient involvement Poor Business Processes Communication Needs Market Poor Non- Functional Testing Poor Code quality Poor Functional Testing Production Maintenance Poor Technical Architecture Poor Requirements Poor Functional Architecture CMMI® for DEV
- 11. 11 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me…what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
- 12. 12 LANGUAGE & PATTERNS A matter of
- 13. 13
- 14. 14 How Do We (testers) speak? • Constraints: – Time – Costs – Resources • Requirements: – Coverage – Test depth Coverage Depth Costs, Resources Time
- 15. 15 And What About the Others?
- 16. 16 Understand them Speak their language Fulfil their needs Put yourself into their shoes
- 17. 17 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me…what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
- 18. 18 Use Patterns… • …and make TESTABLE requirements: – FUNCTIONAL • As ROLE I need ACTION in order to OBJECTIVE + context qualifyers • As user I need to access my personal file in order to complete it with the necessary info – SYSTEM • When TRIGGER the system NAME must ACTION/PERMISSION to WHOM/WHAT in order to OBJECTIVE + qualifyers • When book is available, BorrowBook must allow subscribers to borrow if subscription paid and limit not reached
- 20. 20 Often (very often)… • Absence of references • Few measures and metrics (or none) – Unknown coverage – Unclear efficiency – No capitalization • Decisions = estimations • Management cannot evaluate properly
- 21. 21 What Would we Want to Know? • What’s left to do? • Am I on track? • What can I improve? • What is MY quality? • Do defects get solved? • What did I learn? Not measuring = Not knowing
- 22. 22 • No clear specification • What is really needed? – Basic or advanced functionality? – Usability? – Performance? – Security? • When do we have enough tests? – How critical is the functionality? – Deep or shallow test? • How long to design, execute the tests? • What should we test first? • How can we draw a conclusion from the tests? Difficulties Encountered
- 23. 23 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me…what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
- 24. 24 A TRH??? « Logic : a good tool that is sold to us with no user manual » - Pierre Véron
- 25. 25 Hierarchical tree view of application requirements Business process thinking How to reach business goals (notion of benefit) Process-oriented Tracing down from process to functions Quality reference Structuring with a TRH test requirement 1 test requirement 1.1 test requirement 1.2 test requirement 1.2.1 test requirement 1.2.2 test requirement 1.2.3 test requirement 1.3 test requirement 1.4 test requirement 1.3.1 test requirement 1.3.2
- 26. 26 The Fundamental Test Process Test planning and control Test analysis and design Test implementation and execution Evaluating exit criteria and reporting Test closure activities TRH used throughout these stages
- 27. 27 Unit testing Integration testing System testing Acceptance testing Requirements Specification Functional Specification Global design Detailed design Code Business Objectives The TRH and the V-model Test requirement Business requirement System requirement
- 28. 28 The Problem with Requirements • Requirements form the basis for testing • Requirements are not always testable – Often imprecise, ambiguous – Often incomplete – Can be contradicting or totally wrong – Sometimes, requirements simply don’t exist • Challenges – make imprecise, incomplete requirements testable – Identify wrong, missing requirements
- 29. 29 How would you test a train ticket application? A Small Example …
- 30. 30 Constructing the Hierarchy HOW? Let’s bring the Money In Sell tickets Exchange tickets Refund tickets
- 31. 31 Constructing the Hierarchy HOW? Let’s bring the Money In Sell tickets Exchange tickets Refund tickets International National Full price Off peak Reduced - Young Reduced - Sr Reduced Loyalty HOW?
- 32. 32 Asking How, Creatively • Use alternative formulations: – What needs to happen? – Are there any related problems? – What are you likely to have done previously/next? – What needs to happen when this fails? – How does the user know this? – How could things be screwed up? • Use basic test specification techniques to stimulate responses. – “Manage things” → CRUD (create/read/update/delete) – Equivalence partitioning – Boundaries – Negative cases • Make sure sub-requirements cover parent 100%
- 33. 33 Establish Business Context Let’s bring the Money In Sell tickets Exchange tickets Refund tickets International National Full price Off peak Reduced - Young Reduced - Sr Reduced Loyalty WHY? WHY? WHY?
- 34. 34 Asking Why, Unaggressively • The question “why?” by itself can be quite intimidating (or even look foolish) • Use alternative formulations: – What is the purpose of that? – Can you explain why you need it to do that? – What’s the relation to your business? – What business problem is this solving? – What was the thinking behind that? – What is the underlying reason for that? – What would happen if that feature was not available? – …
- 35. 35 How to Get There? • Previous Knowledge & Experience • Reading and retro-analysis • Face-to-face meetings • Workshops NEVER ASSUME !!! - Loyalty is not about calculating & redeeming points It is about COMMUNICATION
- 36. 36 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
- 37. 37 A TRH What for? • Prioritise • Communicate • Pilot • Estimate • Test strategy backbone test requirement 1 test requirement 1.1 test requirement 1.2 test requirement 1.2.1 test requirement 1.2.2 test requirement 1.2.3 test requirement 1.3 test requirement 1.4 test requirement 1.3.1 test requirement 1.3.2 1 1 1 2 2 3 3
- 38. 38 Test Implementation and Execution TRH Test design spec Test design spec Test procedure spec Test case spec Test procedure spec Test Plan bugs Baseline planning Progress report
- 39. 39 TRH and Test Design TEST SPECIFICATION ID: XYZ PURPOSE: Verify that search function examines articles. APPROACH REFINEMENTS: …
- 40. 40 TRH and Process Test procedure Scenario Data Validation points Physical Design WHAT ? Test procedure Scenario Data Validation points Test case1 Test case2 Test case3 Test case5 Test case4 Test Requirements Hierarchy WHY / WHAT? test requirement 1 test requirement 1.1 test requirement 1.2 test requirement 1.2.1 test requirement 1.2.2 test requirement 1.2.3 test requirement 1.3 test requirement 1.4 test requirement 1.3.1 test requirement 1.3.2 Logical Design WHAT?
- 41. 41 Who’s the Pilot?…Damn’ it’s the TRH! • Nb testable requirements (leaves) – Identified / Developed / written – % Coverage(s) – % Executed Progress (t vs. t-1 vs. planned) – Risk / Impact (before and after execution) – Planning / Effort
- 42. 42 In Practice • Planning - Execution – 100 requirements to test – 100 persons*days budget – Writing / Execution = 60/40 • Quality – 50% requirements coverage – 50% passed product – Grey zones (50 shades)?
- 43. 43 Software Tools…Who Cares? • There are (too) many of them – From Excel to Doors via TestLink & others – Integrated in suites (HP, IBM) or not – OpenSource or Commercial • Most (at least the known ones) are good • Tools are NOT the issue
- 44. 44 …Provided they Allow… • Real-time share and follow • Information and reporting (collect and process) • Decisions • Anticipation • Adapt the tool to YOUR organisation
- 45. 45 Word Processor in Outline Mode
- 47. 47 Spreadsheets
- 48. 48 …but How to Use them Matters • They must be robust… – The « in-house » requirements tracker • …fit to your needs and organisation • …and should allow to share and pilot – Customer / supplier « encode twice »
- 49. 49 Requirements • Total Nb & Testable >> coverage • Repartition / severity • Repartition / phase / version • Repartition / priority • Nb incorrect or maintained requirements • Nb scenarios / Requirement • Nb unused requirements • Nb added requirements • Nb changed requirements
- 50. 50 Test Scenarii – Test Designs • Total & Testable (Nb) • Répartition / severity • Repartition / phase or version • Repartition / priority • Nb scenarii reviewed and/or incorrect • Nb scenarii / Requirement • Nb scenarii unused • Nb scenarii added Measurements details Fine but complex vs.coarse but blurry estimations Differs from phase to phase depending on goals
- 51. 51 Defects • Total Nb • Repartition / severity • Repartition / priority • Repartition / status • Repartition / responsible • Nb found / unit time • Nb false positive • Nb unsolved • Nb found next phase
- 52. 52 Adjust your Strategy You are HERE Time Scope EXPECTED Deadline EXTENDED Deadline 100% 0% 50% 1 3 2 66% Productivity
- 53. 53 Different Mitigation Options • Option 2 – Limit Scope (shrink till you drop) • Quality drop • Increased risk – Let it slip (the whoosh sound of dealines) • Late market presence • Extra costs • Option 3 – Increase resources (Stakhanov – Taylor) • Extra costs • 2 women + 4,5 mths = NO baby
- 54. 54 One Example • Mitigating options 2 (slippage) and 3 (increase ressources) – Use what you already know to predict and plan • Risk oriented strategy – Sanity check – Flexible resources – Planning by roles and capability • Use existing resources • Identify and manage activities – Test new functionality – Re-test (bug fix) – Regression test (re-validation) – …/… • Multiply PC (automation) – Execute based on risk / priority • Most important first • Most important = deeper and longer BA
- 55. 55 Estimation Model Estimated (found) Fixed Planning Exported (next phase) Inherited (previous phase) TEST Phase n
- 56. 56 Test Campaigns 1 2 3 4 5 6 7 Go / No Go SP 2.1 SP 2.2 SP 2.3 ? Production? Production? h1 h2 h3 Inherited from SP1.x h1 h2 Σhi Σhi d2 d3 Manual Tests (n) Manual tests (n+1) + automated (n) • ! Performance • ! Availability • ! Stability 27/03 02/05 21/05 Inherited from SP2.0 d1 0 ? ? ? Correction > c1 Correction > c2 SC 21/03 18/04 8 ? 2/04 di = Nb estimated defcts / phase ci = Nb fixed defects in the same phase hi = Nb inherited defects Σ’hi Σ’hi
- 57. 57 And if you Have…TIME • Today (status time) – Consumed vs, attended resources – Forecast based upon experience • Mean time (write / fix / execute) • Mean time (code / analyse / fix) • Application up time • Application performance • MTBE, MTBF Good stop criterion… if enough maturity
- 58. 58 Ressources metrics • Time span / person (per profile) – # Hrs / test activity • Preparation • Planning • Design • Environment • Execution • Maintenance • Fixing tests • Reporting • Overhead expenditure
- 59. 59 Work Product Trend 0 50 100 150 200 250 300 350 400 450 JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC planned designed executed passed
- 60. 60 Do NOT Overperform Date 27/08 Sprint 2 Release 3 # days 15/23 15 # Testers Effective Cum. % planned resources for SP2.3 Execution test designs 5,5 Check printed documents 3 Ad hoc tests 5 63% Quotation (Product Management) 0 100% Re-test SP1 + SP2 ad hoc defects 0 100% Traduction 0 100% Total resources coverage Release 3 13,5 62% Availability - Lost time Availability of the application that day 100,0% week 1/5 week 2/5 week 3/5 week 4/5 week 5/5 Average availability for SP2.3 85,7% 80,1% 78,6% 95,6% 85,7% Test Weather Report 58% Test execution (2849 data sets) # data sets foreseen that date 170 Found defects # data sets tested that date 90 % data set coverage SP2.3 - against planned ds 53% OK NOK Not testable Intermediate # data sets executed SP2.3 2148 1850 233 0 65 86,1% 10,8% 0,0% 3,0% % data set coverage SP2.3 - against total ds 75,4% Docs waiting for check # SP2 defects foreseen to be re-tested during SP2.3 445 # SP2 defects re-tested that date 22 OK NOK Not testable # SP2 defects re-tested while executing ds SP2.3 211 184 26 1 87,2% 12,3% 0,5% % re-test defects coverage 47% Average datasets/p/d foreseen for SP2.3 19,9 *Average test design (15 ds/p/d) and check document (29 ds/p/d) Average datasets/p/d effective that day 16,4 Average datasets/p/d effective in SP2.3 17,3 Re-test SP2 AD HOC defects # SP2 ad hoc defects foreseen to be re-tested during SP2.3 106 # SP2 ad hoc defects foreseen that date 0 NOK + OK + # SP2 ad hoc defects re-tested that date 0 OK NOK New defect New defect # SP2 ad hoc defects re-tested in SP2.3 99 81 16 0 0 81,8% 16,2% 0,0% 0,0% % SP2 ad hoc defects re-test coverage 93% Average defects re-tested/p/d foreseen for SP2.3 10,0 Average defects re-tested/p/d effective that day - Average datasets/p/d effective in SP2.3 - % data sets executed SP2.3 Cumul # SP2 ad hoc defects re-tested in SP2.3 Cumul # SP2 defects re-tested while executing ds SP2.3 # Defects # encoded defects that day 18 Urgent High Medium Low # active defects in Defect Tracker for SP2.3 102 20 51 24 7 102 # active defects in Defect Tracker 411 29 257 110 15 411 Open IT exam Pend. TEST Sub-total Active Whole IT Factory 65 0 21 86 Ready TLF Pend. TLF User Exam Test Factory 2 297 26 325 Closed Deferred Archived Not active defects 454 47 1507 2008 Test exam 0 0 Remarks: Today : Finalised retest of fiches linked to "marques-modèles" has been done Application has been better regarding response time but a lot of instabilities (T000): see daily urgent defects sheet for details. Program for tomorrow : Same as today 2419 411 Fitness Program Data set foreseen / executed 0 500 1000 1500 2000 2500 3000 5/aug 7/aug 9/aug 13/aug 19/aug 21/aug 23/aug 27/aug 29/aug 2/sep 4/sep 6/sep Executed Cumulated DS DS foreseen this day (intial planning) 120 140 0 500 1000 1500 2000 2500 3000 Availability of the application 91%92% 24% 100% 93% 64% 89% 82% 86% 100%100%100% 92% 71% 100% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 5/aug 7/aug 9/aug 11/aug 13/aug 15/aug 17/aug 19/aug 21/aug 23/aug 25/aug 27/aug 29/aug 31/aug 2/sep 4/sep 6/sep Availability Retest defects SP1 foreseen / executed 0 20 40 60 80 100 120 140 5/aug 7/aug 9/aug 13/aug 19/aug 21/aug 23/aug 27/aug 29/aug 2/sep 4/sep 6/sep Re-tested this daySP1 + SP2 ad hoc Foreseen to be retested this day (intial planning)
- 61. 61 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
- 62. 62 Yes, Sure…However • You are also accountable for Quality • You can set-up and follow what everyone understands – Added value – Business means • You bring the missing link to life
- 63. 63 I Can Do Everything – I’m An Expert
- 64. 64 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
- 65. 65