55th세미나 발표자료
2 likes1,047 views
The document summarizes ISO/IEC 20246, which provides guidance on work product reviews. It describes the generic review process, including planning, individual reviews, issue collation, and reporting. It also covers different review techniques like checklist-based and scenario-based reviewing. Finally, it discusses factors to consider when choosing a review approach and provides an overview of the ISO 20246 standard development process and opportunities to get involved.
55th세미나 발표자료
- 1. Stuart Reid PhD, FBCS (stuart@sta.co.kr) © 2015 Stuart Reid Making Your Reviews More Effective Using the New ISO Standard (NIPA – 2nd July 2015)
- 2. • Context - ISO/IEC 20246 & the ISO 29119 standards • The generic review process • Individual review techniques • Review types vs review attributes • Current status • Question? Scope
- 3. Putting ISO 20246 in Context
- 4. ISO 29119/33063/20246 – Structure BS 7925-1 BS 7925-2 IEEE 829 Concepts & Vocabulary Part 1 Testing Techniques Part 4 Documentation Part 3Part 2 Processes Keyword-Driven Testing Part 5 Process Assessment ISO/IEC 33063 Reviews ISO/IEC 20246 IEEE 1028
- 5. ISO 29119 Part 2: Testing Processes TEST MANAGEMENT PROCESSES ORGANIZATIONAL TEST PROCESS DYNAMIC TEST PROCESSES
- 6. TEST MANAGEMENT PROCESSES STATIC TEST PROCESSES ORGANIZATIONAL TEST PROCESS DYNAMIC TEST PROCESSES Testing Processes – the intention REVIEWS STATIC ANALYSIS
- 7. V&V Taxonomy – ISO/DoD Washington, 2010 Verification & Validation Formal Methods Model Checking Proof of Correctness Specification- Based Structure-Based Experience- Based V&V Analysis Simulation Evaluation Quality Metrics Model Verification Static Analysis Reviews Static Testing Dynamic Testing Testing
- 8. ISO/IEC/IEEE 29119 TEST MANAGEMENT PROCESSES ORGANIZATIONAL TEST PROCESS DYNAMIC TEST PROCESSES Testing Processes – the end result ISO/IEC/IEEE 20246 REVIEW PROCESS
- 9. ISO/IEC/IEEE 29119 TEST MANAGEMENT PROCESSES ORGANIZATIONAL TEST PROCESS DYNAMIC TEST PROCESSES Testing Processes – the next step? ISO/IEC/IEEE 20246 REVIEW PROCESS ISO/IEC/IEEE ????? STATIC ANALYSIS
- 10. ORGANIZATIONAL TEST PROCESS TEST MANAGEMENT PROCESSES TEST PLANNING TEST MONITORING & CONTROL TEST COMPLETION ORGANIZATIONAL TEST DOCUMENTATION FEEDBACK ON ORGANIZATIONAL TEST DOCUMENTATION TEST PLAN UPDATES TEST PLAN TEST COMPLETION REPORT DYNAMIC TEST PROCESSES TEST MANAGEMENT PROCESSES TEST PLAN, TEST COMPLETION REPORT, TEST MEASURES TEST MEASURES TEST PLAN, CONTROL DIRECTIVES TEST PLAN, CONTROL DIRECTIVES Overall Test Management Processes WORK PRODUCT REVIEW PROCESS REVIEW MEASURES TEST PLAN, CONTROL DIRECTIVES
- 11. Organise Test Plan Development Identify & Estimate Risks Design Test Strategy Determine Staffing and Scheduling Document Test Plan Schedule, Staffing Profile Test Strategy Estimated Risks Scope Identify Risk Mitigation Approaches Gain Consensus on Test Plan Approved Test Plan Draft Test Plan Test Plan Publish Test Plan Understand Context Treatment Approaches ISO 29119 Part 2 - Test Planning Process Understand Context MUST INCLUDE REVIEWS
- 13. • Anything can be reviewed – in parts or the whole thing • For example: – Contracts – Requirements Specification – Design Specification – Code – Test Specification – User Manual – User Procedures – Project Plan – Test Plan – Development Plan Why ‘Work Product’ Reviews?
- 14. • What do you think? • Find defects • Measure quality • Educate reviewers • Gain consensus • Generate new ideas • Motivate authors to improve their practices Software Reviews – Main Purpose
- 16. Work Product Reviews - Generic Process Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting
- 17. • Identify review scope – what is being reviewed – the purpose of the review (there may be more than one) – relevant supporting information, such as standards – the timeframes for the review • Identify review characteristics – review activities – individual review techniques – checklists • Distribute review materials (as early as possible) – product to be reviewed – baseline specifications – checklists Planning & Preparation Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting
- 18. • Choose appropriate reviewers – agreed with Project Manager – assign specific roles • Agree with each reviewer – availability (and a substitute, just in case) – the review role and focus Planning & Preparation – Reviewers Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting
- 19. • Optional • Review Leader – presents the scope of the review to the review participants – presents related documents to reviewers – explains the role, responsibilities and focus of each review participant – details the timeline for the review • Reviewers (and author) – formally commit to the review Overview Meeting Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting
- 20. • Each reviewer identifies issues with the work product • Identified issues shall be communicated to the review leader / author Individual Review Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting
- 21. • The author collates the Issues identified by the reviewers • Collated issues are analysed and a decision made on what to do with each issue • Issue Review Status – ‘rejected’ – ‘issue to be noted but no action’ – ‘issue to be addressed’ – ‘issue updated due to analysis’ – etc. • Issues are then assigned based on their review status. – to authors – to other responsible individuals (e.g. to update earlier specifications, organization standards) Issue Collation & Analysis – 1 (of 2) Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting
- 22. • Finally the review decision is made on the reviewed product: – used as is – updated based on the identified issues and used – reworked and re-reviewed – discarded • The BIG Question – is this analysis and decision-making done by the author or done by a group of reviewers as part of a Review Meeting??? Issue Collation & Analysis – 2 (of 2) Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting
- 23. • The author – addresses the outstanding issues – organizes for others to fix their outstanding issues in related documents • in ‘earlier’ specifications • in organizational standards – checks that all issues have been addressed – either: • publishes the updated work product • submits the updated work product for re-review • The reviewers – sign off on the final version (confirming that issues addressed) • A Review Report is generated Fixing & Reporting Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting
- 24. Review Timelines Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting 3 days 5-10 days 3 days 5-10 days 2 days They vary!! But need to be set and agreed.
- 25. Generic Review - Roles Reviewer Author Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting Review Leader Reader Recorder/Scribe Moderator QA SME Customer Management Technical Lead
- 26. Individual Reviews Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting
- 27. • Ad Hoc Reviewing • Checklist-Based Reviewing • Scenario-Based Reviewing • Perspective-Based Reading (PBR) Individual Review Techniques
- 28. • This is the most common approach (completely unstructured) • Each reviewer is expected to find as many defects as possible of any type • Little or no guidance on how to review is provided • This approach is highly-dependent on reviewer skills • Often the same issues are identified by different reviewers Ad Hoc Individual Review “60% of reviewers don’t prepare at all”
- 29. Individual Reviews - Checklist-based Reviewing Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting
- 30. • A systematic approach to identifying defects • Typically review checklists take the form of a set of questions based on potential defects (and risk) • Assign different reviewers to different checklists to – get wider coverage – prevent the duplication inherent in the ad hoc approach • There is a danger that some reviewers limit themselves to only considering the checklist entries • Will normally take longer than using an ad hoc approach – often multiple passes are needed to cover all questions (especially if checklist is long) – Requires a systematic approach Checklist-based Review “50% OF REVIEWERS USE CHECKLISTS”
- 31. • Checklist defects are derived from experience – within the project – within the organization – across the industry as a whole • Checklists should be specific to – the product under review (be wary of generic questions) – the methodology used to develop the work product (e.g. there may be different checklist questions for requirements in the form of plain text to those in the form of use cases or user stories) – the application domain of the work product (e. g. a checklist for a banking work product may be based on banking regulations while a checklist for an avionics work product would be based on avionics standards) • Checklists should be based on risk – to ensure that the most important (and the most frequently- occurring) risks are reviewed in more depth Creating Checklists
- 32. • Many checklists are too long and never change • The ideal checklist should be constrained to about 10 entries and regularly updated – as entries become stale and find fewer issues (hopefully because the authors have learned and improved) – new entries should reflect issues missed in the recent past Managing Checklists
- 33. Individual Reviews – Scenario-based Reviewing Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting
- 34. • Used where multiple scenarios can be identified for different users • Reviewers perform ‘dry runs’ on the work product to check functionality and handling of error conditions – following scenarios rather than reading the document from top to bottom • Assign different reviewers to different scenarios to: – get wider coverage – prevent the duplication inherent in the ad hoc approach • There is a danger that some reviewers limit themselves to only considering the provided scenarios – and will miss defects of omission, where required functionality is not included in the work product under review • Research indicates that scenario-based reviews – can be more cost effective than checklist-based reviews – but they do take longer Scenario-based Reviewing
- 35. • Where requirements, designs (or tests) are documented in a scenario- type format (e.g. use cases) then the existing specifications are a good starting point – the review can then use the modelled scenarios to drive the review • However, scenarios typically also need to be created (we need multiple scenarios to cover everything): – overview scenario – most common scenarios – alternative scenarios – error condition scenarios – growth scenarios (e.g. for an architecture review) – input handling scenarios – output generating scenario – enquiry scenarios (getting specific results) • Scenarios should be based on risk – to ensure that the most important (and the most frequently-used) scenarios are reviewed in more depth Identifying Scenarios
- 36. Individual Reviews - Perspective-based Reading Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting
- 37. Perspectives and Roles PRODUCT UNDER REVIEW View 2 View 5 View 7
- 38. Perspective-based Reading (PBR) Identify Stakeholders Create/Identify PBR Scenarios Perform ReviewPerform ReviewPerform ReviewPerform Review • Understand Stakeholder Perspective • Attempt to Create a Work Product • Use Checklist on the Work Product Describes the stakeholder role the reviewer is taking for this review – and their interest in the work product under review Describes the high level work product that the stakeholder would be expected to develop A checklist of questions specific to the high-level work product developed
- 40. • Milestone Reviews • Inspections • Technical Reviews • Peer Reviews • Walkthroughs • Informal Reviews • Author Check • Buddy Check • Pair Review • Peer Deskcheck Work Product Reviews – Types or Bespoke Planning & Preparation Overview Meeting Individual Review Issue Collation & Analysis Fixing & Reporting Chosen Attributes OR
- 41. • Purpose • Roles • Individual review techniques • Optional activities • Number of reviewers • Planned number of reviews • Work product type • Work product format • Formal reporting • Training required • Review improvement • Entry and exit criteria • Geographic distribution of reviewers Work Product Reviews - Attributes
- 42. • Factors – purpose – product type – product format – risks – reviewer availability – reviewer skills (and availability of training) – life cycle model – budget (time and cost) Choosing your Review
- 43. • Foreword • Introduction • 1 Scope • 2 Conformance • 3 Normative References • 4 Terms and Definitions • 5 Work Product Reviews - Introduction • 6 Software Review Process • 7 Review Techniques • Annex A (informative) Review Documentation • Annex B (informative) Review Roles • Annex C (informative) Review Attributes • Annex D (informative) Review Types (and attribute mapping) • Annex E (informative) Mapping to IEEE 1028 • Annex F (informative) Reviews - Life Cycle Mapping • Annex G (informative) Review Measurement & Improvement • Annex H (informative) Tool Support • Annex I (informative) Bibliography ISO/IEC 20246 - Work Product Reviews - Contents
- 44. ISO/IEC 20246 Development Working Draft (WD) Committee Draft (CD) Draft International Standard (DIS) Final Draft International Standard (FDIS) Final International Standard (FIS) May 2014 May 2015 May 2016 WD1 DIS FDIS WD2 DIS2
- 45. ISO 29119/30363/20246 – Current Status Testing Concepts & Vocabulary 29119 Part 1 Testing Techniques 29119 Part 4 Test Documentation 29119 Part 329119 Part 2 Keyword-Driven Testing 29119 Part 5 Process Assessment ISO/IEC 33063 AUG ‘13 AUG ‘13AUG ‘13@ FDIS @ DIS2 @ FDIS Work Product Reviews ISO/IEC 20246 DIS Test Processes
- 46. • Context - ISO/IEC 20246 & the ISO 29119 standards • The generic review process • Individual review techniques • Review types vs review attributes • Current status • Question? Summary
- 47. Thank you for listening Any Questions?
- 48. • Join ISO Working Group 26 – representing your national standards body – 6 day meetings, every 6 months – contribute between meetings • Join a WG26 mirror group – for your national standards body • Contribute materials • Review drafts • Trial the standards on real projects Do you want to be involved?
- 49. • stuart@sta.co.kr – if you have any questions on the standards – if you are interested in trialling the standard on a project, reviewing drafts or writing examples • http://softwaretestingstandard.org/ – WG26 website • http://www.jtc1-sc7.org/ – access to official documents released by WG 26 Finally…