Tools for Managing Quality (Nov 2010 BenchmarkQA Software Quality Forum)
2 likes1,034 views
The document compares Microsoft's emerging toolset, particularly Team Foundation Server (TFS) and Visual Studio 2010, against Quality Center in various aspects of software quality management, including project management, test case management, test execution, defect management, and reporting. It concludes that while TFS excels in features like project management and automated testing, Quality Center holds advantages in areas like manual test execution and reporting. Ultimately, the document positions Microsoft's tools as strong contenders in the quality management landscape despite some limitations.
Tools for Managing Quality (Nov 2010 BenchmarkQA Software Quality Forum)
- 1. Tools for Managing Quality How Microsoft s Emerging Tool Set Stacks Up Against Quality Center Software Quality Forum Thursday, November 11, 2010 5:00 7:00 pm
- 2. BenchmarkQA helps project teams deliver higher quality software through: Quality Assurance Consulting Contract & Permanent Staffing QA/Test Training & Seminars Local Outsourcing Test Lab © 2010 BenchmarkQA
- 3. Agenda Deciphering the MS Marketing Comparisons 1. Project/requirements management 2. Test case management 3. Test execution 4. Defect Management 5. Reporting 6. Administration 7. Automated Testing Conclusions 11/15/2010 © 2010 BenchmarkQA Slide 3
- 4. Overview of MS Tools 4 primary components This is primarily a developer view of the world 11/15/2010 © 2010 BenchmarkQA Slide 4 Reporting Services
- 5. Deciphering the MS Marketing TFS Database (think Access application) Visual Studio Developer tool (unit testing) Automated testing (Coded UI) MTM Microsoft Test Manager Reporting Reporting Services 11/15/2010 © 2010 BenchmarkQA Slide 5
- 6. Team Foundation Server (TFS) Database repository Views from Visual Studio, Reporting Services, IE and MTM Query tools 11/15/2010 © 2010 BenchmarkQA Slide 6
- 7. Visual Studio Developer tool Unit testing Coded UI testing Source code management 11/15/2010 © 2010 BenchmarkQA Slide 7
- 8. Microsoft Test Manager (MTM) Test case management Test execution Integrated defect management 11/15/2010 © 2010 BenchmarkQA Slide 8
- 9. Reporting Reporting Services Primarily warehouse based Excel 11/15/2010 © 2010 BenchmarkQA Slide 9
- 10. © 2010 BenchmarkQA Comparison 1 PROJECT/ REQUIREMENTS MANAGEMENT
- 11. TFS Project/Requirements Management User Stories (Validation (UAT) tests linked here) Tasks (Verification tests linked here) 11/15/2010 © 2010 BenchmarkQA Slide 11
- 12. TFS Requirements User Stories High level More detailed user stories linked to higher levels States Active/Resolved/Closed Area Path Modules or groupings of application functionality Can be used to sort of create a tree structure of user stories in a report Iteration Can be used to manage releases 11/15/2010 © 2010 BenchmarkQA Slide 12
- 13. Task Form 11/15/2010 © 2010 BenchmarkQA Slide 13
- 14. Tasks Tasks (linked to user stories) Assigned to person Estimate of effort Actual hours Activity Design (Verification tests linked here) Development Testing Task States Active Closed 11/15/2010 © 2010 BenchmarkQA Slide 14
- 15. Requirement-Design Links Use task activities to deal with design elements 11/15/2010 © 2010 BenchmarkQA Slide 15
- 16. Scoring - Project Management Scheduling TFS Resource management TFS QC Can be added to requirements section Workflow would be challenging Additional reporting needed (burn down charts, etc.) 11/15/2010 © 2010 BenchmarkQA Slide 16
- 17. Scoring Requirement Management Enter requirements Wash Organize requirements QC Link requirements to test cases Wash Link requirements to defects Wash 11/15/2010 © 2010 BenchmarkQA Slide 17
- 18. Comparison 2 TEST CASE MANAGEMENT © 2010 BenchmarkQA
- 19. TFS Manual Test Cases Similar to QC Action/step Expected results Parameterized test fields Defined at test level rather than run-time Can import from Excel Test case importer (CodePlex) Needs customization Organization similar to Test Suite in QC 11/15/2010 © 2010 BenchmarkQA Slide 19
- 20. MTM Test Case Management 11/15/2010 © 2010 BenchmarkQA Slide 20
- 21. Test Case Entry Test case entry very similar to QC Action Result Parameters Shared steps (call another test in QC) 11/15/2010 Slide 21© 2010 BenchmarkQA
- 22. Scoring - Test Case Management Enter test cases QC Organize test cases QC Import from Excel QC Link test cases to requirements Wash Link test cases to defects Wash Integration of automated tests QC Unit tests - VS 11/15/2010 © 2010 BenchmarkQA Slide 22
- 23. Comparison 3 TEST EXECUTION © 2010 BenchmarkQA
- 24. MTM Test Execution Management Note configurations Progress bar 11/15/2010 © 2010 BenchmarkQA Slide 24
- 25. TFS Test Execution Very similar to QC Filter by environment Cannot stop and restart test Recorded as fail. Rearranges your desktop History not linked to test case List of tests to run 11/15/2010 © 2010 BenchmarkQA Slide 25
- 26. Scoring - Test Execution (Manual) Organize test runs Wash Execution speed Wash Restart test QC Environment support MTM Execution history - QC 11/15/2010 © 2010 BenchmarkQA Slide 26
- 27. Comparison 4 DEFECT MANAGEMENT © 2010 BenchmarkQA
- 28. TFS Defect Management Remember this is a developer view of the world Active/Resolved/Closed Changing workflow is not as easy as QC Option to verify defect Major issue-Defect verification is recorded in test suite defect was found rather than test suite it was fixed in. Only reruns if defect created from test run 11/15/2010 © 2010 BenchmarkQA Slide 28
- 29. Scoring - Defect Management Defect entry Wash Creating defect while running test Wash Managing defects QC Verifying defects - Wash 11/15/2010 © 2010 BenchmarkQA Slide 29
- 30. Comparison 5 REPORTING © 2010 BenchmarkQA
- 31. TFS Reporting Uses a warehouse so data is old Reporting services Much higher skill level and permissions required to create reports Good for roll-up/summary reporting Difficult to nearly impossible to generate accurate requirements coverage report 11/15/2010 © 2010 BenchmarkQA Slide 31
- 32. Scoring - Reporting No contest - QC 11/15/2010 © 2010 BenchmarkQA Slide 32
- 33. Comparison 6 ADMINISTRATION © 2010 BenchmarkQA
- 34. Scoring - Administration Project customization QC User administration Wash Defect workflow QC Other workflow Not easy in either 11/15/2010 © 2010 BenchmarkQA Slide 34
- 35. Comparison 7 AUTOMATED TESTING © 2010 BenchmarkQA
- 36. Unit Testing Integrated part of VS Easy for developer to create tests for methods that return values MOQ for visual unit test Code coverage 11/15/2010 © 2010 BenchmarkQA Slide 36
- 37. Coded UI Testing Capture/replay Manually create object repository Higher skill level than QTP to customize tests Much more flexible than QTP No waiting for support 11/15/2010 © 2010 BenchmarkQA Slide 37
- 38. Case Study 1 QTP 9 Crashed QTP on login screen QTP 10 QTP hung on login screen QTP 11 QTP recorded, could not replay login screen (some customization would get past this issue) VS 2010 Records and plays back for simple actions. Customization for dealing with grid actions 11/15/2010 © 2010 BenchmarkQA Slide 38
- 39. Case Study 2 QTP 10 Cannot find any objects on screen, not even the menus QTP 11 Can find most standard controls Cannot see any 3D objects VS2010 Sees all objects View object properties (center, radius, etc.) 11/15/2010 © 2010 BenchmarkQA Slide 39
- 40. Scoring - Automated Testing No contest Visual Studio 11/15/2010 © 2010 BenchmarkQA Slide 40
- 41. Summary Project management TFS Requirements QC (by a hair) Manual test case management QC Manual test execution QC (by a hair) Defects Toss up Reporting QC Administration QC Automated testing VS2010 11/15/2010 © 2010 BenchmarkQA Slide 41
- 42. Considerations Cost Initial (Project, QC, VS, QTP) Maintenance Conversion Customization Productivity improvements Skill level of testing staff Best of Breed vs. One Solution If starting from scratch (i.e. want to get off Excel) 11/15/2010 © 2010 BenchmarkQA Slide 42
- 43. And the winner is? Microsoft s Team Foundation Server/Visual Studio 2010 11/15/2010 © 2010 BenchmarkQA Slide 43
- 44. How can we help? Project engagements Tool selection and implementation As needed support Staffing Senior automation resources Quality Path Tools and Technology Broad and deep assessment Process improvement program 11/15/2010 Slide 44© 2010 BenchmarkQA
- 45. Thank You! For additional information on how BenchmarkQA can be of assistance, please contact: Warren McLeod 952.392.2400 warren.mcleod@benchmarkqa.com © 2010 BenchmarkQA