Verification audit
3 likes4,452 views
The document outlines questions for a test team to audit themselves to identify areas for improvement. It covers topics such as vision/mission, organization/process, tools/automation, risk/scope, methods, KPIs/metrics, and priorities. The purpose is for the team to rate their compliance with each best practice and note comments, not to evaluate teams against each other. Identifying gaps can help test teams enhance their effectiveness and efficiency.
Verification audit
- 1. Verification Audit Helping Test Teams to Improve
- 2. Purpose • This document outlines questions suitable for an audit on a test team • By asking themselves these questions the team can understand what areas are in need of improvement • The purpose of the audit is to support the test team, not to evaluate and compare to other test teams
- 3. Audit Overview Vision & Mission & Mindset Organization & Process & Stakeholders Tools & Environment & Automation Risk & Scope & Methods KPI & Metrics & Priorities
- 4. Vision & Mission Does the test team have a vision and a mission that are applicable to them? What is the test team trying to achieve and how are they trying to achieve it? • Most likely the organization the test team is a part of has a vision and a mission of some kind – these should be known by the test team, and broken down to a suitable level that is useful to the team • What are the overall priorities? Time-to-market? Quality? • What are the teams goals? Finding all relevant defects? Providing accurate decision material? Reducing risk?
- 5. Mindset What is a tester’s role in the organization? • Quality Police? Development Support? Testing-as-a-service? • A test team must know what there role is • A test team must understand the difference between trying to find all defects in a piece of software, and helping the developer create better quality code • Depending on what the tester’s role is in the organization, the job of the test team differs and their focus must be different
- 6. Organization & Process How is the test organization set up at the company? What does the overall test process look like? Who tests what and when? How is the test process connected to the development process? • It is important for every test team to understand how they fit in to the bigger picture without this understanding the team can easily overlap other teams, and miss gaps in their test effort • It is also important that the test team knows who they should communicate with in other parts of the company • Knowing how the team’s activities fit in to the overall testing process is also critical to be able to work efficiently and effectively • How does the test process and the development process interact? This includes requirement work, stand-up meetings and similar activities
- 7. Stakeholders Who are the test team’s stakeholders and what is it that the test team delivers to these stakeholders? • Who receives the test reports of the test team? • What are the needs of these stakeholders? • Are other test teams dependant on the test report to set their test scopes? • What decisions are based on the test reports? • Is the test report good enough? Could it be improved? • How can the test team support stakeholders in other ways than test reports? • How can the test team support developers?
- 8. Tool & Environment What specific test tools and test environment s are needed for the test team to perform their testing? How are these tools optimized for the job? If the team could change or improve some tool, would they? • Test tools and environments should empower the tester to become more efficient and effective – is this the case in the test team? • How could the tools be improved? • Are there any tools missing today that would improve efficiency? • How are the tools version handled and maintained? • Tool documentation?
- 9. Automation What is the maturity of test automation within the team? Why does the automated scope look like it does? • Are there any limitations to automation? Tools? Difficulties in automating certain test cases? • How did the team come up with what test scope to automated? Did the team start by creating manual test cases and then try to automated these? Or did they start by looking at what is possible to automated and build the scope based on that? • How is competence in the team for test automation? • How is testability a part of the test teams work?
- 10. Risk & Scope How does the test team perform risk analysis? How does the test team set their scope for different test activities? • What inputs does the test team user for their risk analysis? • How much is gut feeling when it comes to risk analysis? • Is there extensive historical data to use? • Are there any tools available to facilitate risk analysis? • How is scope setting influenced by what other teams test? • How do you take into account testing different versions, customizations and variants of the software? • Are there any guidelines available to support in the scope setting and risk analysis • How are test plans created?
- 11. Methods How does the test team perform their testing? Scripted tests or exploratory testing? Formal methods or gut feeling when creating tests? • Does the team perform manual scripted testing? When and why? • Does the team perform exploratory testing? When and why? • Does the team use any quality model/heuristic model when creating tests? • Does the team use formal test techniques when creating tests? • Are there any guidelines available for creating tests?
- 12. KPI & Metrics Which KPI & Metrics are relevant to the test team? How is the test team measured by management? Do these metrics drive good or bad behaviour? • How is the test team performance measured? • Are these measurements meaningful? • What behaviour do these measurements encourage? • How can the teams performance be measured in a meaningful way without promoting bad behaviour?
- 13. Priorities When the workload is high, how does the test team prioritize between activities, and how do they prioritize during scope setting? When the workload is low, how is the extra time spent? • Is it clear for the team how they should prioritize their work? • How is this prioritization done? • What does the test team do in periods of low activity? Improvement projects? Additional testing in the projects they are involved in?
- 14. Results • Rate the answers of each question using the following system Compliant Partly Compliant Not Compliant • And write a short comment for each question to summarize the answer
- 15. Summary • These questions should raise thoughts and ideas about how the test team can improve, both within the team, and from the auditor • The auditor must be an experienced tester to ask the right follow-up questions to the team in each question • These results should not be used to compare teams to each other, but instead to find potential improvement areas – both from the team itself and from the auditor • If all test teams in the organization can answer all these questions satisfactory then the state of verification in the company can be considered quite good