Dunning-Krueger Effect in Software Testing
1 like158 views
The document discusses the Dunning-Krueger effect as it relates to software testing. It provides examples of how testers and managers can underestimate their abilities. It emphasizes that testers are responsible for communicating objective information about quality but not for quality itself or go/no-go decisions. Test automation should enrich testing but not replace exploratory testing or testing as an intellectual process. The document cautions against overreliance on automation or viewing testing as a commodity. It stresses the importance of testers having a variety of skills beyond just automation.
Dunning-Krueger Effect in Software Testing
- 1. Dunning-Krueger Effect and Software Testing Alexander Alexandrov Luxoft, Moscow, Russia Mikhail Pavlov Striven Consulting, Toronto, Canada 1
- 2. Content • Dunning-Krueger Effect manifests in software testing • Specific examples from authors’ experience • Test manager / test lead and testing team responsibilities • Project stakeholders impact on software testing process • Testing as intellectual activity and test automation • Negative trends in modern software testing 2
- 3. Dunning- Krueger Effect: Short Definition • Scientifically proved in 1999 • Has ancient roots • Consists of two opposite observations • People with low abilities at a task overestimate their ability. It is related to the cognitive bias of illusory superiority and comes from the inability of people to recognize their lack of ability. • On the contrary, highly competent people tend to underestimate their ability. They suffer from insufficient self-confidence and consider others as more competent. 3
- 4. Dunning- Krueger Effect in One Phrase 4
- 5. Dunning- Krueger Effect in One Phrase • Another definition of Dunning-Krueger Effect Item 2 is imposter syndrome • Imposter Syndrome is fear that a person’s knowledge and skills are not sufficient to carry out their duties 5
- 6. Testing Team Responsibilities • Is not responsible for product quality • Does not create quality, only verifies it • Cannot formally influence quality improvement • Does not take go/no go decision regarding product delivery • Does not have authority • Does not have information regarding: • Contract details • Informal arrangements between customer and contractor stakeholders • Is not paid for it • Responsible for collecting objective, timely, comprehensive and accurate information regarding product quality and communicating it to project stakeholders 6
- 7. Test Lead Responsibilities • Highly skilled resource • Adequate understanding of the role of testing in a project • Can testing itself improve product quality? • Should a test lead formally sign off the release/project? • Influencing stakeholders having insufficient competence in testing • Why? • How? 7
- 8. Example 1: Taking Go/No Go Decision • There are 2 critical and 22 serious open defects in the bug tracker today. The release delivery deadline is scheduled tomorrow. • The Project Manager offers the the test lead to take go/no go decision for the release tomorrow • The contract contains penalties of $200K for late delivery and $1K for each missed defect (the contract is not available to the test manager) • What should the test lead decide? 8
- 9. Example 1: Taking Go/No Go Decision • Share detailed information regarding product quality and quality risks for the release with the project managers • Offer project manager to take go/no go decision using information they possess • If the project manager insists on delegating go/no go decision to the test lead, remind gently that test lead is not paid for taking such decisions 9
- 10. Influencing Stakeholders – Role of Testing • Project managers often do not fully understand the role of testing in a project: • Testers are responsible for product quality • If the released release turned out to be of poor quality, then the testing group is fully responsible for this. • And if it’s high-quality, then they forget about testers • The test manager is required to sign off the product release (thereby assuming responsibility for quality) • What should a test lead do? 10
- 11. Influencing Project Stakeholders – Role of Testing • What can test lead do? • Ideally find common grounds with PM before project kick off • Accept together with PM that zero defects in production is a myth (or at least this event has very low probability) • Align expectations regarding the test team results including quantitative indicators based on historical data and the specifics of the project • Do not afraid to say No, do not accepts commitments which cannot be obviously achieved • Define test team responsibilities in one of key project documents (project management plan, test strategy, test approach, etc.) • Do not bend over for management, so that later you won’t feel extreme pain • If it does not help and issues arise • Analyze mistakes made before • Propose and implement a corrective action plan • Finally do what should have been done before the project kick off 11
- 12. Influencing Project Stakeholders – Test Automation • Some project managers and their bosses have the following idea of ’ideal testing’ • Written test cases cover ALL requirements / code • Completed (especially automatic) test cases guarantee detection of ALL defects • Regression (auto) testing ensures the integrity and quality of the product after the changes • Phrase “automated testing” became a kind of drug for such project managers 12
- 13. Testing is a challenging intellectual process • To test is to seek the true status of a product, which in complex products is hidden from casual view • A tester, working with limited resources, must sniff out trouble before it’s too late • Testers engage in sensemaking, critical thinking, and experimentation, none of which can be done by mechanical means • Taken from Michal Bolton, James Bach “A Context-Driven Approach to Automation in Testing” 13
- 14. What is realistic objective of writing automated tests? • Automation enriches testing • Allows to execute test cases of certain types more often and faster and, hence, uncover defects earlier and in more efficient way • Allows to expand testing in new areas (e.g., performance testing) • At the same time test automation • Does not exist outside of established testing process • Cannot replace exploratory testing • Inefficient without relevant test design • Is accompanied with the detailed analysis of test execution results • Can be less efficient than its rejection 14
- 15. Michael Bolton and James Bach about test automation “Since the testing humans actually do cannot be put into words, it cannot be encoded and therefore cannot be automated”1. 1 Michal Bolton, James Bach “A Context-Driven Approach to Automation in Testing” 15
- 16. Management Wet Dream – Big Red Button • Fire all testers • Hire a test automation engineer • Assign them a task to automate full test coverage • Fire the test automation engineer as well • Push the Big Red Button every night • Get test automation logs with complete list of defects every morning 16
- 17. Who should write automated test scripts? • Everyone? • ”Test automation is not more than a tool; every skilled tester should be able to automate test cases. Test automation should be applied for its intended purpose and needs. A tester should be unique, should be able to cook porridge if necessary, in my humble opinion” • “Anyone can write an automated test script while not everyone can figure out what is the test about and how to write it” • Those who are interested in it? • Testers who can code? • Coders who can test? • Professional experienced software developers? 17
- 18. Who should write automated test scripts? • Automated test cases coding is a new code development activity • Create/maintain/store automated test code tasks should be treated like create/maintain/store product code tasks • Test specifications selected for automation (test design) is a definition for test automation code • Test automation code should be stored in a project version control system (code repository) together with the product code • Test automation code should be written by professional programmers • Test code development and maintenance can be more complex than the code of the system-under-test • Test analyst role in test automation includes but is not limited to: select and prepare candidates for automation, product knowledge transfer, evaluation and analysis of test execution results 18
- 19. Test Automation Tool Fetish • Having specific tool knowledge and skills is a positive factor, however it is not enough • Knowledge and skills required: • Application domain and project • Testing techniques and technologies • Defect management • Exploratory testing • etc. • Can a single test automation engineer have all these skills? • It is unrealistic (unfortunately, Leonardo Da Vinci is not with us anymore) • Can other resources (testers) have these skills? • Absolutely (there are many examples of companies and projects) 19
- 20. Which testing is needed? • As fulfillment of assigned / expected responsibilities (see above) • As intellectual activity • testing is evaluating a product by learning about it through exploration and experimentation1 • As economical activity • As basis for automation 1 Michal Bolton, James Bach “A Context-Driven Approach to Automation in Testing” 20
- 21. Job Market and Test Automation Trend • Unfortunately, career focus of modern tester is mostly aimed on programming and various test automation tools skills development • As the result, modern testers know how to write automated test scripts and some of them can do it while most of them cannot test. Specifically: • They have gaps in test design techniques understanding • Often do not know basic testing techniques (equivalence partitioning, boundary values) • Cannot analyse test execution results and make conclusions, generalize or identify specific cases • Do not have process culture 21
- 22. Conclusions • Test team should be responsible for communicating information about product quality to project stakeholders. Test team is not responsible for the product quality • Test automation is one of testing techniques which success depends on many factors • A concept about testing as a commodity popular among some development managers and their bosses' poses a threat to software testing evolution and improvement, testers’ training and career development • The informal influence a test manager has on stakeholders and their perseverance while negotiating role of testing are critical for the the project success and testing risks mitigation 22