The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing
- 1. W14 Test Techniques 5/4/16 15:00 The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing Presented by: Stephen Burlingame Navicure Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ info@techwell.com -‐ http://www.stareast.techwell.com/
- 2. Stephen Burlingame Navicure Stephen Burlingame’s introduction to software quality assurance occurred during his university senior project where he discovered critical defects in software designed to diagnosis breast cancer in digitized mammograms. Since graduation, Stephen has worked exclusively in the quality assurance field. In 2006, he joined the medical claims clearinghouse Navicure, Inc., where he is director of quality assurance, responsible for overseeing the testing for an application that processes more than one million transactions daily. Living in the Atlanta metro area with his wife, Julia, Stephen has served as president of Atlanta SPIN (Software and Systems Process Improvement Network). Â
- 3. The Canary in the Coal Mine Creating an Early Warning QA Tool to Help Your Testing 1
- 4. Slides written by Stephen Burlingame Director Of Quality Assurance at Navicure Inc., a medical claims clearinghouse offering additional billing and analysis tools for medical professionals. Board member of Atlanta SPIN, a non-profit organization dedicated to promoting software process improvement 2
- 5. Who’s familiar with vCAC from VMWare? 3
- 6. vCAC from VMWare Example Interface 4
- 7. Created CentOS Installation with Custom Software Deployed Now our QA team can request environments on as needed basis Software build is deployed and configured automatically Near “push button” operation 5
- 8. Yea! We’ve created virtual test environments! Our work is completed and now we can all go home! In reality the hard work is not complete You’ve solved some difficult problems but possibly created new challenges 6
- 9. Environment Chaos! Old Setup CERTIMP New Setup CERTIMP Virtual 1 Virtual 2 Virtual N Virtual 3 7
- 10. Managing and keeping track of environments is getting out of hand! QA engineers spend more time figuring out what systems they have and the health those systems than ever before. More systems = more problems 8
- 11. Problem: How Can We Monitor Those Systems and Not Spend More Time in Maintenance than Testing? 9
- 12. Solution: Create a Daily Report This report will highlight system health and basic performance This report will focus on displaying key performance indicators in graphical form that is intuitive This report will alert on known conditions that indicate defects 10
- 14. “QA Health Checker” Data Examples Start Simple! o Disk space alerts o Indicate services are up or down o Look for directories present or missing or when last updated 12
- 15. “QA Health Checker” Data Examples Get More Complicated! o Write SQL queries that look for “bad” data or missing data o Look for errors in logs or indicate that error logs have content o Look for scenarios that indicate system is experiencing issues 13
- 16. “QA Health Checker” Data Examples Let’s Get Crazy! o Create a uniform CSS o Implement basic pictographic principles o Fit this report into retrospective discussions so you improve the metrics captured 14
- 17. 15
- 18. 16
- 19. 17
- 20. A More Technical Look Test systems run nightly cron jobs to gather data and place it on their local hard drive Central reporting machine knows which IP addresses to read from and pulls the stats into a prepared report Report is emailed to a distribution list early every weekday 18
- 21. A More Technical Look Used Python as the scripting language Created a deployment script and documented the release process Documented the report contents and general architecture Used Confluence as the documentation portal 19
- 22. Changing the Conversation With Employees 20
- 23. Old Conversation QA Engineers didn’t always know what was going on or how interpret data. They complained about “phantom issues” and “non-reproducible bugs”. Used vague terms like “slow”, “buggy”, or “weird”. Documentation was simply not enough. 21
- 24. New Conversation QA Engineers know right away the types of issues they are having or about to have in the near future. Team leads are keeping their focus on early signs of issues leading up to release. Management is having pro-active conversation with employees rather than a reactive dialog. 22
- 25. Closing Thought Recommendation: Take a slight risk with this report and experiment Keep it adjusting and evolving! 23
- 26. Time For Questions Contact Information: Email Address – sburlingame@navicure.com If in Atlanta, come to an Atlanta SPIN meeting! 24