Bug best practice
- 1. BUG BEST PRACTICE LIANG GAO (LIANGG@GMAIL.COM)
- 2. AGENDA Bug filing best practice You should learn…. Value added for bug management Defect management Bug follow up best practice
- 3. BUG IS IMPORTANT Your performance is partially judged by the quality and amount of the bug you filed. As an engineer, reputation is the most important thing. A tester’s reputation is as good as the quality of your bug can get. Most of the time, it is the only way you communicate with the developer. Who will read your bug report? – developer, your managers, developer’s manager, product manager, marketing folks, customers……
- 4. WHERE ARE BUGS COME FROM? Laziness (does not perform error handling) Inexperience (does not know the impact of code changes) Carelessness (blind cut-n-paste) Bad design/infrastructure (bad module design un- maintainable code) Incomplete documentation (misunderstood customer requirements, missing information in func spec) Complexity of software (hard to maintain/enhance/test) Bad tools (compiler, CM tool, merge tool, etc) Lack unit-level testing (misses lots of basic bugs) Time pressure (fatigue) Laziness and carelessness Code scales linearly, the complexity scales exponentially
- 5. THE QUALITY OF A TESTERScript Quality Test Case Quality Bug Quality Case Execution Quality
- 6. HOW TO FIND A BUG? You think it is a bug because? • Understand the product/function spec correctly • Understand the use case scenarios correctly • Put yourself in a user’s perspective. • Your experience counts a lot, 上 兵纸 谈 is harmful Bad bugs cause you trouble • Junk bug is counter productivity, and ruin your reputation • Bad documented bugs cause you a lot of time and energy, like…. living in a nightmare.
- 7. HOW TO FIND A BUG? Debugging is like watching Magic Tricks (Michael Amma17:40 - 19:00) James Brown 5:10
- 8. WHY I CAN FIND HIGH QUALITY BUGS BUT YOU CAN’T? High quality bug • Affect the major functionality QoS still work after delete the configuration • Not Cosmetic Bug 3744 - WebUI 的导航栏没有滚动条 , 不便于用户浏览操 作 Because I understand the functionality deeper than you Because I use more testing types than you. Because I design more use case than you Because I carefully execute the cases than you Because I pay more attention to the details than you. Because once I found a crack, I’ll keep on going, but you didn’t
- 9. NONREPRODUCIBLE BUGS – HEADACHE? Random is not equal to irreproducible
- 10. NONREPRODUCIBLE BUGS – HEADACHE? Always report non-reproducible bugs, they might be time bombs. Non-reproducible bugs are always reproducible. • Timing and delay? • Only happen during installation • Depend on some data like corrupted database? • Sequence of the execution? • Interaction with other programs in the background?
- 11. NONREPRODUCIBLE BUGS – ELABORATE a. I saw this one, but was never able to make it happen again. b. I see this when I run with this setup, but sometimes I don't. c. This is the same anomaly I see under several setups, but not all the time. d. Under X setup, this happens sometimes and not other times. There may be something different from one X to the other, but I' not seeing it. What is the impact if this happens all the time?....
- 12. NONREPRODUCIBLE BUGS – DEBUGGING STORY ²»¿É¸´ÏÖBug ʵս ר¼Ò̸֮ ר¼Ò̸֮ - 2
- 13. AVOID CRITICAL BUG IN THE LATE DEVELOPMENT Execute test cases based on priorities Discover critical bug in the early development stage • It gets fixed early, less resource waste • It gives developer more lead time – critical bugs are hard to fix
- 14. MOST IMPORTANT THINGS ON A BUG Subject, Subject, Subject • One line • Exactly pinpoint the problem. • Accurately describe the problem Bug 3747 - WebUI 界面上的字体、图标等在 Linux 系统下的显示不 正常 Bug 3774 - 客户端能够通过 Admin 端口与 DUT 建立 PPTP 连接 , 不符 合设计规范 Bug 3768 - admin route: 不能为同一个服务创建多条 admin 路由 Bug 3788 - 在配置 DUT 的时候， DUT 死机 Which one is better?
- 15. WHAT IS THE BUG QUALITY Reproducible • Testers filed bug need to be reproducible in developer’s environment. • Customer found bugs sometime is hard to reproduce, but that not tester’s concern. • Test case is not the step to reproduce. • Need to find the main cause by ruling our the unrelated factors. Need to pinpoint the root cause (Strong Analytical skills). • 蛛 迹只是 象，需要 一 追踪，找到 的本丝马 现 进 步 问题 质。 ISIC crash the box, can you find out which packet, and why?
- 16. HOW TO WRITE A REPRODUCIBLE BUG REPORT A short paragraph for detail description. A topology file with detail annotation. (IP address, DUT type etc) Step to reproduce • Step 1. execution step with detail output. • Step 2… execution step with detail output • Seep 3: where the problem is. What is the expected output, what is the actual output. Detail DUT configuration dump, Anything else (document) you think can claim the validation of your bug? Review each others bug reports
- 17. AGENDA Bug filing best practice Bug follow up best practice Defect management Integration of bug tracking system with other engineering tools
- 18. UNDERSTAND BUG STATE FIRST! N: A: I: Information needed: Take action ASAP R: Need Tester’s verify. Take action ASAP V: Monitor your bug daily: usually you will get an email reminder whenever a bug state is changed. Pay special attention to “I” and “R” Best practice: I and R state bug need to be updated within 24 hours. Ask your team lead for help, monitoring bug every day is hard.
- 19. INTERACT WITH DEVELOPERS -1 Understand the developer’s response. It is like writing an email, need to be polite. Need to have start paragraph and end paragraph. It needs to be thorough and accurate (provide as many useful info as possible). A quick and good response help you gain the respect from developer. Ask your team lead for help is a good idea.
- 20. INTERACT WITH DEVELOPERS -2 Meet the programmers who will read your report The best approach may be to demonstrate your bugs to the programmers. When the programmer say it is fixed, make sure it isn’t still broken Verify bug fixed promptly When fixes fail, talk with the programmer. Bug report should be closed by testers..
- 21. AN EXAMPLE OF QUERY State-Changed-From-To: open->info State-Changed-By: murphyw State-Changed-When: Sun, 24 Jun 2007 23:14:08 -0700 State-Changed-Why: 1. ‘set policy id 1 attack "CS:ftp" action close’, does this CLI mean that we should drop FTP traffic? 2. Please help to provide debug information on successful case. 3. Please turn on snoop and snoop detail during debugging. Thanks! How Would you response?
- 22. AGENDA Bug filing best practice Bug follow up best practice Defect management Integration of bug tracking system with other engineering tools
- 23. DEFECT MANAGEMENT How to know how many bugs filed per person, per day – bug search is your best friend. How to know how many bugs filed for certain type of testing • Search for testing type – regression; performance; functionality;… • Add key word in front of the bug subject field then search the keyword REG: extended access list can not block traffic. Pre-Test:-Pascal-when the packets can't pass through VPN tunnel with deep inspect • Be careful when add the interest field: those are the person will be notified every time this bug state is changed. • If there is a field called attribute, use it. It helps to categorize and manage bugs we filed.
- 24. FILING BUG REPORTS File every bug found (except duplicate bugs), annotate if a bug: • Blocker: High priority bug preventing further testing • Regression: Bugs that introduce in new build • Unreproducable: Bugs that cannot be reproduced • Customer found: Bugs found by customers (Beta/FCS) Pick right Priority (usage) & Severity (impact) • P3/S1 : Login 200 times with 200 characters user name will crash the device with exception core dump • P1/S3 : GUI splash screen misspelled company name Bug report is a statistics tool, use to predict readiness of software and future releases
- 27. BUG TREND CONVERGENCE Bug Fixed Release Date Bug Filed
- 28. BUG TREND CONVERGENCE 发现 Bug 正常测试 分散测试 发布日期 Bug 修复 Bug 严重程度 日期 Bug 数目
- 29. BUG TREND CONVERGENCE 发现 Bug 正常测试 分散测试 发布日期 Bug 修复 Bug 严重程度 日期 Bug 数目
- 30. BUG TREND CONVERGENCE 发现 Bug 正常测试 分散测试 发布日期 Bug 修复 Bug 严重程度 日期 Bug 数目
- 31. USE BUG TREND TO FIND QUALITY PROBLEM
- 32. USE BUG TREND TO FIND QUALITY PROBLEM
- 33. BUG FIELDS FOR METRICS Status Date Reported Who Found Assigned To Program Area Severity Priority Resolution
- 34. INTERESTING BUG METRICS Priority 1 Find Rate − This is the find rate of bugs that at any point in time become priority 1. It is a retrospective analysis. Open Bug Aging Report − A histogram of how long bugs have been open. This is most useful early in long projects. Deferred Bug Aging Report − A histogram of how long deferred bugs have been waiting on the list. It helps spot chronically deferred bugs. Promotions/Demotions/Deferrals Chart − Three lines that help us see the triage process at work.
- 35. AVOID CONTROL METRICS; EMBRACE INQUIRY METRICS A control metric is any metric that drives decisions. − Any metric you use to control a self-aware system will be used by that system to control YOU. An inquiry metric is any metric that helps you ask the right questions at the right time. − An inquiry metric might look like a control metric. The difference is how you use it and what you infer from it.
- 36. EXAMPLE CONTROL METRICS “The developer who creates the fewest bugs will receive a bonus.” “Testers must average at least three bugs found per day.” “The product may not be released unless we’ve gone at least one week without finding a bug.” “The product may not be released with more than 10 high severity bugs.”
- 37. EXAMPLE INQUIRY METRICS Why do some developers have more bugs reported on their code than others? How might it be good to have more bugs reported? How might it be bad?” “How do find rates differ among testers? What makes them differ?” “What does the bug find rate tell us about the readiness of the product? Could the find rate be falling because testing is slack, and not because the product is good?” “What are the high severity bugs? Should we fix them?”
- 38. INQUIRY METRICS INVITE LEARNING Observe: You try to see what's happening. Inquire: You conjecture about the meaning and significance of the observations. You collect additional observations to corroborate or refute our conjectures. Model: You form and improve theories about why the system behaves as it does, how you know that it behaves that way, and what you can do about it.
- 39. QUESTIONS BUG METRICS MIGHT HELP WITH TEST PROCESS Productivity • − Are the testers working at full speed? • − Are they speeding up or slowing down? • − Is anything blocking their work? • − Are the testers burning out or staying sharp? • − Are testers and developers getting along together? Status • − How close is the test effort to reaching a point of diminishing returns? • − What kinds of problems are being found? • − Is the bug tracking system being used properly? Focus • − Are the testers focused on the areas of greatest risk or need? • − Could the testers use some outside help?
- 40. QUESTIONS BUG METRICS MIGHT HELP WITHQUALITY IMPROVEMENT Productivity • − Are the developers working at full speed? • − Are they speeding up or slowing down? • − Is anything blocking their work? Status • − Is the product improving? • − How close is the product to being good enough? • − What must happen in order to meet the schedule? Focus • − Is the work distributed effectively among the developers? • − Do any areas of the product need more attention to improve more quickly? • − Is the triage process working well?
- 41. DAILY METRICS! Consider producing a bug metric summary: • − Daily • − Automatic • − Archived • − Web Based This becomes a resource for retrospective analysis. Additionally, daily graphs can help you notice fast breaking dynamics. Include weekends in the graphs.
- 42. EXAMPLE: WHY WAS CONVERGENCE HARDER DURING THE SECOND CYCLE?
- 43. CYCLE 1 CRITICAL BUGS
- 44. CYCLE 2 CRITICAL BUGS
- 45. DIFFERENT SLOPS
- 47. OPEN BUGS BEFORE ENTER CYCLE 2
- 48. LOW PRIORITY BUG WAIT FOR THE FIX
- 49. AGENDA Bug filing best practice Bug follow up best practice Defect management Integration of bug tracking system with other engineering tools
- 50. VALUE ADDED DEFECT/CODE INTEGRATION Bug system can be integrated with CVS, Subversion etc code control tool When check in code, developer can specify the bug id, the bug will be automatically updated with code diff. When check in code, developer can specify the bug id, and bug’s state will be automatically changed to “R”, and trigger tester to verify it. Other scenarios you can think of? This is the area that we can provide the value added for our customer
- 51. THINGS YOU SHOULD REMEMBER AFTER THIS Bug quality means it can be reproduced by developers in their environment. Most important thing on a bug is its subject Junk bug is counter productive, and ruin your reputation. I, R state bug need to be updated ASAP. Use Sigma Bug Template to file bugs.
- 52. BUG REVIEW CHECK LIST - 1 Is the summary short (about 50-70 characters) and descriptive? Can you understand the report? • As you read the description, do you understand what the reporter did? • Can you envision what the program did in response? • Do you understand what the failure was? Is it obvious where to start (what state to bring the program to) to replicate the bug? Is it obvious what files to use (if any)? Is it obvious what you would type? Is the replication sequence provided as a numbered set of steps, which tell you exactly what to do and, when useful, what you will see?
- 53. BUG REVIEW CHECK LIST - 2 Does the report include unnecessary information, personal opinions or anecdotes that seem out of place? Is the tone of the report insulting? Are any words in the report potentially insulting? Does the report seem too long? Too short? Does it seem to have a lot of unnecessary steps? (This is your first impression—you might be mistaken. After all, you haven’t replicated it yet. But does it LOOK like there’s a lot of excess in the report?) Does the report seem overly general (“Insert a file and you will see” – what file? What kind of file? Is there an example, like “Insert a file like blah.foo or blah2.fee”?)
- 54. BUG REVIEW CHECK LIST – 3Can you replicate the bug? Did you need additional information or steps? Did you get lost or wonder whether you had done a step correctly? Would additional feedback (like, “the program will respond like this...”) have helped? Did you have to guess about what to do next? Did you have to change your configuration or environment in any way that wasn’t specified in the report? Did some steps appear unnecessary? Were they unnecessary? Did the description accurately describe the failure? Did the summary accurate describe the failure? Does the description include non-factual information (such as the tester’s guesses about the underlying fault) and if so, does this information seem credible and useful or not?
- 55. BUG REVIEW CHECK LIST – 4Does the description include non-factual information (such as the tester’s guesses about the underlying fault) and if so, does this information seem credible and useful or not? Does the description include statements about why this bug would be important to the customer or to someone else?
- 56. VERSION CONTROL 1st draft, Dec 7th, 2007, Liang Gao 2nd draft, Dec 27th, 2007, Liang Gao