Because you can’t fix what you don’t know is broken...
0 likes1,785 views
This document provides an overview of automated error reporting. It discusses how errors are collected from users and analyzed before being sent to developers. Key steps include filtering duplicate and invalid errors, anonymizing sensitive data, and notifying committers through a web interface. Charts show the types of errors reported and projects most affected. The goal is to improve developer awareness of issues and speed up the bug fixing process.
Because you can’t fix what you don’t know is broken...
- 1. Because you can’t fix what you don’t know is broken…
- 2. About me 2 • Initiator of the Automated Error Reporting Initiative • Project Lead of Eclipse Code Recommenders • Eclipse Committer since 2010 • Plug-in Developer for 10+ years (Eclipse 2.1) • Member of the Eclipse Architecture Council • Co-Lead of Java User Group Darmstadt • CEO of Codetrails • Passion to improve developers day-to-day work with intelligent and practical tools - mostly using Data Mining on Big (Software Engineering) Data. @marcelbruch marcelbruch
- 3. Adding a new character to the plot 3 Software User { ——–} Application Developer Product Manager
- 4. Imagine the following situation… 4 Software User { ——–} Application Developer Product Manager
- 5. But when the user starts the app… 5
- 6. 6Can you feel their pain…? This image is not available in your country
- 7. In the logs you’d find… 7
- 8. ✘ ✘ ✘ The tragedy of manual error reports 8 ✘ ✘ ! maintenance ✘ ✘ ✘ Release SR 2 production SR 1
- 9. ✘ ! The tragedy of manual error reports 9 maintenance Release SR 2 production SR 1 ✘ ✘ ✘
- 10. 10Not too happy but feeling okay… This image is not available in your country
- 11. What can we do? 11 Application Developer ? { ——–}
- 12. The Automated Error Reporting Initiative 12
- 13. Live Demo 13
- 14. About server responses, Bugzilla, and immediate feedback „Keep users informed about your progress…“ 14
- 15. Systems Overview 15 AERI Web UI Eclipse BugzillaEclipse IDEs „Direct Reporter Feedback“
- 16. Not yet tracked problems… 16
- 17. Committers needinfo… 17 Status: – – – Resolution: – – – Bug 442500 – ….
- 18. Fixed… 18 Status: – – – Resolution: – – – Bug 442500 – ….
- 19. Won’t Fix… 19 Status: – – – Resolution: – – – Bug 442500 – ….
- 20. Invalid (aka log messages) 20 Status: – – – Resolution: – – – Bug 442500 – ….
- 21. About caching, filtering, anonymization, and other shortcuts… „What goes to eclipse.org?“ 21
- 22. Do we send all errors to Eclipse…? 22 really
- 23. The error reporting funnel 23 ✘ structural filter
- 24. #1: Structural Filtering 24 java.net.UnknownHostException: marketplace.eclipse.org at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) at java.net.Socket.connect(Socket.java:538) at com.example.HttpClient.openServer(HttpClient.java:527) at com.example.http.HttpClient.<init>(HttpClient.java:211) at com.example.HttpURLConnection.connect(HttpURLConnection.java:932) at org.eclipse.recommenders.internal.news.rcp.PollFeedJob.run(PollFeedJob.java:69) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55) Error connecting to ‘http://marketplace.eclipse.org/rss.xml’.
- 25. #2: Did I already sent this? 25 ✘ structural filter personal filter
- 26. #3: Did anyone else sent this? 26 ✘ structural filter personal filter global hist.
- 27. structural filter personal filter If you hit a known issue… 27 ✘ global hist.
- 28. #4: Review 28 review ✘ structural filter personal filter global hist.
- 29. #4: Review 29
- 30. The server side Walkthrough through the committer front-end 30
- 32. The Core Concepts 32 ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘✘ Incidents Problems Projects
- 38. Weekly or daily email digests 38
- 39. Some fun facts e = mc2 —> errors = (more code)2 39
- 40. Disclaimer Don’t draw preliminary conclusions! Please consult your personal statistican…
- 41. How many milestones testers? 450 simrel committers 1200 eclipse committers Testers?~7000 error reporters
- 42. Milestones adoption 42 RC3 RC2 RC1 M7 M6 M5 M4 M3 M2 M1 1000 220 160 650 450 400 560 600 Do new milestones introduce new problems? 70 10 # reporters one week high SR2 SR1 R Luna Mars
- 43. 0 350 700 1050 1400 4.4.0 4.4.1 4.4.2 M1 M2 M3 M4 M5 M6 M7 RC1 RC2 RC3 Milestones adoption 43 Christmas Addition to all EPP packages RCs Addition to first EPP packages
- 44. How hard do they suffer? 44 300 - 3000 reports 25 testers 50-300 reports 330 testers 3-50 reports 3600 testers 1-2 reports 3000 testers
- 45. What do they report? 45 0 750 1500 2250 3000 ‚Normal‘ Errors Out of Memory Freezes Stack Overflows 3rd Party Involved Top 100 of 7000 Me
- 46. What kind of errors are send? 46 3rd Party Involved 16 % Stack Overflows 0 % Freezes 9 % Out Of Memory 1 % 'Normal' Errors 74 %
- 47. What are the most popular exceptions? 47 32 % 12 % 25 % 3 % 8 % 2 % 18 %
- 48. What are the most popular exceptions? 48 Other 32 % No {Class | Method | Field} DefFoundError 12 % NullPointerException 25 % IndexOutOfBoundsException 3 % IllegalArgumentException 8 % ClassCastException 2 % {File} IOException 18 %
- 49. NullPointerExceptions by Projects 49 0 1000 2000 3000 NullPointerException Code Recommenders 65 Statisticians, note: Projects automatically marked as „being responsible“ fixing these NullPointerExceptions but does not mean the NPE happened in their code.
- 50. Tip: Enable null analysis 50 @Override protected void run(@Nullable List<Incident> slice) { for (Incident incident : slice) {
- 51. JDT offers a lot… 51
- 52. Tip: Never return null 52 @Override public Optional<String> getReason() { return Optional.ofNullable(cause); } Use java.util.Optional or com.google.common.base.Optional instead
- 53. Oh Class, Were Art Thou? 53 Use proper import ranges and define uses on your exports to get rid of {Class | Method | Field} DefNotFoundErrors, LinkageErrors, and some ClassCastExceptions. OSGI needs that information to work properly… Export-Package: org.eclipse.recommenders.utils; uses:="org.osgi.framework, org.eclipse.core.runtime, com.google.common.base, com.google.common.collect" Import-Package: com.google.common.annotations;version="[15.0.0,16.0.0)",
- 54. Index out of bounds? 54 a b c d e -1 and array.length do not work. No matter how often you try… There is no tool support for this. Just unit tests and code that tries to fail gracefully.
- 55. Tip: If error messages are not helpful… 55 try { // … } catch (Exception e) { log(„An error occurred“, e); } 21% of all logged errors do not contain stack traces. try { // … } catch (Exception e) { log(„Invalid argument {}“, value, e); }
- 56. Are your users ready for Java 8? 56 both 2,5% Java 1.7 33% Java 1,8 64% Java 1.9 0,3% Which VM do users run the Eclipse IDE with?
- 57. Do people anonymize their messages? 57 Sometimes 3 % No 81 % Yes 16 % Are you concerned that your exception message may reveal sensitive data?
- 58. Bugfixing Rates Speak plain: Does it work? 58
- 59. Bugzilla Summary 59 1170 bugs 275 fixed (24%) 100 invalid 70 not eclipse 467 in progress (40%) 172 duplicates (15%)69 works 4 me 28 wont fix Bugzilla
- 60. Fixed bugs by projects 60 More (35) 84 --- 11 EGit 15 Papyrus 19 Recommenders 41 Xtext 42 Oomph 58
- 61. In progress & fixed bugs 61 0 30 60 90 120 Oomph TMF Recommenders Papyrus EGit --- 97 1914 17 29 20 11 15 19 4142 58 Fixed In Progress
- 62. Future Work? Feature requests are tracked in bugzilla. Make sure your request is in there and vote for it. Client: • Separate view to „review, analyze and send later“ • Making client reusable in other (Eclipse- or SWT-based) applications • Additional client-side automated analysis Server: • Ensure scalability of the system • Improve duplicate detections • Go Neon if feedback is good. Hosting an error reporting service for Java software? • Need one? TTML. 62
- 63. How to reach EPP Logging 63 • Code: Check out the EPP Logging Oomph profile • Wiki: https://wiki.eclipse.org/EPP/Logging • Manual: https://dev.eclipse.org/recommenders/community/confess/#/about • Forum: https://www.eclipse.org/forums/index.php/f/69/ • Bugzilla: https://bugs.eclipse.org/bugs/buglist.cgi?component=logging&product=EPP • Mail: error-reports-inbox@eclipse.org
- 64. Credits • Daniel Haftstein, Codetrails – Eclipse client conception and implementation • Marcel Bruch, Codetrails – Server-side conception and implementation • Webmasters, Eclipse Foundation – Eclipse server infrastructure & integration • Thanks to many early adopters and committers providing feedback 64
- 65. Feedback 65
- 66. Contact Us 66 Codetrails GmbH Robert-Bosch-Str. 7 64293 Darmstadt, Germany www: codetrails.com phone: 06151 / 2767 092 email: info@codetrails.com Dr. Marcel Bruch CEO mobile: 0179 / 1317 721 email: marcel.bruch@codetrails.com
- 67. Feedback 67