Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton
4 likes1,501 views
Matthew Skelton's talk at TechUG discusses common issues with logging in software development, highlighting its often neglected state, including discontinuity, limited error reporting, and lack of aggregation. He proposes strategies to improve logging practices, such as using continuous event IDs, transaction tracing, and log aggregation tools. The goal is to enhance operability and collaboration between development and operations teams by making logging a first-class component of the software process.
![{
"eventmappings": {
"events": {
"event": [ {
"id": "CacheServiceStarted",
"severity": { "level": "Information" }
}, {
"id": "PageCachePurged",
"severity": { "level": "Debug" },
"state": { "enabled": false }
}, {
"id": "DatabaseConnectionTimeOut",
"severity": { "level": "Error" }
} ]
}
}
}](https://image.slidesharecdn.com/un-broken-logging-technologyug-leeds-matthew-skelton-151022172156-lva1-app6891/85/Un-broken-Logging-TechnologyUG-Leeds-Matthew-Skelton-112-320.jpg)
Un-broken Logging - TechnologyUG - Leeds - Matthew Skelton
- 1. Un-Broken Logging the foundation of software operability TechnologUG, Leeds #techug Thursday 22th October 2015 Matthew Skelton Skelton Thatcher Consulting @matthewpskelton
- 3. The way we use logging is (often) broken How to make our logging more awesome Why we should care
- 13. confession: I am a big fan of logging
- 16. execution trace
- 17. BAD STUFF
- 18. Logging is often unloved 1. Discontinuous 2. Errors only, or arbitrary 3. ‘Bolted on’ 4. No aggregation & search 5. Specify severity up front
- 19. GOOD STUFF
- 20. How to make logging awesome 1. Continuous event IDs 2. Transaction tracing 3. Log aggregation & search tools 4. Design for logging 5. Decoupled severity
- 21. reduce time-to-detect increase team engagement increase configurability enhance DevOps collaboration #operability
- 22. Background
- 26. MRI brain scan imaging
- 39. logging makes things work
- 41. How is logging usually broken?
- 42. Logging is often unloved 1. Discontinuous 2. Errors only, or arbitrary 3. ‘Bolted on’ 4. No aggregation & search 5. Specify severity up front
- 44. using logging mainly for errors
- 46. inconsistent use of logging
- 48. logging slows down the software
- 50. logging ‘pollutes’ my precious domain model
- 52. logging is just for those weird Ops people
- 54. logging assumed to be free ($0) to implement no budget for aggregating logs across machines
- 56. log aggregation happens only in Production logs not available to Devs
- 58. fights over log severity levels
- 62. Some history, with pirates
- 67. weather, course, sightings, latitude, longitude, … (even when quiet)
- 68. JohnHarrison
- 69. Why log?
- 72. - June 13th – Pirates!!!! - Weds – Sharks!!! - 19th Jun –BIGGER sharks!!!!
- 73. How to make logging awesome
- 74. How to make logging awesome 1. Continuous event IDs 2. Transaction tracing 3. Log aggregation & search tools 4. Design for logging 5. Decoupled severity
- 78. How many distinct event types (state transitions) in your application?
- 81. enum Human-readable sets: unique values, sparse, immutable C#, Java, Python, node (Ruby, PHP, …)
- 82. public enum EventID { // Badly-initialised logging data NotSet = 0, // An unrecognised event has occurred UnexpectedError = 10000, ApplicationStarted = 20000, ApplicationShutdownNoticeReceived = 20001, PageGenerationStarted = 30000, PageGenerationCompleted = 30001, MessageQueued = 40000, MessagePeeked = 40001, BasketItemAdded = 60001, BasketItemRemoved = 60002, CreditCardDetailsSubmitted = 70001, // ... }
- 83. Technical Domain public enum EventID { // Badly-initialised logging data NotSet = 0, // An unrecognised event has occurred UnexpectedError = 10000, ApplicationStarted = 20000, ApplicationShutdownNoticeReceived = 20001, PageGenerationStarted = 30000, PageGenerationCompleted = 30001, MessageQueued = 40000, MessagePeeked = 40001, BasketItemAdded = 60001, BasketItemRemoved = 60002, CreditCardDetailsSubmitted = 70001, // ... }
- 85. BasketItemAdded = 60001 BasketItemRemoved = 60002
- 86. BasketItemAdded = 60001 BasketItemRemoved = 60002
- 91. Monolith to microservices: debugger does not have the full view
- 92. Even with remote debugger, it’s boring to attach and detach
- 99. ‘Unique-ish’ identifier for each request Passed through downstream layers
- 100. Unique-ish ID
- 101. What about APM?
- 102. APM gives us application insight BUT How much do we learn? Is APM available on the Dev box? It’s not just ‘an Ops problem’!
- 104. Helps us to understand how the software really works Small overhead is worth it
- 106. Which log level is right?
- 107. DEBUG, INFO, WARNING, ERROR, CRITICAL
- 108. Log level should *not* be fixed at compile or build time!
- 109. Tune log levels
- 110. Tune log levels
- 111. Tune log levels
- 112. { "eventmappings": { "events": { "event": [ { "id": "CacheServiceStarted", "severity": { "level": "Information" } }, { "id": "PageCachePurged", "severity": { "level": "Debug" }, "state": { "enabled": false } }, { "id": "DatabaseConnectionTimeOut", "severity": { "level": "Error" } } ] } } }
- 114. Tune severity levels of specific event IDs
- 116. Event tracing Use enumerations (or closest thing) Technical and Domain event types Distributed systems: debuggers less useful Trace calls with ‘unique-enough’ handles Tune log levels via config
- 117. Log aggregation & search tools
- 122. Design for log aggregation
- 123. develop the software using log aggregation as a first-class thing
- 126. stories for testing logging
- 128. logging is (‘just’) another system component
- 129. NTP
- 130. Dev and Ops collaboration* * and testers too!
- 132. Where?
- 135. Recap
- 136. Logging is often unloved 1. Discontinuous 2. Errors only, or arbitrary 3. ‘Bolted on’ 4. No aggregation & search 5. Specify severity up front
- 137. How to make logging awesome 1. Continuous event IDs 2. Transaction tracing 3. Log aggregation & search tools 4. Design for logging 5. Decoupled severity
- 138. logging makes things work
- 139. “There is no thought behind aspect-oriented programming”
- 140. MINDFUL LOGGING (?!)
- 142. ‘Structured Logging’ TW: “Adopt” (May 2015) https://www.thoughtworks.com/radar/techniques/structured-logging http://gregoryszorc.com/ .NET: http://serilog.net/ Java: https://github.com/fluent/fluent-logger-java
- 143. sanity
- 145. More Ditch the Debugger and Use Log Analysis Instead Matthew Skelton https://blog.logentries.com/2015/07/ditch- the-debugger-and-use-log-analysis-instead/
- 146. More Using Log Aggregation Across Dev & Ops: The Pricing Advantage Rob Thatcher https://blog.logentries.com/2015/08/using- log-aggregation-across-dev-ops-the-pricing- advantage/
- 149. Thank you http://skeltonthatcher.com/ enquiries@skeltonthatcher.com @SkeltonThatcher +44 (0)20 8242 4103 @matthewpskelton