Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
1 like1,148 views
The document outlines practical operability techniques for teams, focusing on modern logging, run book dialogue sheets, endpoint health checks, correlation IDs, and user personas to enhance collaboration and operational efficiency. It emphasizes the importance of structured logging and health checks for system performance, as well as understanding user roles for improved decision-making. Key methodologies and examples are provided to illustrate concepts and improve software operability in production environments.
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
- 1. Practical Operability Techniques for Teams Matthew Skelton Skelton Thatcher Consulting skeltonthatcher.com / @SkeltonThatcher Agile in the City, Bristol 2017, London – 02 Nov 2017
- 2. Today What is operability? Modern logging Run Book dialogue sheets Endpoint healthchecks Correlation IDs User Personas for dashboards
- 3. You Software Developer Tester / QA DevOps Engineer Team Leader Head of Department
- 4. Operability: use modern logging, Run Book dialogue sheets, endpoint healthchecks, correlation IDs, and user personas as team collaboration techniques
- 6. About me Matthew Skelton @matthewpskelton Co-founder at Skelton Thatcher Consulting skeltonthatcher.com
- 7. Team Guide to Software Operability Matthew Skelton & Rob Thatcher skeltonthatcher.com/publications Download a free sample chapter
- 8. Team Guide series #Operability #BusinessMetrics #Testability #Releasability skeltonthatcher.com/publications
- 9. Practical Operability Techniques for Teams
- 11. Operability making software work well in Production
- 13. Logging with Event IDs
- 15. logging with Event IDs reduce time to detect problems increase team engagement increase configurability enhance collaboration #operability
- 18. How many distinct event types (state transitions) in your application?
- 21. enum Human-readable sets: unique values, sparse, immutable C#, Java, Python, node (Ruby, PHP, …)
- 22. Technical Domain public enum EventID { // Badly-initialised logging data NotSet = 0, // An unrecognised event has occurred UnexpectedError = 10000, ApplicationStarted = 20000, ApplicationShutdownNoticeReceived = 20001, MessageQueued = 40000, MessagePeeked = 40001, BasketItemAdded = 60001, BasketItemRemoved = 60002, CreditCardDetailsSubmitted = 70001, // ... }
- 23. BasketItemAdded = 60001 BasketItemRemoved = 60002
- 24. log using Event IDs with a modern ‘structured logging’ library
- 28. Example: video processing On-demand processing of TV and mobile streaming adverts Ad-agency TV broadcaster High throughput Glitch-free video & audio
- 32. Example: video processing Discover processing bottlenecks Trigger alerts via LogEntries / HostedGraphite Report on KPIs Target areas for improvement
- 34. Run Book dialogue sheets
- 36. Run Book dialogue sheets Checklists for typical operational considerations Team-friendly exploration
- 37. runbooktemplate.infoRun Book dialogue sheets
- 38. System characteristics Hours of operation During what hours does the service or system actually need to operate? Can portions or features of the system be unavailable at times if needed? Hours of operation - core features (e.g. 03:00-01:00 GMT+0) Hours of operation - secondary features (e.g. 07:00-23:00 GMT+0) Data and processing flows How and where does data flow through the system? What controls or triggers data flows? (e.g. mobile requests / scheduled batch jobs / inbound IoT sensor data ) …
- 40. runbooktemplate.infoRun Book dialogue sheets
- 43. endpoint healthchecks Every runnable app/service/daemon exposes /status/health An HTTP GET to the endpoint returns: 200 – "I am healthy" 500 – "I am sick"
- 44. endpoint healthchecks Each component is responsible for determining its own health and viability – this is very contextual
- 45. endpoint healthchecks Use JSON as a response type – parsable by both machines and humans!
- 47. endpoint healthchecks For databases and other non-HTTP components, run a lightweight HTTP service in front of the component 200 / 500 responses
- 48. Helper service
- 50. Correlation IDs
- 52. ‘Unique-ish’ identifier for each request Passed through downstream layers
- 53. Unique-ish ID
- 54. Synchronous HTTP: X-HEADER e.g. X-trace-id X-trace-id: 348e1cf8 If header is present, pass it on (Yes, RFC6648, but this is internal only)
- 55. Asynchonous (queues, etc.): Message Attributes, name:value pair e.g. "trace-id":"348e1cf8" AWS SQS: SendMessage() / ReceiveMessage() Log the Correlation ID if present
- 56. Example: electronic trading High speed, low latency Trading options & derivatives Connected to stock exchanges Sub-millisecond timings > £1 million per day traded
- 60. Correlations IDs for trading Evidence for timely operation Help identify bottlenecks Target areas for perf tuning Identify race conditions Increase operability
- 61. Example: OpenTracing / PCF 3 tracing elements: TraceID, SpanID, ParentSpan "X-B3-TraceId" "X-B3-SpanId" "X-B3-ParentSpan"
- 62. Example: OpenTracing / PCF Always log the TraceID as-is Log calling SpanID as ParentSpan Log new SpanID
- 66. Lightweight user personas: Ops Engineer Test Engineer Build & Deployment Engineer Service Owner
- 67. Lightweight user personas: Consider the User Experience (UX) of engineers and team members using and working with the software
- 70. Lightweight user personas: What data does the User Persona need visible on a dashboard in order to make decisions rapidly & safely?
- 71. Summary
- 72. Operability making software work well in Production
- 73. logging with Event IDs use enum-based Event IDs to explore runtime behaviour and fault conditions
- 74. Run Book dialogue sheets explore and establish operational requirements as a team, around a physical table, together
- 75. endpoint healthchecks HTTP 200 / 500 responses to /status/health call with JSON details – good for tools and humans
- 76. Correlation IDs trace execution using correlation IDs: synchronous (HTTP X-trace-id) async (SQS MessageAttribute)
- 77. lightweight user personas explore the UX and needs of different roles for rapid decisions via dashboards
- 78. Operability use modern logging, Run Book dialogue sheets, endpoint healthchecks, correlation IDs, and user personas as team collaboration techniques
- 79. Team Guide to Software Operability Matthew Skelton & Rob Thatcher skeltonthatcher.com/publications Download a free sample chapter
- 80. Team Guide series #Operability #BusinessMetrics #Testability #Releasability skeltonthatcher.com/publications
- 81. Resources • Team Guide to Software Operability by Matthew Skelton and Rob Thatcher (Skelton Thatcher Publications, 2016) http://operabilitybook.com/ • Run Book template & Run Book dialogue sheets http://runbooktemplate.info/
- 82. Questions? Twitter: @SkeltonThatcher | #operability email: questions@skeltonthatcher.com