Practical operability techniques - Matthew Skelton - Unicom DevOps Showcase North 2019
1 like333 views
The document presents five practical operability techniques for software teams, emphasizing the importance of operability in production environments. Techniques include modern event-based logging, run book dialogue sheets, endpoint healthchecks, correlation ids, and lightweight user personas to enhance team collaboration and system awareness. The author, Matthew Skelton, advocates for shared responsibility in operability among team members and provides resources for further learning.
Practical operability techniques - Matthew Skelton - Unicom DevOps Showcase North 2019
- 1. 1 5 practical operability techniques for teams Matthew Skelton, Conflux @matthewpskelton confluxdigital.net UNICOM DevOps Showcase North 2019 Manchester, 27 February 2019
- 2. Practical operability Why do we need a focus on operability? 5 practical operability techniques that work 2
- 3. modern event-based logging Run Book dialogue sheets endpoint healthchecks correlation IDs user personas 3
- 5. About me Matthew Skelton, Conflux @matthewpskelton matthewskelton.net Leeds, UK 5 20% discount for DevOps Showcase!
- 6. Team Guide to Software Operability Matthew Skelton & Rob Thatcher operabilitybook.com 20% discount for Unicom events http://leanpub.com/SoftwareOperability/c/UNICOM 6
- 7. You? Software Developer Tester / QA DevOps Engineer Ops Engineer / SRE Head of Department 7
- 8. 5 Practical Operability Techniques for Teams 8
- 9. Operability making software work well in Production 9
- 11. “But can’t we just give those things to an SRE?” 11
- 12. “But can’t we just give those things to the DevOp?” 12
- 13. Operability is a shared concern #BizDevTestSecOps 13
- 14. Operability is a shared concern #BizDevTestSecOps 14
- 15. 15
- 16. 16 A self-managed Kubernetes cluster near you
- 17. Operability is a shared concern #BizDevTestSecOps 17
- 18. 18 SRE: operability consultants Collaborate on operability here CC BY-SA devopstopologies.com
- 19. Practical operability techniques 1. Modern logging with event IDs 2. Run Book dialogue sheets 3. Endpoint healthchecks 4. Correlation IDs 5. Lightweight User Personas 19
- 20. 20 Logging with Event IDs
- 21. Lack of observability for distributed systems 21
- 22. Modern logging w/ Event IDs Distinct application states No “logorrhoea” (!) Distributed tracing via logs Build a shared understanding 22
- 23. search by event Event ID {Delivered, InTransit, Arrived} 23
- 25. Modern logging with event IDs helps to produce a well-defined event space: human-readable events 25
- 26. Which calls might fail? 26
- 27. How many distinct event types (state transitions) in your application? 27
- 28. 28
- 30. enum Human-readable sets: unique values, sparse, immutable C#, Java, Python, node (Ruby, PHP, …) 30
- 31. Technical Domain public enum EventID { // Badly-initialised logging data NotSet = 0, // An unrecognised event has occurred UnexpectedError = 10000, ApplicationStarted = 20000, ApplicationShutdownNoticeReceived = 20001, MessageQueued = 40000, MessagePeeked = 40001, BasketItemAdded = 60001, BasketItemRemoved = 60002, CreditCardDetailsSubmitted = 70001, // ... } 31
- 32. BasketItemAdded = 60001 BasketItemRemoved = 60002 32
- 34. Example: video processing On-demand processing of TV and mobile streaming adverts Ad-agency → TV broadcaster High throughput Glitch-free video & audio 34
- 36. 36
- 37. 37
- 38. Example: video processing Discover processing bottlenecks Trigger alerts Report on KPIs Target areas for improvement 38
- 39. Modern logging w/ Event IDs clarity about software behavior reduce time to detect problems increase team engagement enhance collaboration 39
- 40. Modern Logging: Collaborate on Event IDs and Correlation traces for better system awareness 40
- 41. Run Book dialogue sheets 41
- 42. Operational aspects not addressed, or addressed too late in the cycle 42
- 43. Run Book dialogue sheets Checklists for typical operational considerations Team-friendly exploration 43
- 44. Run Book dialogue sheets help to increase awareness of operability within teams 44
- 45. runbooktemplate.infoRun Book dialogue sheets 45
- 46. System characteristics Hours of operation During what hours does the service or system actually need to operate? Can portions or features of the system be unavailable at times if needed? Hours of operation - core features (e.g. 03:00-01:00 GMT+0) Hours of operation - secondary features (e.g. 07:00-23:00 GMT+0) Data and processing flows How and where does data flow through the system? What controls or triggers data flows? (e.g. mobile requests / scheduled batch jobs / inbound IoT sensor data ) … 46
- 48. runbooktemplate.infoRun Book dialogue sheets 48
- 49. Run Book dialogue sheets Early discovery of operational needs Input to team backlog “Shift-left” testing Avoid operational problems 49
- 51. OperabilityQuestions.com Freeform, exploratory questions for teams Usability, viability, reliability, observability, securability, … (Github, CC BY-SA) 51
- 52. Run Book dialogue sheets: Collaborate on operational requirements for better system awareness 52
- 54. “Why has my deployment failed again?” “Why is Pre-Prod always so flaky?” 54
- 55. Endpoint healthchecks Simple HTTP check Common way to assess any service/app/component Key operational requirement 55
- 56. endpoint healthchecks Every runnable app/service/daemon exposes /status/health An HTTP GET to the endpoint returns: 200 – "I am healthy" 500 – "I am sick" 56
- 57. Endpoint healthchecks help teams to collaborate on service viability 57
- 58. endpoint healthchecks Each component is responsible for determining its own health and viability – this is very contextual 58
- 59. endpoint healthchecks Use JSON as a response type – parsable by both machines and humans! 59
- 60. 60
- 61. endpoint healthchecks For databases and other non-HTTP components, run a lightweight HTTP service in front of the component 200 / 500 responses 61
- 64. 64 Question: What does this look like for Serverless? ¯_(ツ)_/¯
- 65. Endpoint healthchecks Rapid diagnosis and visibility Reduce confusion around environment state “Fail fast” → “learn sooner” 65
- 66. Endpoint healthchecks: Collaborate on component health status for better system awareness 66
- 68. “Which nodes handled the request?” 68
- 69. Correlation IDs Unique-ish identifiers Trace calls across machine & container boundaries Re-assemble the HTTP call later 69
- 70. ‘Unique-ish’ identifier for each request Passed through downstream layers 70
- 71. Correlation IDs help teams to think about the big picture: end-to-end outcomes 71
- 72. Unique-ish ID 72
- 73. Synchronous HTTP: X-HEADER e.g. X-trace-id X-trace-id: 348e1cf8 If header is present, pass it on (Yes, RFC6648, but this is internal only) 73
- 74. Asynchonous (queues, etc.): Message Attributes, name:value pair e.g. "trace-id":"348e1cf8" AWS SQS: SendMessage() / ReceiveMessage() Log the Correlation ID if present 74
- 75. Example: OpenTracing / PCF 3 tracing elements: TraceID, SpanID, ParentSpan "X-B3-TraceId" "X-B3-SpanId" "X-B3-ParentSpan" 75
- 76. Example: OpenTracing / PCF Always log the TraceID as-is Log calling SpanID as ParentSpan Log new SpanID 76
- 78. Correlation IDs Detect bottlenecks and unexpected interactions Increase transparency Learn about the system 78
- 79. Correlation IDs: Collaborate on distributed tracing for better system awareness 79
- 81. Software is difficult to operate: poor UX for Ops. 81
- 82. Lightweight User Personas Simple characterisation of user needs for Dev/Test/Ops Based on full UX user personas but less detailed 82
- 83. Lightweight user personas: Ops Engineer Test Engineer Build & Deployment Engineer Service Owner 83
- 84. Lightweight user personas help teams to build systems with good UX for all users 84
- 85. Lightweight user personas: Consider the User Experience (UX) of engineers and team members using and working with the software 85
- 86. http://www.keepitusable.com/blog/?tag=alan-cooper 86 Motivations Goals Frustrations
- 87. Lightweight user personas: What data does the User Persona need visible on a dashboard in order to make decisions rapidly & safely? 87
- 89. Lightweight User Personas Empathise better with people from other roles Capture missing operational requirements 89
- 90. Lightweight User Personas: Collaborate on user needs for better system awareness 90
- 91. Summary 91
- 92. Operability making software work well in Production 92
- 93. 93
- 94. Lack of observability Operational aspects not known “Why has deployment failed?” What handled the request? Poor UX for Ops 94
- 95. 95 SRE: operability consultants Collaborate on operability here CC BY-SA devopstopologies.com
- 96. Logging with Event IDs use enum-based Event IDs to explore runtime behaviour and fault conditions 96
- 97. Run Book dialogue sheets explore and establish operational requirements as a team, around a physical table, together 97
- 98. Endpoint healthchecks HTTP 200 / 500 responses to /status/health call with JSON details – good for tools and humans 98
- 99. Correlation IDs trace execution using correlation IDs: synchronous (HTTP X-trace-id) async (SQS MessageAttribute) 99
- 100. Lightweight user personas explore the UX and needs of different roles for rapid decisions via dashboards 100
- 101. use modern logging, Run Book dialogue sheets, endpoint healthchecks, correlation IDs, and user personas as team collaboration techniques 101
- 102. Team Guide to Software Operability Matthew Skelton & Rob Thatcher operabilitybook.com 20% discount for Unicom events http://leanpub.com/SoftwareOperability/c/UNICOM 102
- 103. Resources •Team Guide to Software Operability by Matthew Skelton and Rob Thatcher http://operabilitybook.com/ •Run Book template & Run Book dialogue sheets http://runbooktemplate.info/ •Operability Questions http://operabilityquestions.com/ •5 proven operability techniques for software teams https://techbeacon.com/5-proven-operability-techniques-s oftware-teams 103
- 104. thank you 104 @matthewpskelton / operabilitybook.com @ConfluxHQ / confluxdigital.net