Operational Insight: Concepts and Examples (w/o Presenter Notes)
- 1. Operational InsightJune 15, 2015 Roy Rapoport @royrapoport / linkedin.com/in/royrapoport / rrapoport@netflix.com
- 2. Oh, The Places We’ll Go!
- 4. John Boyd
- 6. Observe
- 11. Observe Orient Decide Act OODA “This approach favors agility over raw power in dealing with human opponents in any endeavor” - Wikipedia
- 12. This Is What We Do
- 16. OODA KPI
- 17. OODA KPI Speed
- 19. OODA KPI Speed Effort Reliability
- 24. Implications … for Observation (aka measurement, telemetry, metrics)
- 25. Implications … for Observation (aka measurement, telemetry, metrics) • Make It Easy
- 26. Implications … for Observation (aka measurement, telemetry, metrics) • Make It Easy • Make It Scalable
- 27. Implications … for Observation (aka measurement, telemetry, metrics) • Make It Easy • Make It Scalable • Make it pluggable
- 28. Implications … for Observation (aka measurement, telemetry, metrics) • Make It Easy • Make It Scalable • Make it pluggable • (Eventually) Ruthlessly Cull
- 29. Implications … for Observation (aka measurement, telemetry, metrics) • Make It Easy • Make It Scalable • Make it pluggable • (Eventually) Ruthlessly Cull “What decision will this help me make?”
- 30. A Joke
- 32. 52 48
- 33. % of servers in major region with an even IP address
- 34. Implications … for Orientation (aka graphing, visualization)
- 35. Implications … for Orientation (aka graphing, visualization) • First-class product
- 36. Implications … for Orientation (aka graphing, visualization) • First-class product • Different decisions require different viz
- 37. Implications … for Orientation (aka graphing, visualization) • First-class product • Different decisions require different viz • Low cognitive load better than
- 38. Implications … for Orientation (aka graphing, visualization) • First-class product • Different decisions require different viz • Low cognitive load better than • High refresh rates
- 39. Implications … for Orientation (aka graphing, visualization) • First-class product • Different decisions require different viz • Low cognitive load better than • High refresh rates • Deep data density
- 40. Better Like This …
- 41. Or Better Like That …
- 42. Implications … for Decisions (aka alerting, real-time analytics, etc)
- 43. Implications … for Decisions (aka alerting, real-time analytics, etc) • You already have (some of) this
- 44. Implications … for Decisions (aka alerting, real-time analytics, etc) • You already have (some of) this • Incremental improvement
- 45. Implications … for Decisions (aka alerting, real-time analytics, etc) • You already have (some of) this • Incremental improvement • Sky’s the limit
- 46. Implications … for Decisions (aka alerting, real-time analytics, etc) • You already have (some of) this • Incremental improvement • Sky’s the limit • For benefits
- 47. Implications … for Decisions (aka alerting, real-time analytics, etc) • You already have (some of) this • Incremental improvement • Sky’s the limit • For benefits • For cost
- 49. Implications … for Action 1. Humans beat bureaucracy
- 50. Implications … for Action 1. Humans beat bureaucracy 2. Machines beat humans
- 51. Implications … for Action 1. Humans beat bureaucracy 2. Machines beat humans 3. Repeatability beats one-offs
- 52. Implications … for Action 1. Humans beat bureaucracy 2. Machines beat humans 3. Repeatability beats one-offs Repeatable machine processes TROUNCE one-off human bureaucracy
- 53. Implications … for Action 1. Humans beat bureaucracy 2. Machines beat humans 3. Repeatability beats one-offs 4. Start with humans Repeatable machine processes TROUNCE one-off human bureaucracy
- 54. Implications … for Action 1. Humans beat bureaucracy 2. Machines beat humans 3. Repeatability beats one-offs 4. Start with humans 5. If IFTTT, deprecate humans Repeatable machine processes TROUNCE one-off human bureaucracy
- 55. Decision: Do I Have Enough Instances?
- 58. Decision: Is My Canary Good?
- 59. 25
- 61. Been there. • Started in the Data Center Done that. Manually.Artisanally. 25
- 62. Been there. • Started in the Data Center • Manual, dashboard-driven Done that. Manually.Artisanally. 25
- 65. Been there. Done that. Manually. • Context vs Precision 27
- 66. Been there. Done that. Manually. • Context vs Precision • No … 27
- 67. Been there. Done that. Manually. • Context vs Precision • No … • Repeatability 27
- 68. Been there. Done that. Manually. • Context vs Precision • No … • Repeatability • Trending 27
- 69. Been there. Done that. Manually. • Context vs Precision • No … • Repeatability • Trending • Manual effort is manual 27
- 70. So Now What? 28
- 71. So Now What? • Automate Analysis 28
- 72. So Now What? • Automate Analysis • Took Some Effort 28
- 73. So Now What? • Automate Analysis • Took Some Effort • Approach and analytics 28
- 74. So Now What? • Automate Analysis • Took Some Effort • Approach and analytics • Presentation matters 28
- 75. Version Control System 1000 servers @ 1.0.1 Customers Build & Deployment System Automated Canary Analysis Pretty Pictures 29
- 76. Version Control System 1000 servers @ 1.0.1 Customers Build & Deployment System 1 server @ 1.0.2 Automated Canary Analysis Pretty Pictures 29
- 77. 10 servers @ 1.0.2 Version Control System 1000 servers @ 1.0.1 Customers Build & Deployment System Automated Canary Analysis Pretty Pictures 29
- 78. 1000 servers @ 1.0.2 Version Control System 1000 servers @ 1.0.1 Customers Build & Deployment System Automated Canary Analysis Pretty Pictures 29
- 79. Versi on 1000 servers @ 1.0.1 Custome Build & Deployment Automat ed 1000 servers @ 1.0.2 Pretty Pictures 30 Version Control System Build & Deployment System Automated Canary Analysis Customers
- 80. Versi on Custome Build & Deployment Automat ed 1000 servers @ 1.0.2 Pretty Pictures 30 Version Control System Build & Deployment System Automated Canary Analysis Customers
- 81. Versi on 1000 servers @ 1.0.1 Custome Build & Deployment Automat ed 1000 servers @ 1.0.2 Pretty Pictures 31 Version Control System Build & Deployment System Automated Canary Analysis
- 82. Versi on 1000 servers @ 1.0.1 Custome Build & Deployment Automat ed 1000 servers @ 1.0.2 Pretty Pictures 31 Version Control System Build & Deployment System Automated Canary Analysis
- 83. Just The Stats 4-Week View
- 84. Just The Stats 4-Week View 6309 canary analysis cycles
- 85. Just The Stats 4-Week View 6309 canary analysis cycles 16% canaries failed
- 86. Decision: Do I Have an Outlier?
- 88. Would You Like to Play a Game?
- 89. Spot the Outlier
- 94. Just The Stats 4-Week View
- 95. Just The Stats 4-Week View 739 Server Terminations
- 97. In a Nutshell Observe Orient Decide Act Need This First http://bit.ly/nflx-atlas-2013 http://metrics20.org
- 98. In a Nutshell Observe Orient Decide Act Need This First http://bit.ly/nflx-atlas-2013 http://metrics20.org Understand the decision http://bit.ly/nflx-qcon-aca-2014
- 99. In a Nutshell Observe Orient Decide Act Need This First http://bit.ly/nflx-atlas-2013 http://metrics20.org Understand the decision http://bit.ly/nflx-qcon-aca-2014 Make it easier for humans
- 100. In a Nutshell Observe Orient Decide Act Need This First http://bit.ly/nflx-atlas-2013 http://metrics20.org Understand the decision http://bit.ly/nflx-qcon-aca-2014 Make it easier for humans Make machines
 do it
- 101. In a Nutshell Observe Orient Decide Act Need This First http://bit.ly/nflx-atlas-2013 http://metrics20.org Understand the decision http://bit.ly/nflx-qcon-aca-2014 Make it easier for humans Make machines
 do it Higher speed Lower effort Higher reliability