SlideShare a Scribd company logo

How our security requirements turned us into accidental chaos engineers

P
Paul Carleton

The document discusses the challenges and experiences faced by Paul Carleton at Stripe regarding cloud instance management and the importance of lifespan management. It highlights the negative impacts of old instances and details their implementation of automated termination strategies, alongside chaotic discoveries made during the process. The takeaway emphasizes the utility of chaos in solving automation problems and understanding instance age.

Technology
1 of 113
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
@paulcarletonjr How our security requirements turned us into accidental chaos engineers
@paulcarletonjr “We don’t make mistakes, we have happy accidents - Bob Ross
@paulcarletonjr@paulcarletonjr Hello! I’m Paul Carleton @ Stripe (we’re hiring!) 3
@paulcarletonjr Topics / Spoilers 1. Old Instances are bad 2. Enter Lifespan Management 3. Some stories of how things broke 4. What we learned
@paulcarletonjr 1. Instance Age What is it and why do I care?
@paulcarletonjr Terminology ▷ Instance: Cloud Hosted VM (EC2) ▷ Age: Time since launch
@paulcarletonjr OldYoung # of hosts Just launched Instance Age
@paulcarletonjr OldYoung # of hosts Weeks later Instance Age
@paulcarletonjr OldYoung # of hosts Months later Instance Age
@paulcarletonjr OldYoung # of hosts Terminate & Replace Instance Age
@paulcarletonjr OldYoung # of hosts Instance Age
@paulcarletonjr OldYoung # of hosts Instance Age New hadoop cluster
@paulcarletonjr OldYoung # of hosts Instance Age New hadoop cluster Big migration
@paulcarletonjr OldYoung # of hosts Instance Age New hadoop cluster Big migration
@paulcarletonjr What’s wrong with old instances?
@paulcarletonjr What’s wrong with old instances? Replacement is like a fire extinguisher...
@paulcarletonjr OldYoung # of hosts Instance Age Last replacement Breaking changes
@paulcarletonjr What’s wrong with old instances? Replacement is like a fire extinguisher... … that might catch on fire
@paulcarletonjr What’s wrong with old instances? Replacement is like a fire extinguisher... … that might catch on fire
@paulcarletonjr What’s wrong with old instances?
@paulcarletonjr What’s wrong with old instances? Will replacing a with a work?
@paulcarletonjr What’s wrong with old instances?
@paulcarletonjr OldYoung # of hosts CVE Patch Released
@paulcarletonjr OldYoung # of hosts CVE Patch Released
@paulcarletonjr OldYoung # of hosts CVE Patch Released
@paulcarletonjr OldYoung # of hosts CVE Patch Released
@paulcarletonjr OldYoung # of hosts CVE Patch Released
@paulcarletonjr OldYoung # of hosts CVE Patch Released
@paulcarletonjr Old Instances are bad
@paulcarletonjr 2. Lifespan Management
@paulcarletonjr Components ▷ ASG ▷ Terminator ▷ Lifespan Manager:
@paulcarletonjr What is an auto-scaling group ? ✨ ASG ✨
@paulcarletonjr Terminator Terminate Wait AWS Shave yaks Shutdown 2 What is a terminator? 3 4 1
@paulcarletonjr Lifespan Manager (waiting) Lifespan Manager ASG
@paulcarletonjr Terminate First vs. Launch First Steady State ASG Size Time
@paulcarletonjr Terminate First vs. Launch First Steady State ASG Size Time
@paulcarletonjr “Now what?
Rollout Plan Breaking the problem up with labels 38
Breaking it up with labels Stateless Safe to replace Stateful Automated Replaceable with some graceful state hand-off. Requires Operator Not safe to replace automatically. Want someone watching 39
@paulcarletonjr@paulcarletonjr Automated termination What could go wrong?
@paulcarletonjr A Year Long Journey
@paulcarletonjr 5 Chaotic Discoveries
@paulcarletonjr 3.1 How NOT to health check
@paulcarletonjr “The lifespan manager terminated all the LDAP servers. We’re locked out of QA.
@paulcarletonjr Don’t we check for this?
@paulcarletonjr How did this happen? What’s your health?
@paulcarletonjr How did this happen? What’s your health? LDAP Maintenance
@paulcarletonjr How did this happen? What’s your health? LDAP Maintenance Maintenance
@paulcarletonjr How did this happen? What’s your health? LDAP Maintenance Maintenance Everything’s green! Let’s terminate!
0 unhealthy != healthy 50
@paulcarletonjr How NOT to health check ▷ Pick good defaults ▷ Use pre-shared knowledge to verify health
@paulcarletonjr Explicit Expectations What’s your LDAP health? LDAP Maintenance Maintenance
@paulcarletonjr Explicit Expectations What’s your LDAP health? LDAP Maintenance Maintenance No LDAP? I better wait.
@paulcarletonjr 3.2 RIP Kubernetes Workers
@paulcarletonjr “The Kubernetes workers are going down… HARD!
@paulcarletonjr Terminator Terminate Wait AWS Shave yaks Shutdown 2 Terminator Recap 3 4 1
@paulcarletonjr Terminator Terminate Wait AWS Shave yaks Shutdown 2 Terminator Recap 3 4 1
@paulcarletonjr Terminate Terminate
@paulcarletonjr Terminate Terminate
@paulcarletonjr Terminate Terminate
@paulcarletonjr Terminate Terminate
@paulcarletonjr RIP Kubernetes Workers ● Track feature usage ● Make the chaos easy to turn off
3.3 Blackhole Scenario 63
@paulcarletonjr TerminatorTerminate Wait AWS Shave yaks Shutdown 1 2 34 Terminator Recap
Heartbeat Options Delay termination Proceed with termination … but no Cancel 65
@paulcarletonjr TerminatorTerminate Wait AWS Shave yaks Shutdown 1 2 3 Blackhole Scenario
@paulcarletonjr Blackhole Scenario ● Non-zero exit ● Timeouts ● Rate limits
@paulcarletonjr “The terminations will continue until morale improves!
69
@paulcarletonjr Terminator Terminate AWS Already done! Shutdown 1 2 3 4 Two Touch Termination Terminator Shave yaks 1 2 3
@paulcarletonjr The Blackhole Scenario ● Align incentives ● Systems vary, so adapt to match!
@paulcarletonjr 3.4 Self-Service Meltdown
@paulcarletonjr A Year Long Journey
@paulcarletonjr A Year Long Journey
@paulcarletonjr Enhance!
@paulcarletonjr Enhance!
@paulcarletonjr “If you would like to never think about a kernel upgrade again, consider Lifespan Management!
@paulcarletonjr Part 1: Turning it On
@paulcarletonjr I want to enable lifespan management!
@paulcarletonjr Great! Here are some docs!
@paulcarletonjr
@paulcarletonjr Part 2: Who does what?
@paulcarletonjr What happens during termination?
@paulcarletonjr Let me tell you!
@paulcarletonjr yadda yadda yadda yadda yadda yadda yadda yadda yadda yadda yadda yadda yadda yadda yadda yadda
@paulcarletonjr yadda yadda yadda yadda yadda yadda yadda yadda yadda yadda yadda yadda yadda yadda yadda yadda
@paulcarletonjr What part of that is relevant to me?
@paulcarletonjr … Great question!
@paulcarletonjr Part 3: False Alarms
@paulcarletonjr Did lifespan management just break my thing?
@paulcarletonjr Let me check!
@paulcarletonjr aws
@paulcarletonjr aws 5 minutes later...
@paulcarletonjr Nope!
@paulcarletonjr Okay… what did break my thing?
@paulcarletonjr Okay… what did break my thing?
@paulcarletonjr How we changed
@paulcarletonjr Part 1: Turning it On
@paulcarletonjr Part 2: Who does what?
@paulcarletonjr Part 3: False Alarm aws go/whydead/$instance_id
@paulcarletonjr Self-service Meltdown ▷ Make it easy to adopt safely ▷ Explicitly state the contract ▷ Make it easy to rule chaos out
@paulcarletonjr 3.5 Death by a thousand JIRA tickets
@paulcarletonjr Something’s wrong, I can’t terminate anything These warnings should be tickets!
@paulcarletonjr Great!
@paulcarletonjr
@paulcarletonjr
@paulcarletonjr Death by a thousand JIRA tickets ● File against ourselves first, then automate ● 1% case matters more with 10x terminations ● Measure Quantity and Reliability of tickets
@paulcarletonjr 4. Calling it Done The End… for now
@paulcarletonjr 5. Summary and Closing
@paulcarletonjr Takeaway ● What automation problems can you solve with a little chaos?
@paulcarletonjr Takeaway Do you know how old your instances are?
@paulcarletonjr@paulcarletonjr Thank you!
@paulcarletonjr Credits ● Photo by rawpixel on Unsplash ● Photo by Jens Lelie on Unsplash ● Photo by JohnsonMartin https://pixabay.com/en/wormhole-space-time-light-tunnel-739872/

More Related Content

PPT
My Road To Test Driven Development
Gerard Sychay
 
PDF
Testing in the 21st Century
Alex Soto
 
PDF
Testing Java Microservices Workshop
Alex Soto
 
PDF
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at QCon NYC 2017
Baruch Sadogursky
 
PDF
Habitat Service Discovery
Nell Shamrell-Harrington
 
PDF
TEST SMARTER AND GAIN SOME TIME BACK
Alex Soto
 
PDF
A Taxonomy of Clustering, or, No Container is an Island
Ted M. Young
 
PPTX
Word study 1 vocabulary practice
Steven Gabrys
 
My Road To Test Driven Development
Gerard Sychay
 
Testing in the 21st Century
Alex Soto
 
Testing Java Microservices Workshop
Alex Soto
 
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at QCon NYC 2017
Baruch Sadogursky
 
Habitat Service Discovery
Nell Shamrell-Harrington
 
TEST SMARTER AND GAIN SOME TIME BACK
Alex Soto
 
A Taxonomy of Clustering, or, No Container is an Island
Ted M. Young
 
Word study 1 vocabulary practice
Steven Gabrys
 

Similar to How our security requirements turned us into accidental chaos engineers (20)

PDF
Using Machine Learning to Debug Oracle RAC Issues
Anil Nair
 
PDF
OpenNebulaConf 2013 - Monitoring of OpenNebula installations by Florian Heigl
OpenNebula Project
 
PDF
Monitoring of OpenNebula installations
NETWAYS
 
PDF
DevOpsDays Austin: Helping Horses Become Unicorns, Chef's Operations Maturity...
Matt Ray
 
PDF
Smart monitoring how does oracle rac manage resource, state ukoug19
Anil Nair
 
PDF
Docker Cluster Management with ECS
Matt Callanan
 
PPTX
Oracle real application clusters system tests with demo
Ajith Narayanan
 
PPTX
DevOps and the Death & Rebirth of Childhood Innocence
Robert Douglass
 
PPTX
10 Tips for Your Journey to the Public Cloud
Intuit Inc.
 
PDF
An introduction to_rac_system_test_planning_methods
Ajith Narayanan
 
PPTX
Velocity 2015: Building Self-Healing Systems
SOASTA
 
PPTX
Velocity 2015 building self healing systems (slide share version)
SOASTA
 
PDF
An Engineer's Guide to a Good Night's Sleep
C4Media
 
PDF
Automating Security in Cloud Workloads with DevSecOps
Kristana Kane
 
PDF
Synergy 2015 Session Slides: SYN408 XenDesktop 7.6 Architecture - Dealing Wit...
Citrix
 
KEY
DevOpsSec: Appling DevOps Principles to Security, DevOpsDays Austin 2012
Nick Galbreath
 
PDF
us-19-Shortridge-Forsgren-Controlled-Chaos-the-Inevitable-Marriage-of-DevOps-...
CynthiaRothrock
 
PPTX
Chaos Engineering: Why Breaking Things Should Be Practised.
Adrian Hornsby
 
PPTX
Harnessing the Power of Apache Hadoop Series
Cloudera, Inc.
 
PDF
Immutable infrastructure - Beyond stateless
Jorge Dias
 
Using Machine Learning to Debug Oracle RAC Issues
Anil Nair
 
OpenNebulaConf 2013 - Monitoring of OpenNebula installations by Florian Heigl
OpenNebula Project
 
Monitoring of OpenNebula installations
NETWAYS
 
DevOpsDays Austin: Helping Horses Become Unicorns, Chef's Operations Maturity...
Matt Ray
 
Smart monitoring how does oracle rac manage resource, state ukoug19
Anil Nair
 
Docker Cluster Management with ECS
Matt Callanan
 
Oracle real application clusters system tests with demo
Ajith Narayanan
 
DevOps and the Death & Rebirth of Childhood Innocence
Robert Douglass
 
10 Tips for Your Journey to the Public Cloud
Intuit Inc.
 
An introduction to_rac_system_test_planning_methods
Ajith Narayanan
 
Velocity 2015: Building Self-Healing Systems
SOASTA
 
Velocity 2015 building self healing systems (slide share version)
SOASTA
 
An Engineer's Guide to a Good Night's Sleep
C4Media
 
Automating Security in Cloud Workloads with DevSecOps
Kristana Kane
 
Synergy 2015 Session Slides: SYN408 XenDesktop 7.6 Architecture - Dealing Wit...
Citrix
 
DevOpsSec: Appling DevOps Principles to Security, DevOpsDays Austin 2012
Nick Galbreath
 
us-19-Shortridge-Forsgren-Controlled-Chaos-the-Inevitable-Marriage-of-DevOps-...
CynthiaRothrock
 
Chaos Engineering: Why Breaking Things Should Be Practised.
Adrian Hornsby
 
Harnessing the Power of Apache Hadoop Series
Cloudera, Inc.
 
Immutable infrastructure - Beyond stateless
Jorge Dias
 
Ad

Recently uploaded (20)

PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PDF
Getting Started with Data Integration: FME Form 101
Safe Software
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PPTX
Spectroscopy.pptx food analysis technology
nawahassan45
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PPTX
1. Introduction to Computer Programming.pptx
bonakiduza1024
 
PPTX
Tartificialntelligence_presentation.pptx
ry7r52mzb4
 
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PPTX
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
PDF
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
PDF
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PPTX
SOPHOS-XG Firewall Administrator PPT.pptx
AgnesMunisi
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
A comparative analysis of optical character recognition models for extracting...
IAESIJAI
 
PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
Approach and Philosophy of On baking technology
30anthuong17
 
Getting Started with Data Integration: FME Form 101
Safe Software
 
Encapsulation theory and applications.pdf
gurumoop
 
Spectroscopy.pptx food analysis technology
nawahassan45
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Teaching material agriculture food technology
LiaRayya
 
1. Introduction to Computer Programming.pptx
bonakiduza1024
 
Tartificialntelligence_presentation.pptx
ry7r52mzb4
 
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
SOPHOS-XG Firewall Administrator PPT.pptx
AgnesMunisi
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
A comparative analysis of optical character recognition models for extracting...
IAESIJAI
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
Ad

How our security requirements turned us into accidental chaos engineers