SlideShare a Scribd company logo
@molly_struve
Elasticsearch 5
or and
Bust
1
@molly_struve
2
@molly_struve
TL;DR
3
@molly_struve
4
@molly_struve
5
@molly_struve
6
@molly_struve
7
@molly_struve
8
@molly_struve
9
@molly_struve
Elasticsearch Lingo
10
@molly_struve
11
Elasticsearch
ES
@molly_struve
12
Node NodeNode
Servers
@molly_struve
13
Node NodeNode
Cluster
@molly_struve
14
2.x 5.x
@molly_struve
15
The Story
@molly_struve
16
2017
March
@molly_struve
17
Cluster
@molly_struve
18
@molly_struve
19
@molly_struve
Upgrade Steps
20
1
2
3
4
Shutdown the cluster
@molly_struve
Upgrade Steps
21
1
2
3
4
Shutdown the cluster
Upgrade Elasticsearch on all nodes
@molly_struve
Upgrade Steps
22
1
2
3
4
Shutdown the cluster
Upgrade Elasticsearch on all nodes
Deploy Elasticsearch 5 code changes
@molly_struve
Upgrade Steps
23
1
2
3
4
Shutdown the cluster
Upgrade Elasticsearch on all nodes
Deploy Elasticsearch 5 code changes
Start Elasticsearch on all nodes
šŸ‘
@molly_struve
24
Elasticsearch 5.x Cluster
cpu
load
cpu
load
cpu
load
cpu
load
cpu
load
cpu
load
@molly_struve
25
Elasticsearch 5.x Cluster
cpu
load
cpu
load
cpu
load
cpu
load
cpu
load
cpu
load
😬 😬 😬
@molly_struve
26
@molly_struve
27
x
@molly_struve
28
Elasticsearch 5.x Cluster
☠
☠ ☠ ☠
☠
@molly_struve
29
Elasticsearch 5.x Cluster
@molly_struve
30
Elasticsearch 5.x Cluster
cpu
load
cpu
load
cpu
load
cpu
load
cpu
load
cpu
load
@molly_struve
31
Elasticsearch 5.x Cluster
☠
☠ ☠ ☠
☠
@molly_struve
32
Elasticsearch 5.x Cluster
☠
@molly_struve
33
Debugging Mode
@molly_struve
34
@molly_struve
35
Elasticsearch 5 upgrade followed by cluster crash
@molly_struve
36
Elasticsearch 5 cluster instability
@molly_struve
37
Why does Elasticsearch 5 suck so much?
@molly_struve
38
@molly_struve
The Story
39
@molly_struve
40
@molly_struve
41
x x
@molly_struve
42
@molly_struve
43
😩 15+ hours 😩
@molly_struve
44
@molly_struve
45
@molly_struve
46
Rollback
@molly_struve
47
@molly_struve
48
Rollback
2.x5.x
@molly_struve
49
Rollback
5 days
2.x5.x
@molly_struve
50
x x x x
x x
@molly_struve
51
@molly_struve
52
@molly_struve
53
šŸ™Œ
@molly_struve
54
Workaround Deployed
@molly_struve
55
Workaround Deployed
@molly_struve
56
@molly_struve
Lessons Learned
57
@molly_struve
58
1
2
3
4
Have a Rollback Plan
5
6
Lessons Learned
@molly_struve
Rollback Plan
Can you rollback the software inline?
59
@molly_struve
Rollback Plan
Can you rollback the software inline?
60
How long and hard will a rollback be?
@molly_struve
Rollback Plan
Can you rollback the software inline?
61
How long and hard will a rollback be?
Worst case scenario the shit out of the upgrade
@molly_struve
62
1
2
3
4
Have a Rollback Plan
5
Do Performance Testing
6
Lessons Learned
@molly_struve
63
The last upgrade was great, this one
will be too!
@molly_struve
64
The last upgrade was great, this one
will be too!
@molly_struve
65
Performance Test
@molly_struve
66
Performance Test
@molly_struve
67
1
2
3
4
Have a Rollback Plan
5
Do Performance Testing
Don't Ignore Small Warning Signs
6
Lessons Learned
@molly_struve
68
Node
Local
Elasticsearch
☠
@molly_struve
69
Don’t ignore small warning
signs
@molly_struve
70
@molly_struve
71
1
2
3
4
Have a Rollback Plan
5
Do Performance Testing
Don't Ignore Small Warning Signs
6
Lessons Learned
@molly_struve
72
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
6
Lessons Learned
@molly_struve
73
Community
@molly_struve
74
@molly_struve
75
Don’t Wait
@molly_struve
76
Ask!
@molly_struve
77
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
@molly_struve
78
Engineers
@molly_struve
79
Vice President of
Engineering
@molly_struve
80
Vice President of
Engineering
@molly_struve
81
Vice President of
Engineering
šŸ›”
@molly_struve
82
Trust
@molly_struve
83
Fail Forward
@molly_struve
84
VP’s
@molly_struve
85
VP’s
Managers
@molly_struve
86
VP’s
Managers
C Suite Execs
@molly_struve
87
@molly_struve
88
šŸ›”
@molly_struve
89
Trust
@molly_struve
90
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
Your Team Matters
@molly_struve
91
Developer/Engineer
šŸ–„
@molly_struve
92
Developer/Engineer
šŸ–„ 😊
@molly_struve
93
😊😊😊
@molly_struve
94
😱😳😩
@molly_struve
95
😱😳😩
😐😟😬
šŸ˜­šŸ˜”šŸ¤”
@molly_struve
96
Character is
everything
@molly_struve
97
Character is
everything
@molly_struve
98
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
Your Team Matters
@molly_struve
99
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
Your Team Matters
@molly_struve
100
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
Your Team Matters
@molly_struve
101
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
Your Team Matters
@molly_struve
102
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
Your Team Matters
@molly_struve
103
Elasticsearch Outage
2017
@molly_struve
104
Embrace Your
Mistakes
@molly_struve
105
Embrace Your
Mistakes
@molly_struve
106
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
Your Team Matters
@molly_struve
107
Embrace it
@molly_struve
108
Learn from it
Embrace it
@molly_struve
109
Learn from it
Share it with others
Embrace it
@molly_struve
110
Questions?

More Related Content

PPTX
Toxic Teams and their Antidotes
PPT
Bh ffm13 exhppt_ch05
PPTX
Seoul Test Conference - Agile in Europe, The way its done.
PDF
Creating a Scalable Monitoring System That Everyone Will Love ADDO
PDF
Creating a Scalable Monitoring System That Everyone Will Love (Velocity Conf)
PDF
Kill All Passwords
PPTX
Best Presentaiton on any topics Slide Modal
PDF
Building a Scalable Monitoring System
Toxic Teams and their Antidotes
Bh ffm13 exhppt_ch05
Seoul Test Conference - Agile in Europe, The way its done.
Creating a Scalable Monitoring System That Everyone Will Love ADDO
Creating a Scalable Monitoring System That Everyone Will Love (Velocity Conf)
Kill All Passwords
Best Presentaiton on any topics Slide Modal
Building a Scalable Monitoring System

More from Molly Struve (10)

PDF
LeadDev NYC 2022: Calling Out a Terrible On-call System
PDF
Talk Horsey to Me
PDF
Eight Timezones, One Cohesive Team
PDF
All Day DevOps: Calling Out A Terrible On-Call System
PDF
Talk Horsey To Me
PDF
Cache is King: RubyConf Columbia
PDF
Cache is King - RailsConf 2019
PDF
Cache is King - RubyHACK 2019
PDF
Cache is King: Get the Most Bang for Your Buck From Ruby
PDF
Taking Elasticsearch From 0 to 88mph
LeadDev NYC 2022: Calling Out a Terrible On-call System
Talk Horsey to Me
Eight Timezones, One Cohesive Team
All Day DevOps: Calling Out A Terrible On-Call System
Talk Horsey To Me
Cache is King: RubyConf Columbia
Cache is King - RailsConf 2019
Cache is King - RubyHACK 2019
Cache is King: Get the Most Bang for Your Buck From Ruby
Taking Elasticsearch From 0 to 88mph
Ad

Recently uploaded (20)

PPTX
Internet of Things (IOT) - A guide to understanding
PPTX
FINAL REVIEW FOR COPD DIANOSIS FOR PULMONARY DISEASE.pptx
DOCX
ASol_English-Language-Literature-Set-1-27-02-2023-converted.docx
PDF
III.4.1.2_The_Space_Environment.p pdffdf
PPTX
Fundamentals of safety and accident prevention -final (1).pptx
PPTX
M Tech Sem 1 Civil Engineering Environmental Sciences.pptx
PDF
SM_6th-Sem__Cse_Internet-of-Things.pdf IOT
PDF
Automation-in-Manufacturing-Chapter-Introduction.pdf
PPTX
Foundation to blockchain - A guide to Blockchain Tech
PPTX
Current and future trends in Computer Vision.pptx
PPTX
Artificial Intelligence
PPTX
CARTOGRAPHY AND GEOINFORMATION VISUALIZATION chapter1 NPTE (2).pptx
PDF
Operating System & Kernel Study Guide-1 - converted.pdf
PPTX
CYBER-CRIMES AND SECURITY A guide to understanding
PPTX
Geodesy 1.pptx...............................................
PDF
R24 SURVEYING LAB MANUAL for civil enggi
PDF
BMEC211 - INTRODUCTION TO MECHATRONICS-1.pdf
PPTX
UNIT-1 - COAL BASED THERMAL POWER PLANTS
PDF
composite construction of structures.pdf
PDF
keyrequirementskkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk
Internet of Things (IOT) - A guide to understanding
FINAL REVIEW FOR COPD DIANOSIS FOR PULMONARY DISEASE.pptx
ASol_English-Language-Literature-Set-1-27-02-2023-converted.docx
III.4.1.2_The_Space_Environment.p pdffdf
Fundamentals of safety and accident prevention -final (1).pptx
M Tech Sem 1 Civil Engineering Environmental Sciences.pptx
SM_6th-Sem__Cse_Internet-of-Things.pdf IOT
Automation-in-Manufacturing-Chapter-Introduction.pdf
Foundation to blockchain - A guide to Blockchain Tech
Current and future trends in Computer Vision.pptx
Artificial Intelligence
CARTOGRAPHY AND GEOINFORMATION VISUALIZATION chapter1 NPTE (2).pptx
Operating System & Kernel Study Guide-1 - converted.pdf
CYBER-CRIMES AND SECURITY A guide to understanding
Geodesy 1.pptx...............................................
R24 SURVEYING LAB MANUAL for civil enggi
BMEC211 - INTRODUCTION TO MECHATRONICS-1.pdf
UNIT-1 - COAL BASED THERMAL POWER PLANTS
composite construction of structures.pdf
keyrequirementskkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk
Ad

Elasticsearch 5 and Bust (RubyConf 2019)