DevOps Patterns & Antipatterns for Continuous Software Updates @ NADOG April 2020

DevOps Patterns & Antipatterns
for Continuous Software Updates
“What can possibly go wrong?!”

@jbaruch #NADOG #LiquidSoftware http://jfrog.com/shownotes

“As everycompanybecome asoftware company,
Security vulnerabilities are the new oil spills”

Identify Fix Deploy

Identify
As fast as possible
Fix
As fast as possible
Deploy
As fast as possible

This is not
a new idea!
XP: short feedback
Scrum: reducing cycle time
to absolute minimum
TPS: Decide as late as
possible and Deliver as fast
as possible
Kanban: Incremental
change

@jbaruch #dockercon jfrog.com/shownotes@ErinMeyerINSEAD’s “Culture Map”
🎩

shownotes
http://jfrog.com/shownotes
Slides
Video
Links
Comments, Ratings
Raffle

Do we want
it?
Let’s
update!
Are there
any high
risks?
How about no
Do we trust
the
update?
Update
available
No
Yes
Yes
Yes
NoNo

complexity
@jbaruch @jfrog #LiquidSoftware www.liquidsoftware.com
IoT
Serverless
Docker
Infrastructure as Code
Continuous Delivery
Continuous Integration
Agile
Microservices
2000
Today

The problem is not the code, it’s the data. Big data.

Do we want
it?
Let’s
update!
Are there
any high
risks?
Can we
verify the
update?
How about no
Do we trust
the
update?
Update
available
No
Yes
Yes
Yes
Yes
NoNoNo
Time
consuming
verification

Features
that we
want
Acceptance
tests costs

Continuous
updates pattern:
Local rollback
Problem: update went
catastrophically wrong and
an over the-air patch can’t
reach the device
Solution: Have a previous
version saved on the device
prior to update. Rollback in
case problem occurred

Continuous
updates pattern:
OTA software
updates
Problem: physical recalls
are costly. Extremely costly.
Also, you can’t force an
upgrade.
Solution: Implement over
the air software updates,
preferably, continuous
updates.

continuous OTA updates are like normal OTA updates,
but better

Continuous
updates pattern:
continuous
updates
Problem: In batch updates
important features wait for
non-important features.
Solution: Implement
continuous updates.

You thought your problems are hard?
Things under your control Server-side Updates IoT (Mobile, Automotive,
Edge) Updates
The availability of the target
✓ ✕
The state of the target
✓ ✕
The version on the target
✓ ✕
The access to the target
✓ ✕

KNIGHT-MARE
New system reused old APIs
1 out of 8 servers was not
updated
New clients sent requests to
machine contained old code
Engineers undeployed
working code from updated
servers, increasing the load on
the not-updated server
No monitoring, no alerting, no
debugging

Continuous
updates pattern:
Automated
deployment
Problem: People suck at
repetitive tasks.
Solution: Automate
everything.

Continuous
updates pattern:
frequent updates
Problem: Seldom
deployments generate
anxiety and stress, leading
to errors.
Solution: Update
frequently to develop skill
and habit.

Continuous
updates pattern:
state awareness
Problem: Target state can
affect the update process
and the behavior of the
system after the update.
Solution: Know and
consider target state when
updating.
Reverting might require
revering the state.

Cloud-dark
New rules are deployed
frequently to battle attacks
Deployment of a single
misconfigured rule
Included regex to spike
CPU to 100%
“Affected region: Earth”

Continuous
updates pattern:
Progressive
Delivery
Problem: Releasing a bug
affects ALL the users.
Solution: Release to a
small number of users first
effectively reducing the
blast radius and observe.
If a problem occurs, stop
the release, revert or
update the affected users.

Continuous
updates pattern:
observability
Problem: Some problems
are hard to trace relying on
user feedback only
Solution: Implement
tracing, monitoring and
logging

Continuous
updates pattern:
Rollbacks
Problem: Fixes might take
time, users suffer in a
meanwhile
Solution: Implement
rollback, the ability to
deploy a previous version
without delay

Continuous
updates pattern:
feature flags
Problem: Rollbacks are not
always supported by the
deployment target platform
Solution: Embed 2 versions
of the features in the app
itself and trigger them with
API calls

Do we want
it?
Let’s update!
Are there
any high
risks?
Do we trust
the
update?
Update
available
No
Yes
Yes
Yes
Sure, why not?
(auto update)

Our goal is to transition from bulk and rare software updates to
extremely tiny and extremely frequent software updates;
so tiny and so frequent that they provide an illusion of
software flowing from development to the update target.
We call it the Liquid Software vision.
”

Corner cases?

Q&A and twitter ads
@jbaruch
#LiquidSoftware
#NADOG
https://liquidsoftware.com
https://jfrog.com/shownotes

DevOps Patterns & Antipatterns for Continuous Software Updates @ NADOG April 2020

More Related Content

What's hot (20)

Similar to DevOps Patterns & Antipatterns for Continuous Software Updates @ NADOG April 2020 (20)

More from Baruch Sadogursky (20)

Recently uploaded (20)

DevOps Patterns & Antipatterns for Continuous Software Updates @ NADOG April 2020