10 tips for failing at microservices @ DevExperience 2018

@koenighotze#DevExperience18 #10TipsMicroservices
10 (or so) tips for failing
at microservices - badly

David Schmitz
Senacor Technologies
@koenighotze

Disclaimer(s)

Anecdote
IS NOT
Evidence

Your boss wants to modernise
You don’t want to modernise
Things to do to sabotage your project

…your Bossreads
about microservices?

Extreme agile
cloud native digital
microservices!
Disrupt or be
disrupted!
Sounds awesome.
And cloud and agile
equal good?!?
Industry 4.0!
FTW!

“IT People!
Make it happen!
Build me some
microservices”

What the hell is a microservice and why should I care?

Microservices
Single bounded context
UI and services
Independent
Self-contained
Local storage
Share nothing
User Interface
Services
Persistence
Operations

TLDR;User Interface
Services
Persistence
Operations
Too lazy to actually read all
this stuff…looks boring and I
want to look at funny cat
pics.

Do you prefer change…
https://flic.kr/p/bVPzUE

@koenighotze#DevExperience18 #10TipsMicroservices https://flic.kr/p/bajCVX
…or rest?

Sabotage countdown

Go full scale polyglot

Where is Waldo?

William-Wallace-Tribute-team!
Groundhog code
Maintenance field trips
Synergies: polyglot UI with beta status
Find new friends on Stackoverflow
Governance and moderation are for
cowards

Still successful?

Semantic Versioning
Major.Minor.BugfixMajor.Minor.BugfixMajor.Minor.BugfixMajor.Minor.Bugfix
Semantic Versioning

$ npm ls | grep @0

ui-broker@0.1.0
│ ├─┬ fbjs@0.8.8
│ │ │ ├─┬ encoding@0.1.12
│ │ │ │ └── iconv-lite@0.4.15
│ │ └── ua-parser-js@0.7.12
├─┬ react-scripts@0.8.5
│ │ ├── normalize-range@0.1.2
│ │ │ │ └── json5@0.5.1
│ │ │ └── …

Polyglot?

Polyglot?
thx @Codepitbull

The data monolith
(c) Disney Corporation

Credit Account Payment
Expensive
enterprise SQL
Datastore

Account name must be
VARCHAR(200) instead of
VARCHAR(50)

Credit Account

Sharing is a Good ThingTM
Avoid table or schema ownership
Ubiquitous domain modelling paralysis
Keep people awake at night with
insidious dependencies

The event monolith

Eventsourcing
marketing
pitch

Customer
Homeaddress
Updated
Credit
acquired
New
Customer
Created
…
Direction of time
Customer Credit
Event Store

Audit trail - immutable history for free
Distributed data management
Decoupled microservices with CQRS
…and much more!

‘Das schöne
Österreich’ will no longer be
accepted as a country code

Country:
“AU”
Country:
“Das schöne
Österreich”
…
Direction of time
Customer
Event Store
Credit

How to deal with shared code?

- C P*Windows version coming soon

Country:
“AU”
Country:
“Das schöne
Österreich”
…
Direction of time
Customer Credit
Event Store

2 months later…

Customer Credit Reporting Accounting Profile
Loan Trading Exchange Broker Purchase

WET is the new DRY
Never question the design
All systems should be eventsourced
Eventual consistency = “Maybe consistent”
Extra tip: Use Kafka for eventsourcing

The home grown monolith

– Carl Sagan (slightly paraphrased)
“If you wish to
make
MICROSERVICES
from scratch, you
must first create a
FRAMEWORK”

{
"type": "NewCustomerCreated",
"id": 1234,
"payload": {
}
}
CustomerAccount Reporting
ES V1 ES V1 ES V1

The customer system needs
special, non-numeric ids

Major.Minor.Bugfix

Customer V2
{
"id": "1234-5678",
"payload": {
...
}
}
ES V2

And at runtime???

Change the
world event

Customer V2Account Reporting
ES V1 ES V1ES V2
{
"id": "1234-5678",
"payload": {
...
}
}

Customer V2Account V2 Reporting V2
ES V2 ES V2ES V2

Customer
Account
Reporting
From independent releases

Customer
Account
Reporting
to a release monolith
Big bang

Day rates!

Dependencies as social tools
Job insurance via invasive frameworks
Building frameworks is fun!
Ideas: collections, string-utils, logging,
ORM, UI framework

No time to create a
framework, but still
want to leave your
mark on the world?

10/10/1999:21:15:05 +0500
Sep 19 16:27:49
2007-08-31 19:22:21.469 ADT
060516 22:38:54
Fri Sep 09 10:42:29.902022 2011

The meat cloud

Bonus hint:
optimise your revenue!

W K
P T

Who KNOWS how
to do it?K
P T
Who WANTS it
to be done?
Who has the
required
PRIVILEGES?
Who has TIME
to actually
do it?

Delivery
pipeline
Production

And the rest of the
world?

Delivery pipeline
Production
Handcrafted
with love
Soulless
automation

Cattle
not
pets?

Ask yourself:
Whom do YOU love more?

Mr. Wiskers Walking steak

W
Quickly…release a new
Customer microservice
K
P T

Enter the DevOps TeamTM

Ops communication?
P T
W K
P T

Tickets!
P T
W K
P T

P T
KW

P T
W K
P T

…a few minutes later
P T
W K
P T

Infrastructure is expensive…be proud to
take care of it
Microsoft Word is some kind of automation
Play the “we are not Google” card
Combine with polyglot ops for maximum
chaos

The distributed monolith

Customer AccountCredit
Auth

Let’s offer a discount to
new users

Customer AccountCredit
Auth
No problem!
What the..?!?

Best of Both Worlds
complexity
+ rigidity
+ fragility

The network is reliable
Dependencies indicate good design, like in
dependency injection
A system stable as concrete sounds good
Synchronous dependencies are easy
What is half a system good for, anyway? So
avoid Circuit breakers

The SPA monolith

Customer REST Credit RESTAccount REST
Sophisticated portal
frontend

Add a mandatory
social-number field to the
customer

But not that mandatory. E.g. if
the customer is not committed, yet

Customer Domain Credit DomainAccount Domain
Comprehensive UI Domain
Model

Where should the logic go?

Advantage?

Customer REST Credit RESTAccount REST

UI-driven monolith
Customer Domain Credit DomainAccount Domain
Sophisticated portal
frontend

Microservices are REST only (hint: service!)
Avoid techniques like UI composition or
self-contained-systems
Keep the testers busy with side effects
Single point of failure - blame the script
kids working on the UI

The decision monolith

“But we need
Java 7…it is
2018 after all!”

Beware developer
madness

Agile
Rx
Microservice
Actors Augmented realityCloud
AWS
WebRTC IoT
Blockchain

Architecture & Design
Sieve of Sanity

JEE
WebSphere
Agile
Rx
Microservice
Actors Augmented realityCloud
AWS
WebRTC IoT
Blockchain

Frustrate your coworkers
Irritate motivated colleagues
If you are older than 30…play the “This is
_real_ enterprise, kid” card
Ivory towers are beautiful

The business
monolith

So, don’t bother
explaining
anything!
Business
people
cannot
understand
complex
any
architecture

… … …
Product Toolset Something

Let’s force the user to
always enter a valid address

No, the customer
should be able to enter partial
addresses

?????

Requirements Ping Pong

Change management

Requirements management

Bonus:
works with ops, too

… … …
What the…?
Whatever…just
deploy everything

Development
Ops
Business
Waterscrumfall

Avoid X-functional teams
Slow means more time for Twitter!
The servant of many kings is a free man
Conway’s law works both ways

The staffing monolith

It is easier to search for React
developers than for general
craftswomen

Frontend
Development
Service
Development
DevOps Team
Test
Business

But…why?

Frequent Flyer Program
Rail-Bonus Program
Hotel Credit Points

Dysfunctional setup in
3 easy steps

Step 1
How to do X-team
communication?

Magic
Speciﬁcation
Success
Template

Step 2
Protect the teams

Frontend Microservice
Service Microservice
Persistence Stuff
Team A
Team B
Team C

Step 3
Establish bureaucracy

The frontend team
The service team
The database guys
“Product
Owner”
“Product
Owner”
“Product
Owner”

But…what if this
works?

Enter the fakeholder!

shadow product owner
proxy product owner

“Product
Owner”
“UX
Product
Owner”
“Back
Product
Owner”
“Business
Owner”
“Product
Owner”
“Product
Owner”
Doesn’t
care
Cannot
decide

Even
sounds
“agile”!

Technical benefits?

Nice Digital Click UI
Boring HTTP Things
Persistence Stuff
Shared
stability
Shared
release
Convoluted
domains

Programmers do not like other people
Play “find the domain”
“War rooms” sound really important
Avoid real ownership

And: In case of bugs…

…it’s always
the other
team’s fault!

The prescriptive cloud

Agility
Flexibility
Scale as you grow
Use what you need

One Size Fits All
Cloud
The

Rails
TODO app

200 users now…
…but we might become the next Google

Revenge of the CloudOps TeamTM

Docker
Rails
TODO app

Node
Pod
Docker
Rails
TODO app
EtcdService
FEK
Volume
Pod
Docker
Rails
TODO app

VPC Security groupCloudwatch
ELB IAMEBS
EC2 EC2 EC2 EC2
EC2 EC2 EC2 EC2
What I
actually
care about

[610420.800771] kube-state-metr invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=-998
[610420.805428] kube-state-metr cpuset=32c394ad4709430cddb71194ea406d598b82159c2715d42b64927ac5e67f0228 mems_allowed=0
[610420.810504] CPU: 3 PID: 9403 Comm: kube-state-metr Tainted: G E 4.4.65-k8s #1
[610420.813712] Hardware name: Xen HVM domU, BIOS 4.2.amazon 11/11/2016
[610420.813712] 0000000000000286 00000000c5185d6f ffffffff812f67b5 ffff8803e7163e20
[610420.813712] ffff8803e71c8400 ffffffff811d8855 ffffffff81826173 ffff8803e71c8400
[610420.813712] ffffffff81a6b740 0000000000000206 0000000000000002 ffff880308bbf438
[610420.813712] Call Trace:
[610420.813712] [<ffffffff812f67b5>] ? dump_stack+0x5c/0x77
[610420.813712] [<ffffffff811d8855>] ? dump_header+0x62/0x1d7
[610420.813712] [<ffffffff8116ded1>] ? oom_kill_process+0x211/0x3d0
[610420.813712] [<ffffffff811d0f4f>] ? mem_cgroup_iter+0x1cf/0x360
[610420.813712] [<ffffffff811d2de3>] ? mem_cgroup_out_of_memory+0x283/0x2c0
[610420.813712] [<ffffffff811d3abd>] ? mem_cgroup_oom_synchronize+0x32d/0x340
[610420.813712] [<ffffffff811cf170>] ? mem_cgroup_begin_page_stat+0x90/0x90
[610420.813712] [<ffffffff8116e5b4>] ? pagefault_out_of_memory+0x44/0xc0
[610420.813712] [<ffffffff815a65b8>] ? page_fault+0x28/0x30
[610426.700118] Task in /kubepods/pod401c0ec6-97d9-11e7-adb9-021e5eba15b7/32c394ad4709430cddb71194ea406d598b82159
[610426.711591] memory: usage 16384kB, limit 16384kB, failcnt 629216
[610426.715097] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
[610426.719063] kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
[610426.722823] Memory cgroup stats for /kubepods/pod401c0ec6-97d9-11e7-adb9-021e5eba15b7: cache:0KB rss:0KB rss_huge:0KB
[610426.749544] Memory cgroup stats for /kubepods/pod401c0ec6-97d9-11e7-adb9-021e5eba15b7/32c394ad4709430cddb71194ea406d598b82159c2715d42b64927ac5e67f0228:
[610426.765166] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[610426.769304] [ 7288] 0 7288 257 1 4 2 0 -998 pause
[610426.773553] [20340] 0 20340 11395 7113 27 5 0 -998 kube-state-metr
[610426.778753] Memory cgroup out of memory: Kill process 20340 (kube-state-metr) score 769 or sacrifice child
…
…
Approx. 3 billion lines of garbage logs
to analyse for fun

[610426.778753] Memory cgroup out of memory: Kill process
20340 (kube-state-metr) score 769 or sacrifice child

PolyWTF works for ops

Be proud of complexity, we are
engineers for a reason
Ops tooling puts JavaScript to
shame
If Twitter runs on it, so should you
Combine with Meat cloud

Bonus: speedboats and tankers
https://commons.wikimedia.org/wiki/User:Arria_Belli/Salut

“We need 24x7 availability!”

“…but the host is down 4h per week?”

The Bi-modal idea
System of RecordSystems of Engagement

Bi-modal architecture

Wall of innovation

Awesome speedboats
Speed
boat
Speed
boat
Speed
boat
Dirty old
legacy system

Ignore reality
Speed
boat
Speed
boat
Speed
boat
Things you need
written by people
you don’t know
and don’t like

Speedboat
Speedboat
Dirty backend
Big bang
Speedboat
Speedboat
Fast release with speedboats?
Speedboat
Speedboat
Speedboat
Speedboat
1.3.18 1.4.18

Speedboat
Speedboat
Big bang
Speedboat
Speedboat
Hold my beer
Speedboat
Speedboat
Speedboat
Speedboat
1.3.18 1.4.18
Dirty backend
everybody
depends on
1.1.35
Big bang

Result…
Dirty old
legacy system
New
Landscape

Very
New
Landscape
New
Landscape
…5 years later
Dirty old
legacy system

Very
New
Landscape
…5 years later
Dirty old
legacy system
New
Landscape
Techstack hipness gradient
People deprecation gradient

Refactoring is hard, just do not do it
Maintain yet another system
Avoid internal knowledge and build
external skills
Distinguish between cool devs and
not-so-cool ones…openly
Create a hostile environment of
suspicion and distrust

Wrapping up….

The key to failure is
the hidden monolith

Microservices are hard

Brownfield - The
perfect excuse

Never involve ops or
business

Embrace the monolith
both in mind and way
of working

Thank you!
Questions? Disagree? Let’s talk!
@Koenighotze

10 tips for failing at microservices @ DevExperience 2018

More Related Content

Similar to 10 tips for failing at microservices @ DevExperience 2018 (20)

More from David Schmitz (12)

Recently uploaded (20)

10 tips for failing at microservices @ DevExperience 2018