SACon 2019 - Surviving in a Microservices Environment

Surviving In A Microservices
Environment
Steve Pember
Principal Engineer @ Toast
Software Architecture Conf, San Jose, 2019
@svpember

@svpember
Microservice Blog Posts & Presentations

@svpember
Why Choose Microservices?
• Reduce Coupling!
• Team Autonomy
• Forced Boundaries between Functional Areas
• Right Tool for the Job
• Continuous Delivery
• Smaller codebases are easier to reason about
• Easy to replace old services
• Efﬁcient Scaling
• Can move quickly (once you’re up and running)

Please let me know if I missed
anything

@svpember
Topics
• Is it time for Microservices?
• Infrastructure / Operations
• Architecture
• Developer Productivity / Happiness
• People Communication
• Breaking off from the Monolith
• Random

@svpember
Topics

@svpember
Is it time for Microservices?
• … is it?

Not before 8 Engineers.
After You have known
Deﬁned Functional Areas.

You may not be ready, and
that’s OK.

@svpember
Topics

One Microservice is Easy
One Microservice Platform is
HARD

@svpember
Infrastructure / Operations
• Do we have Centralized Logging?

Centralized Log Aggregation is
Critical to Success

@svpember
• What do we use for Metrics?

Your Infrastructure needs to
Scale along with your Platform

–Johnny Appleseed
“Type a quote here.”

@svpember
• How are our builds performed?

@svpember
• When are our builds performed?

Frequent merging provides fast
feedback. Faster Iterations.

CI should be run on each ‘main’
branch and PR commit

CI should produce some
correct Artifact

@svpember
• Can our CI keep up with the the team’s build demand?

@svpember
• Do we have any coding conventions or quality gates?

@svpember
• How are our services deployed?

Continuous Deployment is your
Ultimate Goal

A Certain level of
Operational-Readiness is
required

@svpember
Service Mesh
• Likely don’t need in a small org with few services / Essential as you grow
• Aids in resiliency, can reroute trafﬁc around failing nodes
• App Devs focus on building services vs wiring them up
• Sidecars can handle metric, log collection and auth between services
• Easy Canary Deploys

@svpember
• How are our services deployed?
• Do we have multiple environments? How are our environments
managed?

Goal: standup a new
environment with a
single command

@svpember
Topics
• Architecture

@svpember
Architecture
• Do we have an overall design vision?

@svpember
Architecture
• What technologies do we use? How much freedom do we have in
choosing them?

Be wary of new adopting new
technology

@svpember
Architecture
• What technologies do we use? How much freedom do we have in choosing
them?
• How do we share code between services?

Shared Code can increase
coupling and decrease
Autonomy

@svpember
Shared code?
• Internal Commons (e.g. a custom String Utils) - OK!
• Styling, CSS - Go for it!
• Business Logic - You’re gonna have a bad time
• Domain Objects - Never!

@svpember
Architecture
them?
• How do we test an individual service?

Integration tests are the best tests

@svpember
Architecture
them?
• How do we test the platform as a whole?

@svpember
End-to-End tests are Difﬁcult
1. Reset ﬁxture / default state data on each service’s datastore
2. Execute test by performing some user action at the edge
3. Wait for test to complete
4. Make assertions on each service (e.g. look out for unknown side effects)
5. Repeat

@svpember
Architecture
them?
• Overall architectural style for services?

Are there common, repeatable,
patterns?

Does everyone in the org
know?

Asynchronous, Event Driven
Communications
+
Domain Driven Design

Services that communicate
Synchronously are a
Distributed Monolith

Decoupled Services allow for
team Autonomy

Decoupled Services scale
easily and efﬁciently

–Johnny Appleseed
“Type a quote here.”
Source: https://dzone.com/articles/distributed-sagas-for-microservices

@svpember
Some DDD concepts
• Ubiquitous Language
• Entities
• Aggregates
• Bounded Contexts
• No Direct communications across Boundaries, communicate through some
single point of contact

One Last Thing: Always have a
way to recover

@svpember
Architecture
them?
• Overall architectural style for services?
• When do we decide to create new services?

@svpember
New Service?
• Don’t make services ‘just because’
• Don’t make CRUD services
• Look for places where Eventual Consistency is acceptable
• A service should be thought as wrapping some functional business logic, not
the domain objects inside of it
• When new functionality is proposed: does it fall within the responsibilities of an
existing service?

# of services < # of developers

@svpember
Architecture
them?
• Overall architectural style for a service?
• When do we decide to create new services?
• How do we bootstrap a new service?

An Event Log makes this trivial

@svpember
Topics
• Architecture

@svpember
Developer Productivity / Happiness
• Do we have a service template?

git clone git@github.com:
your_org/services-template.git

@svpember
• What does it take to run our services?

What’s our Developer
experience like?

Running the Monolith on your
laptop will eventually become
problematic

Running a single Microservice
is easy.
Running all of them is
impossible.

@svpember
Developing against other Services
1. Every team publishes a mock service along with the real service
2. Only run services you need at the time
3. On-demand Environments
4. Keep buying bigger laptops

@svpember
• Do our Engineers know how our services are deployed?

@svpember
• Do our Engineers know how our services are deployed?
• How do our Engineers know about other Services?

@svpember
Topics
• Architecture

People are the reason for
Microservices

@svpember
People Communication
• How large are our teams?

Team larger than 4?
Break them up

@svpember
• How are they structured?

@svpember
• Are there clear functional boundaries between teams?

@svpember
• Do we encourage cross-team socializing?

@svpember
• Do we have clear methods for engineers to share & publicize services
they’ve built?

@svpember
• Do we have clear methods for engineers to share & publicize services they’ve
built?
• Do we add more process if something goes wrong?

@svpember
• Do we have clear methods for engineers to share & publicize services they’ve
built?
• Do we add more process if something goes wrong?
• Do we have company ‘Core Values’? Do they include non-technical
skills?

Empathy should be your #1
Core Value

@svpember
Topics
• Architecture
• Breaking off from the Monolith

@svpember
Breaking off from the Monolith
• Do we have a process / guideline for breaking functionality out of the
monolith?

@svpember
• Do we have a process / guideline for breaking functionality out of the monolith?
• Do we have a process / guideline for new services to interact w/ the
monolith?

@svpember
• Do we have a process / guideline for new services to interact w/ the monolith?
• So… how much functionality have we actually split off?

@svpember
• Do we have a process / guideline for new services to interact w/ the monolith?
• So… how much functionality have we actually split off?
• How’s the monolith doing?

All of your dependent systems
need to scale to meet the
demands of your services

@svpember
Slicing code from the Monolith
• Not that straightforward!
• Create programmatic boundaries around the departing code
• Measure trafﬁc and operational load across those boundaries
• Any dependencies on original system?
• Has the monolith accounted for the backpressure from the new service?
• Roll out slowly
• How are the metrics looking?

The First Service will be HARD

@svpember
Random Advice
• API & Message versioning is a real issue

@svpember
Random Advice
• Always have more than one cluster

@svpember
Random Advice
• Don’t get cute with the naming of services

@svpember
Random Advice
• New Feature? Walk backwards from the End User UI

@svpember
Random Advice
• Release when a feature is ready, don’t be afraid of bugs

@svpember
Random Advice
• Release when a feature is ready, don’t be afraid of bugs
• But don’t release Friday afternoon.

Microservices are hard because
of Infrastructure and People

@svpember
Images
• Factory Workers: https://www.stallardkaneassociates.com/hr-employment-law/factory-workers/
• Bad Demolition: https://media.giphy.com/media/zf3skxc4BFBqo/giphy.mp4
• Tom Hanks, Desert: http://www.trbimg.com/img-57ba8bc0/turbine/sdut-lost-in-the-desert-waiting-for-the-king-2016apr21
• Say Anything / John Cusak: http://www.glamour.com/story/happy-birthday-john-cusack-and
• Wilderness Survival Guide: https://www.kobo.com/us/en/ebook/the-wilderness-survival-guide-1
• XKCD: Code Review: https://xkcd.com/1513/
• Monolithic vs Microservices: @alvaro_sanchez
• Drago (break you): https://giphy.com/gifs/GWD5nSpiHxs3K
• Michael Jackson, Bad: https://www.imdb.com/title/tt0124288/mediaviewer/rm4118942720
• Rocky Stairs: http://pixgood.com/rocky-stairs.html
• Assembly Line: http://www.solidsmack.com/culture/humans-need-apply-new-short-film-explores-future-robots-manufacturing-automation/
• Site Reliability Engineer Shirt: https://www.sunfrog.com/102539321-160396457.html
• Watson, “Wait”: https://giphy.com/gifs/B8qMV3f7PH7nq
• Service Mesh: https://www.redhat.com/en/topics/microservices/what-is-a-service-mesh
• Trick question: https://media.giphy.com/media/ZcaSXPd7di8O4ZfsnH/giphy.mp4
• Dumpster fire: https://sentinelksmo.org/lawmakers-spark-dumpster-fire-with-tax-school-funding-bill/shutterstock_49161094/
• Work meeting: https://pxhere.com/en/photo/910324
• People in Conversation: https://preply.com/en/blog/2018/07/06/5-main-principles-of-small-talk/
• Indiana Jones: https://giphy.com/explore/indiana-jones-and-the-raiders-of-the-lost-ark
• Simpsons Mob: https://www.bobleesays.com/2017/06/18/halt-angry-fan-mob/
• Picard: https://fanfilmfactor.com/2019/03/07/the-right-captain-at-the-right-time-why-we-like-pike-discovery-editorial/
• Megaphone: https://imgflip.com/i/332n0j

@svpember
Links
• Eric Evans: DDD and Microservices: https://www.youtube.com/watch?
v=yPvef9R3k-M
•

SACon 2019 - Surviving in a Microservices Environment

More Related Content

Similar to SACon 2019 - Surviving in a Microservices Environment (20)

More from Steve Pember (20)

Recently uploaded (20)

SACon 2019 - Surviving in a Microservices Environment