SlideShare a Scribd company logo
Building
      Scalable,
Highly Concurrent &
   Fault-Tolerant
      Systems:
  Lessons Learned

     Jonas Bonér
       CTO Typesafe
      Twitter: @jboner
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again    Lessons
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                     Learned
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again   through...
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again    Lessons
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                     Learned
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again   through...
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                      Agony
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again    Lessons
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                     Learned
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again   through...
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                      Agony
                                                                                                    and Pain
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again
I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                  I will never use distributed transactions again
                                                                                                    lots of
                                                                                                      Pain
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
I will never use distributed transactions again   I will never use distributed transactions again
Agenda
• It’s All Trade-offs
• Go Concurrent
• Go Reactive
• Go Fault-Tolerant
• Go Distributed
• Go Big
Building Scalable, Highly Concurrent & Fault Tolerant Systems -  Lessons Learned
It’s all
Trade-offs
Performance
     vs
 Scalability
Latency
     vs
Throughput
Availability
    vs
Consistency
Go Concurrent
Shared mutable state
Shared mutable state
Together with threads...
Shared mutable state
              Together with threads...


...leads to
Shared mutable state
              Together with threads...

              ...code that is totally INDETERMINISTIC
...leads to
Shared mutable state
              Together with threads...

              ...code that is totally INDETERMINISTIC
...leads to
              ...and the root of all EVIL
Shared mutable state
              Together with threads...

              ...code that is totally INDETERMINISTIC
...leads to
              ...and the root of all EVIL


Please, avoid it at all cost
Shared mutable state
                     B L E
                 T
               U ! A
     Together with threads...

            M e!!
        IM at EVIL
              ...code that is totally INDETERMINISTIC

      e t
...leads to


     s s      ...and the root of all

    U avoid it at all cost
Please,
The problem with locks
• Locks do not compose
• Locks break encapsulation
• Taking too few locks
• Taking too many locks
• Taking the wrong locks
• Taking locks in the wrong order
• Error recovery is hard
You deserve better tools

• Dataflow Concurrency
• Actors
• Software Transactional Memory (STM)
• Agents
Dataflow Concurrency
• Deterministic
• Declarative
• Data-driven
  • Threads are suspended until data is available
  • Lazy & On-demand
• No difference between:
 • Concurrent code
 • Sequential code
• Examples: Akka & GPars
Actors
•Share NOTHING
•Isolated lightweight event-based processes
•Each actor has a mailbox (message queue)
•Communicates through asynchronous and
 non-blocking message passing
•Location transparent (distributable)
•Examples: Akka & Erlang
STM
• See the memory as a transactional dataset
• Similar to a DB: begin, commit, rollback (ACI)
• Transactions are retried upon collision
• Rolls back the memory on abort
• Transactions can nest and compose
• Use STM instead of abusing your database
  with temporary storage of “scratch” data
• Examples: Haskell, Clojure & Scala
Agents
• Reactive memory cells (STM Ref)
• Send a update function to the Agent, which
  1. adds it to an (ordered) queue, to be
  2. applied to the Agent asynchronously
• Reads are “free”, just dereferences the Ref
• Cooperates with STM
• Examples: Clojure & Akka
If we could start all over...
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
3. Add Mutability selectively - only where needed
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
3. Add Mutability selectively - only where needed
  • Protected by Transactions (STM)
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
3. Add Mutability selectively - only where needed
  • Protected by Transactions (STM)
4. Finally - only if really needed
If we could start all over...
1. Start with a Deterministic, Declarative & Immutable core
  • Logic & Functional Programming
  • Dataflow
2. Add Indeterminism selectively - only where needed
  • Actor/Agent-based Programming
3. Add Mutability selectively - only where needed
  • Protected by Transactions (STM)
4. Finally - only if really needed
  • Add Monitors (Locks) and explicit Threads
Go Reactive
Never block
• ...unless you really have to
• Blocking kills scalability (and performance)
• Never sit on resources you don’t use
• Use non-blocking IO
• Be reactive
• How?
Go Async
  Design for reactive event-driven systems
1. Use asynchronous message passing
2. Use Iteratee-based IO
3. Use push not pull (or poll)
• Examples:
   • Akka or Erlang actors
   • Play’s reactive Iteratee IO
   • Node.js or JavaScript Promises
   • Server-Sent Events or WebSockets
   • Scala’s Futures library
Go Fault-Tolerant
Failure Recovery in Java/C/C# etc.
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
• To make things worse - errors do not
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that
  something have failed
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
• To make things worse - errors do not
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that
  something have failed
• This leads to DEFENSIVE programming with:
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
• To make things worse - errors do not
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that
  something have failed
• This leads to DEFENSIVE programming with:
  • Error handling TANGLED with business logic
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control
• If this thread blows up you are screwed
• So you need to do all explicit error handling
  WITHIN this single thread
• To make things worse - errors do not
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that
  something have failed
• This leads to DEFENSIVE programming with:
  • Error handling TANGLED with business logic
  • SCATTERED all over the code base
Failure Recovery in Java/C/C# etc.
• You are given a SINGLE thread of control


                d o
• If this thread blows up you are screwed

               n !!
• So you need to do all explicit error handling

              a !
            c r
  WITHIN this single thread

           e te
• To make things worse - errors do not

          W et
  propagate between threads so there is NO
  WAY OF EVEN FINDING OUT that

            b
  something have failed
• This leads to DEFENSIVE programming with:
  • Error handling TANGLED with business logic
  • SCATTERED all over the code base
Just

Let It Crash
Building Scalable, Highly Concurrent & Fault Tolerant Systems -  Lessons Learned
The right way
1. Isolated lightweight processes
2. Supervised processes
 • Each running process has a supervising process
 • Errors are sent to the supervisor (asynchronously)
 • Supervisor manages the failure
• Same semantics local as remote
• For example the Actor Model solves it nicely
Go Distributed
Performance
     vs
 Scalability
How do I know if I have a
 performance problem?
How do I know if I have a
 performance problem?

       If your system is
    slow for a single user
How do I know if I have a
  scalability problem?
How do I know if I have a
  scalability problem?

        If your system is
     fast for a single user
  but slow under heavy load
(Three) Misconceptions about
     Reliable Distributed Computing
                     - Werner Vogels

1. Transparency is the ultimate goal
2. Automatic object replication is desirable
3. All replicas are equal and deterministic

  Classic paper: A Note On Distributed Computing - Waldo et. al.
Fallacy 1
Transparent Distributed Computing
• Emulating Consistency and Shared
  Memory in a distributed environment
• Distributed Objects
 • “Sucks like an inverted hurricane” - Martin Fowler
• Distributed Transactions
 • ...don’t get me started...
Fallacy 2
                          RPC
• Emulating synchronous blocking method
  dispatch - across the network
• Ignores:
 • Latency
 • Partial failures
 • General scalability concerns, caching etc.
• “Convenience over Correctness” - Steve Vinoski
Instead
Instead
   Embrace the Network

    Use                                    it
                                    i th
                                   w
Asynchronous               don e

  Message             be
                  d
               an
   Passing
Guaranteed Delivery
  Delivery Semantics
   • No guarantees
   • At most once
   • At least once
   • Once and only once
It’s all lies.
It’s all lies.
The network is inherently unreliable
 and there is no such thing as 100%
        guaranteed delivery


       It’s all lies.
Guaranteed Delivery
Guaranteed Delivery
The question is what to guarantee
Guaranteed Delivery
  The question is what to guarantee
1. The message is - sent out on the network?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
3. The message is - put on the receiver’s queue?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
3. The message is - put on the receiver’s queue?
4. The message is - applied to the receiver?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
3. The message is - put on the receiver’s queue?
4. The message is - applied to the receiver?
5. The message is - starting to be processed by the receiver?
Guaranteed Delivery
   The question is what to guarantee
1. The message is - sent out on the network?
2. The message is - received by the receiver host’s NIC?
3. The message is - put on the receiver’s queue?
4. The message is - applied to the receiver?
5. The message is - starting to be processed by the receiver?
6. The message is - has completed processing by the receiver?
Ok, then what to do?
1. Start with 0 guarantees (0 additional cost)
2. Add the guarantees you need - one by one
Ok, then what to do?
1. Start with 0 guarantees (0 additional cost)
2. Add the guarantees you need - one by one

  Different USE-CASES
            Different GUARANTEES
                Different COSTS
Ok, then what to do?
    1. Start with 0 guarantees (0 additional cost)
    2. Add the guarantees you need - one by one

       Different USE-CASES
                  Different GUARANTEES
                       Different COSTS
 For each additional guarantee you add you will either:
• decrease performance, throughput or scalability
• increase latency
Just
Just


Use ACKing
Just


Use ACKing
and be done with it
Latency
     vs
Throughput
You should strive for
maximal throughput
          with
acceptable latency
Go Big
Go Big
 Data
Big Data
  Imperative OO programming doesn't cut it
• Object-Mathematics Impedance Mismatch
• We need functional processing, transformations etc.
• Examples: Spark, Crunch/Scrunch, Cascading, Cascalog,
  Scalding, Scala Parallel Collections
• Hadoop have been called the:
  • “Assembly language of MapReduce programming”
  • “EJB of our time”
Big Data
     Batch processing doesn't cut it
• Ala Hadoop
• We need real-time data processing
• Examples: Spark, Storm, S4 etc.
• Watch“Why Big Data Needs To Be Functional”
  by Dean Wampler
Go Big
 DB
When is
 a RDBMS
   not
good enough?
Scalingreads
  to a RDBMS
  is   hard
Scalingwrites
   to a RDBMS
is   impossible
Do we
really need
a RDBMS?
Do we
really need
a RDBMS?
Sometimes...
Do we
really need
a RDBMS?
Do we
      really need
      a RDBMS?
But many times we don’t
Atomic
Consistent
Isolated
Durable
Availability
    vs
Consistency
Brewer’s



CAP
theorem
You can only pick   2
     Consistency
     Availability
     Partition tolerance
At a given point in time
Centralized system
• In a centralized system (RDBMS etc.)
  we don’t have network partitions,
  e.g. P in CAP
• So you get both:

        Consistency
        Availability
Distributed system
• In a distributed (scalable) system
  we will have network partitions,
  e.g. P in CAP
• So you get to only pick one:

       Consistency
       Availability
Basically Available
Soft state
Eventually consistent
Think about your data
             Then think again
• When do you need ACID?
• When is Eventual Consistency a better fit?
• Different kinds of data has different needs
• You need full consistency less than you think
How fast is fast enough?
• Never guess: Measure, measure and measure
• Start by defining a baseline
 • Where are we now?
• Define what is “good enough” - i.e. SLAs
 • Where do we want to go?
 • When are we done?
• Beware of micro-benchmarks
...or, when can we go for a beer?
• Never guess: Measure, measure and measure
• Start by defining a baseline
 • Where are we now?
• Define what is “good enough” - i.e. SLAs
 • Where do we want to go?
 • When are we done?
• Beware of micro-benchmarks
To sum things up...
1. Maximizing a specific metric impacts others
  • Every strategic decision involves a trade-off
  • There's no "silver bullet"
2. Applying yesterday's best practices to the
   problems faced today will lead to:
  • Waste of resources
  • Performance and scalability bottlenecks
  • Unreliable systems
SO
GO
...now home and build yourself
            Scalable,
      Highly Concurrent &
         Fault-Tolerant
            Systems
Thank You
Email: jonas@typesafe.com
Web: typesafe.com
Twitter: @jboner

More Related Content

PDF
게임서비스를 위한 ElastiCache 활용 전략 :: 구승모 솔루션즈 아키텍트 :: Gaming on AWS 2016
PDF
Java null survival guide
PDF
開発速度が速い #とは(LayerX社内資料)
PDF
Jitney, Kafka at Airbnb
PDF
HCCJP 23回勉強会 5分で分かる APIゲートウェイと サービスメッシュの違い なぜAPIの管理とサービスメッシュは異なるユースケース を補完するパ...
PPTX
分散トレーシングAWS:X-Rayとの上手い付き合い方
PDF
MySQLアーキテクチャ図解講座
PDF
インメモリーデータグリッドの選択肢
게임서비스를 위한 ElastiCache 활용 전략 :: 구승모 솔루션즈 아키텍트 :: Gaming on AWS 2016
Java null survival guide
開発速度が速い #とは(LayerX社内資料)
Jitney, Kafka at Airbnb
HCCJP 23回勉強会 5分で分かる APIゲートウェイと サービスメッシュの違い なぜAPIの管理とサービスメッシュは異なるユースケース を補完するパ...
分散トレーシングAWS:X-Rayとの上手い付き合い方
MySQLアーキテクチャ図解講座
インメモリーデータグリッドの選択肢

What's hot (20)

PDF
마이크로서비스를 위한 AWS 아키텍처 패턴 및 모범 사례 - AWS Summit Seoul 2017
PPTX
Zookeeper Tutorial for beginners
PDF
それでも環境依存は残っている~起きたり起きなかったりする問題のお話~
PDF
ゲームアーキテクチャパターン (Aurora Serverless / DynamoDB)
PDF
Fluentd and Distributed Logging at Kubecon
PDF
マルチテナント化で知っておきたいデータベースのこと
PDF
마이크로서비스 기반 클라우드 아키텍처 구성 모범 사례 - 윤석찬 (AWS 테크에반젤리스트)
PDF
AWS X-Ray를 통한 서버리스 분산 애플리케이션 추적하기 - 윤석찬 (AWS 테크에반젤리스트)
PPTX
Kafka and Avro with Confluent Schema Registry
PDF
Amazon OpenSearch Deep dive - 내부구조, 성능최적화 그리고 스케일링
PPTX
Stephan Ewen - Experiences running Flink at Very Large Scale
PPTX
今こそ知りたいSpring Web(Spring Fest 2020講演資料)
PDF
angular fundamentals.pdf angular fundamentals.pdf
PDF
[AWS EXpert Online for JAWS-UG 18] 見せてやるよ、Step Functions の本気ってやつをな
PDF
AWSでDockerを扱うためのベストプラクティス
PDF
[AKIBA.AWS] AWS Elemental MediaConvertから学ぶコーデック入門
PDF
20200630 AWS Black Belt Online Seminar Amazon Cognito
PDF
PostgreSQLによるデータ分析ことはじめ
PDF
[WhaTap DevOps Day] 세션 4 : 롯데ON MSA 모니터링 최적화 사례
PDF
Amazon DocumentDB vs MongoDB 의 내부 아키텍쳐 와 장단점 비교
마이크로서비스를 위한 AWS 아키텍처 패턴 및 모범 사례 - AWS Summit Seoul 2017
Zookeeper Tutorial for beginners
それでも環境依存は残っている~起きたり起きなかったりする問題のお話~
ゲームアーキテクチャパターン (Aurora Serverless / DynamoDB)
Fluentd and Distributed Logging at Kubecon
マルチテナント化で知っておきたいデータベースのこと
마이크로서비스 기반 클라우드 아키텍처 구성 모범 사례 - 윤석찬 (AWS 테크에반젤리스트)
AWS X-Ray를 통한 서버리스 분산 애플리케이션 추적하기 - 윤석찬 (AWS 테크에반젤리스트)
Kafka and Avro with Confluent Schema Registry
Amazon OpenSearch Deep dive - 내부구조, 성능최적화 그리고 스케일링
Stephan Ewen - Experiences running Flink at Very Large Scale
今こそ知りたいSpring Web(Spring Fest 2020講演資料)
angular fundamentals.pdf angular fundamentals.pdf
[AWS EXpert Online for JAWS-UG 18] 見せてやるよ、Step Functions の本気ってやつをな
AWSでDockerを扱うためのベストプラクティス
[AKIBA.AWS] AWS Elemental MediaConvertから学ぶコーデック入門
20200630 AWS Black Belt Online Seminar Amazon Cognito
PostgreSQLによるデータ分析ことはじめ
[WhaTap DevOps Day] 세션 4 : 롯데ON MSA 모니터링 최적화 사례
Amazon DocumentDB vs MongoDB 의 내부 아키텍쳐 와 장단점 비교
Ad

Viewers also liked (7)

PDF
Akka in Practice: Designing Actor-based Applications
PDF
Akka cluster overview at 010dev
PDF
Slides - Intro to Akka.Cluster
KEY
Introduction to Actor Model and Akka
PDF
Introduction to Akka
PDF
Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...
PDF
Building Reactive Systems with Akka (in Java 8 or Scala)
Akka in Practice: Designing Actor-based Applications
Akka cluster overview at 010dev
Slides - Intro to Akka.Cluster
Introduction to Actor Model and Akka
Introduction to Akka
Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Ac...
Building Reactive Systems with Akka (in Java 8 or Scala)
Ad

More from Jonas Bonér (18)

PDF
We are drowning in complexity—can we do better?
PDF
Kalix: Tackling the The Cloud to Edge Continuum
PDF
The Reactive Principles: Design Principles For Cloud Native Applications
PDF
Cloudstate—Towards Stateful Serverless
PDF
Designing Events-first Microservices
PDF
How Events Are Reshaping Modern Systems
PDF
Reactive Microsystems: The Evolution of Microservices at Scale
PDF
From Microliths To Microsystems
PDF
Without Resilience, Nothing Else Matters
PDF
Life Beyond the Illusion of Present
PDF
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven Systems
PDF
Reactive Supply To Changing Demand
PDF
Go Reactive: Event-Driven, Scalable, Resilient & Responsive Systems
PDF
Introducing Akka
PDF
Event Driven-Architecture from a Scalability perspective
PDF
Scalability, Availability & Stability Patterns
PDF
State: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVM
PDF
Pragmatic Real-World Scala (short version)
We are drowning in complexity—can we do better?
Kalix: Tackling the The Cloud to Edge Continuum
The Reactive Principles: Design Principles For Cloud Native Applications
Cloudstate—Towards Stateful Serverless
Designing Events-first Microservices
How Events Are Reshaping Modern Systems
Reactive Microsystems: The Evolution of Microservices at Scale
From Microliths To Microsystems
Without Resilience, Nothing Else Matters
Life Beyond the Illusion of Present
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven Systems
Reactive Supply To Changing Demand
Go Reactive: Event-Driven, Scalable, Resilient & Responsive Systems
Introducing Akka
Event Driven-Architecture from a Scalability perspective
Scalability, Availability & Stability Patterns
State: You're Doing It Wrong - Alternative Concurrency Paradigms For The JVM
Pragmatic Real-World Scala (short version)

Recently uploaded (20)

PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
PDF
Chapter 3 Spatial Domain Image Processing.pdf
PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
PPT
Teaching material agriculture food technology
PDF
NewMind AI Monthly Chronicles - July 2025
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PDF
Modernizing your data center with Dell and AMD
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
PDF
Approach and Philosophy of On baking technology
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
PDF
Network Security Unit 5.pdf for BCA BBA.
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
PPTX
Big Data Technologies - Introduction.pptx
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
PPTX
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
Reach Out and Touch Someone: Haptics and Empathic Computing
Chapter 3 Spatial Domain Image Processing.pdf
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Teaching material agriculture food technology
NewMind AI Monthly Chronicles - July 2025
Advanced methodologies resolving dimensionality complications for autism neur...
Modernizing your data center with Dell and AMD
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
Dropbox Q2 2025 Financial Results & Investor Presentation
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
Approach and Philosophy of On baking technology
Mobile App Security Testing_ A Comprehensive Guide.pdf
Network Security Unit 5.pdf for BCA BBA.
The Rise and Fall of 3GPP – Time for a Sabbatical?
Big Data Technologies - Introduction.pptx
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....

Building Scalable, Highly Concurrent & Fault Tolerant Systems - Lessons Learned

  • 1. Building Scalable, Highly Concurrent & Fault-Tolerant Systems: Lessons Learned Jonas Bonér CTO Typesafe Twitter: @jboner
  • 2. I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again
  • 3. I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Lessons I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Learned I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again through... I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again
  • 4. I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Lessons I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Learned I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again through... I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Agony I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again
  • 5. I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Lessons I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Learned I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again through... I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again Agony and Pain I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again lots of Pain I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again I will never use distributed transactions again
  • 6. Agenda • It’s All Trade-offs • Go Concurrent • Go Reactive • Go Fault-Tolerant • Go Distributed • Go Big
  • 9. Performance vs Scalability
  • 10. Latency vs Throughput
  • 11. Availability vs Consistency
  • 14. Shared mutable state Together with threads...
  • 15. Shared mutable state Together with threads... ...leads to
  • 16. Shared mutable state Together with threads... ...code that is totally INDETERMINISTIC ...leads to
  • 17. Shared mutable state Together with threads... ...code that is totally INDETERMINISTIC ...leads to ...and the root of all EVIL
  • 18. Shared mutable state Together with threads... ...code that is totally INDETERMINISTIC ...leads to ...and the root of all EVIL Please, avoid it at all cost
  • 19. Shared mutable state B L E T U ! A Together with threads... M e!! IM at EVIL ...code that is totally INDETERMINISTIC e t ...leads to s s ...and the root of all U avoid it at all cost Please,
  • 20. The problem with locks • Locks do not compose • Locks break encapsulation • Taking too few locks • Taking too many locks • Taking the wrong locks • Taking locks in the wrong order • Error recovery is hard
  • 21. You deserve better tools • Dataflow Concurrency • Actors • Software Transactional Memory (STM) • Agents
  • 22. Dataflow Concurrency • Deterministic • Declarative • Data-driven • Threads are suspended until data is available • Lazy & On-demand • No difference between: • Concurrent code • Sequential code • Examples: Akka & GPars
  • 23. Actors •Share NOTHING •Isolated lightweight event-based processes •Each actor has a mailbox (message queue) •Communicates through asynchronous and non-blocking message passing •Location transparent (distributable) •Examples: Akka & Erlang
  • 24. STM • See the memory as a transactional dataset • Similar to a DB: begin, commit, rollback (ACI) • Transactions are retried upon collision • Rolls back the memory on abort • Transactions can nest and compose • Use STM instead of abusing your database with temporary storage of “scratch” data • Examples: Haskell, Clojure & Scala
  • 25. Agents • Reactive memory cells (STM Ref) • Send a update function to the Agent, which 1. adds it to an (ordered) queue, to be 2. applied to the Agent asynchronously • Reads are “free”, just dereferences the Ref • Cooperates with STM • Examples: Clojure & Akka
  • 26. If we could start all over...
  • 27. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core
  • 28. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming
  • 29. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow
  • 30. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed
  • 31. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming
  • 32. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming 3. Add Mutability selectively - only where needed
  • 33. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming 3. Add Mutability selectively - only where needed • Protected by Transactions (STM)
  • 34. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming 3. Add Mutability selectively - only where needed • Protected by Transactions (STM) 4. Finally - only if really needed
  • 35. If we could start all over... 1. Start with a Deterministic, Declarative & Immutable core • Logic & Functional Programming • Dataflow 2. Add Indeterminism selectively - only where needed • Actor/Agent-based Programming 3. Add Mutability selectively - only where needed • Protected by Transactions (STM) 4. Finally - only if really needed • Add Monitors (Locks) and explicit Threads
  • 37. Never block • ...unless you really have to • Blocking kills scalability (and performance) • Never sit on resources you don’t use • Use non-blocking IO • Be reactive • How?
  • 38. Go Async Design for reactive event-driven systems 1. Use asynchronous message passing 2. Use Iteratee-based IO 3. Use push not pull (or poll) • Examples: • Akka or Erlang actors • Play’s reactive Iteratee IO • Node.js or JavaScript Promises • Server-Sent Events or WebSockets • Scala’s Futures library
  • 40. Failure Recovery in Java/C/C# etc.
  • 41. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control
  • 42. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed
  • 43. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread
  • 44. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread • To make things worse - errors do not propagate between threads so there is NO WAY OF EVEN FINDING OUT that something have failed
  • 45. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread • To make things worse - errors do not propagate between threads so there is NO WAY OF EVEN FINDING OUT that something have failed • This leads to DEFENSIVE programming with:
  • 46. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread • To make things worse - errors do not propagate between threads so there is NO WAY OF EVEN FINDING OUT that something have failed • This leads to DEFENSIVE programming with: • Error handling TANGLED with business logic
  • 47. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control • If this thread blows up you are screwed • So you need to do all explicit error handling WITHIN this single thread • To make things worse - errors do not propagate between threads so there is NO WAY OF EVEN FINDING OUT that something have failed • This leads to DEFENSIVE programming with: • Error handling TANGLED with business logic • SCATTERED all over the code base
  • 48. Failure Recovery in Java/C/C# etc. • You are given a SINGLE thread of control d o • If this thread blows up you are screwed n !! • So you need to do all explicit error handling a ! c r WITHIN this single thread e te • To make things worse - errors do not W et propagate between threads so there is NO WAY OF EVEN FINDING OUT that b something have failed • This leads to DEFENSIVE programming with: • Error handling TANGLED with business logic • SCATTERED all over the code base
  • 51. The right way 1. Isolated lightweight processes 2. Supervised processes • Each running process has a supervising process • Errors are sent to the supervisor (asynchronously) • Supervisor manages the failure • Same semantics local as remote • For example the Actor Model solves it nicely
  • 53. Performance vs Scalability
  • 54. How do I know if I have a performance problem?
  • 55. How do I know if I have a performance problem? If your system is slow for a single user
  • 56. How do I know if I have a scalability problem?
  • 57. How do I know if I have a scalability problem? If your system is fast for a single user but slow under heavy load
  • 58. (Three) Misconceptions about Reliable Distributed Computing - Werner Vogels 1. Transparency is the ultimate goal 2. Automatic object replication is desirable 3. All replicas are equal and deterministic Classic paper: A Note On Distributed Computing - Waldo et. al.
  • 59. Fallacy 1 Transparent Distributed Computing • Emulating Consistency and Shared Memory in a distributed environment • Distributed Objects • “Sucks like an inverted hurricane” - Martin Fowler • Distributed Transactions • ...don’t get me started...
  • 60. Fallacy 2 RPC • Emulating synchronous blocking method dispatch - across the network • Ignores: • Latency • Partial failures • General scalability concerns, caching etc. • “Convenience over Correctness” - Steve Vinoski
  • 62. Instead Embrace the Network Use it i th w Asynchronous don e Message be d an Passing
  • 63. Guaranteed Delivery Delivery Semantics • No guarantees • At most once • At least once • Once and only once
  • 66. The network is inherently unreliable and there is no such thing as 100% guaranteed delivery It’s all lies.
  • 68. Guaranteed Delivery The question is what to guarantee
  • 69. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network?
  • 70. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC?
  • 71. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC? 3. The message is - put on the receiver’s queue?
  • 72. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC? 3. The message is - put on the receiver’s queue? 4. The message is - applied to the receiver?
  • 73. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC? 3. The message is - put on the receiver’s queue? 4. The message is - applied to the receiver? 5. The message is - starting to be processed by the receiver?
  • 74. Guaranteed Delivery The question is what to guarantee 1. The message is - sent out on the network? 2. The message is - received by the receiver host’s NIC? 3. The message is - put on the receiver’s queue? 4. The message is - applied to the receiver? 5. The message is - starting to be processed by the receiver? 6. The message is - has completed processing by the receiver?
  • 75. Ok, then what to do? 1. Start with 0 guarantees (0 additional cost) 2. Add the guarantees you need - one by one
  • 76. Ok, then what to do? 1. Start with 0 guarantees (0 additional cost) 2. Add the guarantees you need - one by one Different USE-CASES Different GUARANTEES Different COSTS
  • 77. Ok, then what to do? 1. Start with 0 guarantees (0 additional cost) 2. Add the guarantees you need - one by one Different USE-CASES Different GUARANTEES Different COSTS For each additional guarantee you add you will either: • decrease performance, throughput or scalability • increase latency
  • 78. Just
  • 80. Just Use ACKing and be done with it
  • 81. Latency vs Throughput
  • 82. You should strive for maximal throughput with acceptable latency
  • 85. Big Data Imperative OO programming doesn't cut it • Object-Mathematics Impedance Mismatch • We need functional processing, transformations etc. • Examples: Spark, Crunch/Scrunch, Cascading, Cascalog, Scalding, Scala Parallel Collections • Hadoop have been called the: • “Assembly language of MapReduce programming” • “EJB of our time”
  • 86. Big Data Batch processing doesn't cut it • Ala Hadoop • We need real-time data processing • Examples: Spark, Storm, S4 etc. • Watch“Why Big Data Needs To Be Functional” by Dean Wampler
  • 88. When is a RDBMS not good enough?
  • 89. Scalingreads to a RDBMS is hard
  • 90. Scalingwrites to a RDBMS is impossible
  • 92. Do we really need a RDBMS? Sometimes...
  • 94. Do we really need a RDBMS? But many times we don’t
  • 96. Availability vs Consistency
  • 98. You can only pick 2 Consistency Availability Partition tolerance At a given point in time
  • 99. Centralized system • In a centralized system (RDBMS etc.) we don’t have network partitions, e.g. P in CAP • So you get both: Consistency Availability
  • 100. Distributed system • In a distributed (scalable) system we will have network partitions, e.g. P in CAP • So you get to only pick one: Consistency Availability
  • 102. Think about your data Then think again • When do you need ACID? • When is Eventual Consistency a better fit? • Different kinds of data has different needs • You need full consistency less than you think
  • 103. How fast is fast enough? • Never guess: Measure, measure and measure • Start by defining a baseline • Where are we now? • Define what is “good enough” - i.e. SLAs • Where do we want to go? • When are we done? • Beware of micro-benchmarks
  • 104. ...or, when can we go for a beer? • Never guess: Measure, measure and measure • Start by defining a baseline • Where are we now? • Define what is “good enough” - i.e. SLAs • Where do we want to go? • When are we done? • Beware of micro-benchmarks
  • 105. To sum things up... 1. Maximizing a specific metric impacts others • Every strategic decision involves a trade-off • There's no "silver bullet" 2. Applying yesterday's best practices to the problems faced today will lead to: • Waste of resources • Performance and scalability bottlenecks • Unreliable systems
  • 106. SO
  • 107. GO
  • 108. ...now home and build yourself Scalable, Highly Concurrent & Fault-Tolerant Systems
  • 109. Thank You Email: jonas@typesafe.com Web: typesafe.com Twitter: @jboner