Cloud-native Event-driven Data Ecosystem Powered by Kafka and Akka (Masih Derkani, Solarwinds) Kafka Summit London 2019

Cloud-Native, Event-Driven Data Ecosystem
Powered by Kafka and Akka
Masih H. Derkani

This presentation contains forward-looking statements regarding future product plans and
development efforts. SolarWinds considers various features and functionality prior to any final
generally available release. Information in this presentation regarding future features and
functionality is not and should not be interpreted as a commitment from SolarWinds that it will
deliver any specific feature or functionality in the future or, if it delivers such feature or
functionality, any time frame when that feature or functionality will be delivered. All information
is based upon current product interests, and product plans and priorities can change at any
time. SolarWinds undertakes no obligation to update any forward-looking statements regarding
future product plans and development efforts if product plans or priorities change.
2

3
Agenda
Background.
The Challenge.
Mission.
EventBus.
EventBus Reborn.
What’s Next.
Takeaways.
© 2019 SolarWinds MSP Canada ULC and SolarWinds MSP UK Ltd. All rights reserved.

4
The MSP business
Background
and a whole lot of subsystems…

5
≈6.3M Managed Devices
Scale
The Challenge

6
≈6.3M Managed Devices
≈22B Events per Day
Scale
The Challenge

7
≈37MDiscovered Devices
in RMM sub-systems alone.
Scale
The Challenge

8
Existing infrastructure
The Challenge

9
Cultural shift
The Challenge
Source: https://www.amazon.co.uk/Ridleys-Games-Chess-Checkers-Wooden/dp/B00OBQL8GO
The game has changed;
time to adapt.

10
Philosophy
Mission
Platform.
Ecosystem.
Enable engineers to assemble,
not fabricate.
Expose essential complexity,
minimize accidental complexity.
DaVinci
D a t a P l a t f o r m

11
Fundamental data flow
Mission

12
Data flow in more detail
Mission

13
The Nervous System of Data Platform.
Uniform data format for all.
Multiple Kafka Clusters.
Custom Java client library.
Keep Kafka swappable.
Guaranteed order and delivery.
DaVinci EventBus
…is born
EventBus

14
High-level architecture
EventBus

15
K8S cluster built to work with SolarWinds services.
Infrastructure as code.
≈5 minutes to deploy a fresh Kafka Cluster.
Need for resilience is built into the environment.
Cloud-native environment
EventBus: Deployment

16
Abstraction over Kafka topics
EventBus: Streams

17
Multicluster management
EventBus: Manager

18
Topic configuration management
EventBus: Manager

19
Stream route file management
EventBus: Manager

20
Stage 1
Cluster Switch Example

21
Stage 2

22
Endpoint and status metadata
Example Route File
{
"updated": 1554901518,
"clusters": [{
"uid": "kingpin",
"name": "kingpin",
"conInternal": ["kingpin-kafka-headless:9092"],
"conExternal": ["kingpin-0.eventbus.solarwindsmsp.com:443"]}
],
"producer": { "cluster_uid": "kingpin", "status": "active", "topic": "fish"},
"consumer": [
{"cluster_uid": "kingpin", "status": "active", "topic": "fish"}
]
}

23
Incoming message flow on rerouting

24
So far, so good…

25
So far, so good…
(until the first client failed)
x

26
Only two levels of Cluster History.
Difficult to diagnose client issues.
Bug fixes = high code churn for clients.
Hard to support multi-language client.
Cultural shift towards event-driven
architecture.
Pushing forward
Room for Improvements
$

27
High-level architecture
EventBus 2.0

28
Sharding and coordination via Akka
EventBus 2.0

29
Single EventBus stream identity across multiple Kafka Clusters
Kafka Cluster Transparency

30
Generic gRPC components for Java, Go, C# and
node.js.
Cross-cluster consumer group coordination.
Automated partition count elasticity.
Automated Kafka Cluster Deployment.
GDPR built-in.
l
Plans
What’s Next

31
Teach the difference between consumption and
processing.
Watch out for leaky abstractions.
Avoid exposing partitions.
Timestamp over offset.
APIs everywhere.
Auto-generate multilanguage client libraries.
&
…the hard way
Lessons We Learned

The SolarWinds and SolarWinds MSP trademarks are the exclusive property of SolarWinds MSP
Canada ULC, SolarWinds MSP UK Ltd. or its affiliates and may be registered or pending
registration with the U.S. Patent and Trademark Office and in other countries. All other
SolarWinds MSP and SolarWinds trademarks, service marks, and logos may be common law
marks or are registered or pending registration. All other trademarks mentioned herein are
used for identification purposes only and are trademarks (and may be registered trademarks) of
their respective companies.
34

Cloud-native Event-driven Data Ecosystem Powered by Kafka and Akka (Masih Derkani, Solarwinds) Kafka Summit London 2019

More Related Content

What's hot (20)

Similar to Cloud-native Event-driven Data Ecosystem Powered by Kafka and Akka (Masih Derkani, Solarwinds) Kafka Summit London 2019 (20)

More from confluent (20)

Recently uploaded (20)

Cloud-native Event-driven Data Ecosystem Powered by Kafka and Akka (Masih Derkani, Solarwinds) Kafka Summit London 2019