30 Of My Favourite Open Source Technologies In 30 Minutes

© Instaclustr Pty Limited, 2024
30 Of My Favourite
Open Source
Technologies
Paul Brebner
Open Source Technology Evangelist

30 Of My Favourite
Open Source
Technologies
In 30 Minutes
Paul Brebner
Open Source Technology Evangelist
Paul Brebner (Netherlands 30
minutes bike parking zone)

What do they have in Common?
• Instaclustr provides some as
managed services
• They are complementary and
can be used together
• And I’ve used them to build
realistic demo applications
over the last 7 years

A Strange Toy I Found At The Shop
• What’s that?!
• An escaped “Pokemon”!
• When my kids were growing up Pokemon lived inside a
“Game Boy”

Format
• Name, Overview, Superpower(s), Watch out for …
• E.g. “Pokemon”
• Name: Charmander
• What: A fire Lizard
• Superpower: Evolves to Charizard, a flying fire breathing lizard
• Watch out for: Water
+ Use Cases and What’s New?

Countdown!
Flicker CCL + Wikimedia CCL

1. Apache Cassandra
Office Typing
Pool, 1918
Wikipedia Public Domain

Apache Cassandra
• What?
• NoSQL Horizontally Scalable Key-Value Database
• Superpowers
• Fast Writes (lots of typewriters)
• Wide Column Store
• Clustering Columns, good for hierarchical data modelling (E.g. Geospatial)
• In-built multi-DC replication
• My Use Cases

Anomaly Detection: 19 Million
checks/day
Apache Cassandra
Apache Kafka
Kubernetes
And more

Global low-latency Fintech

Apache Cassandra
• Watch Our For
• CQL != SQL
• Different data model
§ Design for reads
§ De-normalization is normal
• Consistency < traditional SQL databases
• Reads are slower
• What’s New?
• Vector Search in 5.0

2. Apache Spark
Car Factory
Assembly Line
Wikimedia Public Domain

Apache Spark
• What?
• Cluster batch/stream processing, analytics and ML
• Superpowers
• In-memory à fast
• Good support for ML
§ + Cassandra (wide columns) as a feature store
• Good for heavy transformation operations at scale
• My Use Cases

ML of Cassandra Monitoring Data
Apache Spark
Apache Cassandra
MLlib
DataFrames
Spark Streaming

Apache Spark
• Watch Our For
• Lots of RAM, else OOM (Out-of-Memory Errors)
• Spark Streaming is near real-time (micro-batch)
• What’s New?
• 3.4 has Spark Connect for decoupled client-servers
• Ocean for Apache Spark
(Spot by NetApp)

3. Apache Zeppelin
Graf Zeppelin
exploring the
Arctic, 1931

Apache Zeppelin
• What?
• Web-based notebook for data exploration
• Superpowers
• Interactive “notebook” style tool
• Supports Apache Spark

Apache Zeppelin
• Watch Our For
• Sufficient Zeppelin resources
• We don’t support it anymore
• What’s New?
• Jupyter Notebook!
§ Good Kafka and Cassandra integration
The Galilean moons of Jupiter (Wikimedia CCL)

4. Apache Lucene
A Librarian using
a card catalogue
(1940)
Library of Congress Public
Domain

Apache Lucene
• What?
• Fast Full-featured Search Engine
• Superpowers
• Lucene plugin + Cassandra for enhanced Cassandra search
§ Works as a Cassandra secondary index
§ Support Vector Search too
• Watch Our For
• Performance
• We currently support it: https://github.com/instaclustr/cassandra-lucene-index
• My Use Cases

Geospatial Anomaly Detection
Apache Cassandra
Apache Lucene Plugin
Geospatial searches

5. Apache Kafka
Postal Delivery
Service
Railway Post
Office:
Mail bags
snatched by
speeding train
Wikimedia CCL

Apache Kafka
• What?
• Distributed publish-subscribe messaging system
• Superpowers
• Fast
• Highly distributed and horizontally scalable, available and durable
• Buffering and message replay
• My Use Cases

Xmas Tree Lights Simulation

“Kongo” IoT Logistics Simulation
Apache Kafka
Guava Event Bus
Real-time logistics
Tracking and checking

Apache Kafka
• Watch Our For
• Too many topics/partitions impacts throughput
• What’s New?
• KRaft (replacing ZooKeeper) for faster meta-data operations
§ And maybe even faster data workloads
• Tiered Storage (3.6)
• End-to-end client monitoring (3.7)

6. Apache Kafka Streams
Niagra Falls
Darevevil
Shutterstock

Apache Kafka Streams
• What?
• Stream processing API and client for Kafka
• From/to Kafka cluster
• Superpowers
• Complex stateful stream processing operations (e.g. joins)
• Over time windows and multiple topics and state stores
• My Use Cases

Kafka Streams IoT Application
Truck Overload

Apache Kafka Streams
• Watch Our For
• Complex stream topologies
• Debugging is tricky
• Performance
• What’s New?
• Alternatives (E.g. Apache Flink, RisingWave, etc)

7. Apache Kafka Connect
Telephone
Switchboard
Operators
Connecting Calls

Apache Kafka Connect
• What?
• Kafka API for streaming from source to sink systems
• Via Kafka cluster
• Superpowers
• Heterogeneous integration
• Code-free – just connector configuration
• Independently scalable
§ connectors run on independent Kafka Connect cluster
• My Use Cases

Zero-code Data Pipelines
REST Tidal Data to PostgreSQL + Superset
REST Tidal Data to OpenSearch
OpenSearch sink
connector

Apache Kafka Connect
• Watch Our For
• Open-source connector evaluation and selection
• Error handling
• Source/sink system scalability
• What’s New?
• Debezium

8. Kafka MirrorMaker 2 (MM2)
Head of Kafka
Replicated tiers
move
Shutterstock

Kafka MirrorMaker 2
• What?
• Replicates Kafka topics between clusters
• Superpowers
• Uses Kafka Connect (but reads/writes from/to Kafka clusters)
• Topic renaming, prevents loops
• Complex bi-directional topologies
• Many use cases for multiple Kafka clusters:
§ Cluster migration
§ Geographical distribution
§ Low latency, redundancy
§ Fan-out architectures
§ Edge computing, etc

Kafka MirrorMaker 2
• Watch Our For
• Bi-directional flow requires TWO Kafka Connect Clusters
• Duplicate events (from overlapping topic subscriptions)
• Use topic renaming and the default source cluster alias to
§ Prevent cycles and infinite topic creation
• What’s New?
• For me, automated consumer offset sync across clusters
§ In 2.7.0 (2020)!

9. Apache Camel
Camel Train In
Broome, WA
(Adobe Stock by scottimage)

Apache Camel - Kafka Connectors
• What?
• Apache Camel – Integration framework
• Apache Camel Kafka Connectors – open source Kafka connectors
• Superpowers
• Large number of open source Kafka Connectors – 172 (officially), 179 sources and sinks
• Auto-generated from Camel components

• Watch Our For
• Configuration!
§ Need to read (1) Camel component, (2) Basic connector configuration, and (3)
connector specific documentation
• Some connectors are both sources and sinks (source or sink depends on
configuration)
• What’s New?
• Kamelets!
§ Can appear in the configuration
Apache Camel - Kafka Connectors

10. Kafka Parallel Consumer
Jacquard Loom,
Berlin
(Paul Brebner)

Kafka Parallel Consumer
• What?
• Multi-threaded Kafka Consumer
• Superpowers
• Multi-threaded c.f. default consumers single-threaded
• Higher concurrency with less consumers and partitions
• Use Cases
• Low latency, High Throughput
• Slow consumers
• Replacement for my multiple pool consumer hack

Kafka Parallel Consumer
• Watch Our For
• Configure for
§ Ordering mode
• Partition à Key à Unordered (Increasing concurrency)
§ Max threads
• What’s New?
• Choice of commit modes
§ Consumer Asynchronous, Synchronous and Producer Transactions

11. Apache ZooKeeper
12. Apache Curator
Being a
ZooKeeper in
Australia can be
risky!
(Shutterstock)

Apache ZooKeeper
• What?
• Distributed systems and coordination and
meta-data management
• Superpowers
• High consistency, availability and performance (reads)
• Use Cases
• Until recently, used in Kafka, Pulsar, etc

Apache ZooKeeper (and Curator)
Meet the Dining Philosophers
Wikipedia CCL

Apache ZooKeeper
• Watch Our For
• Low-level
• Apache Curator (high level ZK client) is better with
§ Leader Latch
§ Shared Lock
§ Shared Counter
• Scalability limitations
• Slow for writes, max cluster size is 7 servers
• What’s New?
• KRaft – Kafka based RAFT implementation
§ For meta-data management and leader election
§ Faster meta-data operations, more partitions etc. Potentially faster data workloads

13. Kubernetes
Greek Triremes
ruled the seas
Captained by
Helmsmen
(Kubernetes)
(Wikipedia CCL)

Kubernetes
• What?
• Automation of containerized applications
• Superpowers
• Available on public clouds, E.g. AWS EKS
• Ephemeral Pods are the unit of concurrency
• Easy to scale applications (more or less Pods)
• My Use Cases

Kubernetes
• Watch Our For
• Pod and resource scaling
§ Easy to create many Pods
• With insufficient or lots of resources
• Tuning the application can be tricky
§ Optimize the number of Pods vs Kafka consumers/partitions, Cassandra database
connections, etc
• What’s New?
• Operators
§ E.g. Strimzi for Kafka

14. Prometheus
15. Grafana
Counting on an
Abacus
(Wikimedia Public Domain)

Prometheus + Grafana
• What?
• Prometheus: Monitoring and Alerting
• Grafana: Graphing
• Superpowers
• Instrumentation or Agents (Exporters) to expose application metrics
• Time series data with counter, gauge, histogram and summary metrics
• My Use Cases
• Monitoring and scaling/optimization/debugging
§ Anomaly Detector (Cassandra, Kafka, Kubernetes) application
§ Kafka Connect data pipelines
• Instaclustr’s Monitoring API has a Prometheus version

Prometheus + Grafana
• Watch Our For
• Need to run a Prometheus server
• Configuring Prometheus with Kubernetes is tricky
§ use Prometheus Operator
• What’s New?
• Since using it Grafana is now AGPL licensed
§ modified code has to be open sourced

16. OpenTracing
17. OpenTelemetry
18. Jaeger (and others)
X-Ray Vision!
Public Domain

OpenTracing/OpenTelemetry
• What?
• OpenTracing: End-to-end distributed tracing
• Superpowers
• End-to-end distributed application visibility
§ Traces have Spans
• Visualisation of system topology and times

OpenTracing
OpenTelemetry
• Watch Our For
• Originally used OpenTracing and Jaeger
• Manual instrumentation
• What’s New?
• OpenTelemetry is the new standard
§ Tracing, metrics and logs
§ Automatic instrumentation
§ Lots of open-source visualization tools
• Jaeger, SigNoz, Uptrace, OpenSearch
§ Used in new client monitoring KIP-714, Kafka 3.7.0

SigNoz Service Map for
Toy+Boxes application

19. PostgreSQL
Elephant vs. Tree
Elephants are
Powerful
Adobe

PostgreSQL
• What?
• Powerful SQL Database
• Superpowers
• SQL + Object Database
• Extensible
• JSONB+GIN indexes (efficient storage and search of JSON)

PostgreSQL
• Watch Our For
• Scalability
§ Vertical; limited horizontal
• Benefits from connection pooling
• What’s New?
• PGVector (vector similarity search)
• Significant performance improvement
§ on NetApp Azure Files
• FerretDB (MongoDB front-end)

20. Apache Superset
All superheroes
(B) are a superset
of those who use
weapons (A)
(Shutterstock)

Apache Superset
• What?
• Powerful data visualization tool
• Superpowers
• Reads from SQL sources
• Lots of visualization and graph types including geospatial
• My Use Case
• Visualization of tidal data from Kafka
connect pipeline
§ Easy integration with PostgreSQL + JSONB

21. OpenSearch
22. Dashboard
Library of Congress
Card Division 1919
(City block long)
(Library of Congress Public
Domain)

OpenSearch + Dashboard
• What?
• Open-source version of ElasticSearch
• Based on Lucene à powerful + scalable text searching
• Superpowers
• Ingestion, indexing and searching of JSON documents
• Integrated dashboard for visualization
• Computational linguistics support:
§ Stemming, Lemmatization, Levenshtein Fuzzy Queries,
N-grams, Slop, Partial matching!
• My Use Cases
• Sink and visualization for Kafka connect
tidal data processing pipeline

OpenSearch + Dashboard
• Watch Our For
• Default mappings and ingestion may not work
§ E.g. geospatial data needs custom mappings and ingest pipelines
• Reindexing
• Kafka Connect Sink à OpenSearch throughput
§ Needed the BULK API
• What’s New?
• Vector Search

23. Redis
Look! Up in the sky!
It’s an in-memory
key-value store!
It’s a database!
It’s Redis!
(Shutterstock)

Redis
• What?
• Fast (in-memory) Data Structures server
• Superpowers
• Lots of data types
§ Keys, Strings, Lists, Hashes, Sets, Sorted sets, bitmaps, geospatial, streams, time series,
HyperLogLogs (approximate counting)
• Pub/Sub
§ Connected and disconnected delivery
• Client-side caching for ultra-low latency – e.g. Redisson client

Redis
• Watch Our For
• Pipeline tuning impacts throughput
• Often used as a cache to reduce load on backend database
§ I.e. Efficiency not improved latency
• As other factors may dominate
• What’s New?
• Redis Functions
§ Code executed on the server (Redis 7)
• License change (7.4 source-available)

24. Uber’s Cadence
Railway Signal “man”
(Signalwoman!)
(Wikimedia Public Domain)

Uber’s Cadence
• What?
• Scalable code-as-workflows engine
• Superpowers
• Sequenced, stateful, long-running, scheduled steps
• Scalable and reliable using event-sourcing
§ Workflows are failproof, history is replayed until the point of failure and resumed
• My Use Cases

Drone Delivery Application
Kafka Microservices
Integration of fast/slow systems

Uber’s Cadence
• Watch Our For
• Uses Apache Cassandra and OpenSearch backends
• Code must be deterministic (replayed on failure)
§ Use special functions for non-deterministic functions
• What’s New?
• Potential use cases
§ Scalable push notifications (Uber)
§ ML workflows

25. Debezium
Animal speed transformation (Shutterstock)

Debezium
• What?
• Change Data Capture (CDC)
• Superpowers
• Captures slow database state changes
• Turns them into fast Kafka events
• Uses Kafka: Kafka Connect, and/or DB-specific “Connectors”
• Can be used to replicate databases (same type), or send events to different sink
systems
• My Use Cases
• Debezium Cassandra Connector (doesn’t use Kafka Connect, writes to Kafka directly)
• Debezium PostgreSQL Connector (Kafka source connector)

Debezium
• Watch Our For
• The DB specific connectors need to be configured/run in the DB
• Debezium change data format is complex
§ Actual content depends on the source DB
• Schemas may be inline or just an ID
• May include schema changes
• Tricky to find Kafka Connect sink connectors that work correctly
• Duplicates and ordering issues, latency and scalability challenges
• Schema IDs require a Kafka Schema Registry
• What’s New?
• GA on Instaclustr’s managed Cassandra (Dec 2023)

26. Karapace
Karapace in the
driver's seat!
(Shutterstock)

Karapace
• What?
• Open-source Kafka Schema Registry
• Superpowers
• Adds Schemas to Schemeless Kafka
• Supports multiple schema formats
§ Avro, Protobuf and JSON Schemas
• Kafka cluster is not directly involved
§ Karapace enforces schema checks for clients only
• Use Cases
• Debezium

Karapace
• Watch Our For
• Auto vs. manual schema registration – manual is safer in production
• Schema compatibility, compatibility modes, and evolution: complex!

27. FerretDB
Fish/Shark?
(Adobe)

FerretDB
• What?
• Open-source MongoDB proxy for PostgreSQL
• Superpowers
• Compatible with MongoDB drivers on the front-end
• Pluggable backends including PostgreSQL (using JSONB/GIN indexes)
• Query Pushdown for efficiency/performance

28. RisingWave
Wave processing
(Adobe)

RisingWave
• What?
• Stream processing database – also as a Service
• Superpowers
• Stateful stream processing
§ Using Cloud Native Storage
§ Potential replacement for Kafka Streams
• PostgreSQL compatible
§ Works with Apache Superset
• My Use Cases

Santa’s Elves Toy + Box Packing
Streaming joins to match toys and boxes (Adobe) Service Map using
OpenTelemetry + SigNoz

RisingWave
• Watch Our For
• SQL != Kafka Streams DSL
• Kafka keys not propagated
• Windowing has different semantics

29. TensorFlow
What does the
future hold?
(Adobe)

TensorFlow
• What?
• Neural network ML library
• Superpowers
• Supports incremental ML
• From streaming Kafka data
• My Use Cases

ML Over Streaming Kafka
Data – With Concept Drift
Kafka Streams

TensorFlow
• Watch Our For
• ML over streaming spatiotemporal data with concept drifts is tricky
§ Time/space bias
• Wild model accuracy oscillation
§ Concept shift can result in very low-accuracy models initially
• Train/use Multiple Models
0
0.2
0.4
0.6
0.8
1
0 20 40 60 80 100 120
Concept Drift - incremental training (time
vs accuracy)
same model reset model guessing

30. Yours Here
Invent your own
(DeepAI)

Integration Example 1
Our Customer Facing Monitoring
Before:
Spark and API
requests
à High load on
Cassandra

Our Customer Facing Monitoring
After:
Kafka + Kafka
Streams + Redis
Reduced
Cassandra Load
Recent metrics
served from Redis,
or Cassandra on
cache miss
Postgre
SQL
2 – get data from Redis
3 - or from Cassandra
1 – get meta-data
20k Nodes
Thanks to my colleague
Kuangda He
for this information

Drone Delivery Demo
© Instaclustr Pty
Limited, 2023
Kafka
Streams
Customers
Order
Shops
Busy warnings
Uses Cassandra+OpenSearch
ML over streaming data
Demo/POC

Drone Delivery Prod?
© Instaclustr Pty
Limited, 2023
Kafka
Streams
Customers
Order
Postgre
SQL
Drone Operations
Order Tracking
Shops
Busy warnings
Uses Cassandra+OpenSearch
ML over streaming data
Drone/order locations cached in Redis
Read-through or write-behind
Kafka sink
connectors

www.instaclustr.com
info@instaclustr.com
@instaclustr
THANK
YOU!

30 Of My Favourite Open Source Technologies In 30 Minutes

More Related Content

Similar to 30 Of My Favourite Open Source Technologies In 30 Minutes (20)

More from Paul Brebner (20)

Recently uploaded (20)

30 Of My Favourite Open Source Technologies In 30 Minutes