Embracing Database Diversity with Kafka and Debezium
3 likes1,151 views
The document discusses the implementation of a player-centric model using Kafka and Debezium for change data capture in a sports management context. It highlights the challenges of scaling an Oracle database and the necessity of incorporating diverse data storage solutions like MongoDB and PostgreSQL. The architecture focuses on microservices and event-driven design to efficiently manage and process large volumes of data in real time.
![{
"_id" : "1001",
"Address" : [
{
"zip" : "76036",
"city" : "Euless",
"street" : "3183 Moore Avenue",
"id" : 10,
"state" : "Texas",
"customer_id" : 1001,
"type" : "SHIPPING"
}
],
"last_name" : "Thomas",
"id" : 1001,
"first_name" : "Sally",
"email" : "sally.thomas@acme.com"
}
MongoDB](https://image.slidesharecdn.com/embracingdiversitykafka-180314081016/85/Embracing-Database-Diversity-with-Kafka-and-Debezium-53-320.jpg)
Embracing Database Diversity with Kafka and Debezium
- 1. @lyaruu#Voxxed Embracing Database Diversity with Kafka and Debezium Frank Lyaruu
- 7. @lyaruu#Voxxed Embracing Database Diversity with Kafka and Debezium Frank Lyaruu CTO Dexels Amsterdam
- 8. • Service provider for professional and amateur team sports in the Netherlands & Belgium • 10+ years old • Managing personal data, planning competitions, assigning officials, supplying data feeds
- 9. • 2M+ players • 6K+ clubs • 40K matches a week • Spikey but predictable load
- 10. Technology stack • Oracle database • Cluster of Java based application servers • Diverse set of clients
- 12. Application Code Desktop client Desktop client Web client Web client Mobile client Mobile client Application Code Application Code Application Code
- 13. Challenge • Move to a player centric model instead of a club centric • Few orders of magnitude more users and load • Moving away from Oracle is not feasible in the short term • Scaling Oracle is just too expensive, if at all possible
- 15. Plan 1. Capture data in realtime 2. Dump into Kafka 3. Insert into MongoDB
- 16. Kafka • Kafka Broker • Kafka Connect API • Kafka Streams API
- 17. Kafka • Persistent pub/sub message bus • High throughput • Subscribers can consume at their own speed • Subscribers can request a ‘rewind’ and re-consume a topic • Has some tricks to keep the data volume down • Having both fast and slow consumers is not a problem
- 20. Write ahead log • Archive Log (Oracle) • Oplog (MongoDB) • Write Ahead Log (Postgres) • Binlog (MySQL)
- 21. Write ahead log • Recovery • Replication
- 22. Master / slave replication
- 24. Postgres (>=9.4) 46.2.1. Logical Decoding Logical decoding is the process of extracting all persistent changes to a database's tables into a coherent, easy to understand format which can be interpreted without detailed knowledge of the database's internal state. In PostgreSQL, logical decoding is implemented by decoding the contents of the write-ahead log, which describe changes on a storage level, into an application-specific form such as a stream of tuples or SQL statements. Jeff Klukas / Postgres 9.4 docs
- 25. Debezium • RedHat • Standardize change data capture • Uses Kafka Connect API • Pretty young: 0.7.4 • Based on ‘Bottled Water’ research project
- 26. Debezium • Postgres • Mysql • MongoDB • Oracle*
- 27. Core Service Source database Java Application Server Kafka topic: PERSON { id: 123 name: “Alfredo” dob: 1965-5-1 } User Backend Service MongoDB Database Java Application Server topic: PERSON { id: 123 name: “Alfredo” dob: 1965-5-1 }
- 28. SELECT * FROM Communication C WHERE PersonId = 1 SELECT * FROM Person WHERE PersonId=1 Example
- 30. Stream Processing SQL Record SQL Record Stream transformation MongoDB
- 31. Stream Processing SQL Record SQL Record Kafka Streams MongoDB RocksDB
- 32. Kafka Streams at Scale • ± 500M rows of SQL data • ± 50 joins • 500 topics • 400 Gb of Kafka Data • 300 Gb of RocksDb data • Building a complete replica from scratch takes many hours • After that <100ms latency for changes
- 33. Development cycle • Developing and testing is hard for stateful code • Starting a new ‘generation’ is costly • Contaminated data might show up
- 34. Conclusions • Went into production last June • Generally behaves well (aside from some glitches) • Kafka Streams is in a lot better state than a year ago
- 35. Architecture
- 36. Microservice API Any private store Code in whatever language
- 37. Different parts need the same data … but in a different way
- 38. Application UI SQL Database Code in some language Analytics code Analytics UI
- 39. Application Service UI SQL Database Code in some language Analytics UI Analytics code Analytics Service Analytics Database Analytics API
- 40. Application Service UI SQL Database Code in some language Analytics UI Analytics code Analytics Service Analytics Database Analytics API
- 41. Event Driven Microservices • Services push events instead of a request/response model • Usually backed by a publish/subscribe bus
- 42. Application Service SQL Database Code in some language Event Bus topic: PERSON { id: 123 name: “Alfredo” dob: 1965-5-1 } Analytics Service Analytics Database Code in some language topic: PERSON { id: 123 name: “Alfredo” dob: 1965-5-1 }
- 43. Event Sourcing
- 45. Elasticsearch • Add unstructured search to our application • Reduces load on our source databases • Users expect google-like interfaces
- 46. Neo4J • Graph Database • Some analytics are much easier to express in terms of graphs
- 47. Firebase Realtime • Real-time database • ‘Backend As a Service’ • Essentially one big JSON document • Very easy to use client libraries for web and mobile • Safe to develop
- 48. Caches • We can use our streaming engine to update / invalidate caches
- 49. Push to clients • We can push data to clients in real time
- 51. Mac book Postgres Zookeeper Kafka Kafka Connect / Debezium Kafka Streams MongoDB
- 53. { "_id" : "1001", "Address" : [ { "zip" : "76036", "city" : "Euless", "street" : "3183 Moore Avenue", "id" : 10, "state" : "Texas", "customer_id" : 1001, "type" : "SHIPPING" } ], "last_name" : "Thomas", "id" : 1001, "first_name" : "Sally", "email" : "sally.thomas@acme.com" } MongoDB
- 55. Demo?
- 56. If all you have is a hammer…
- 57. If all you have is SQL… you will always think relational you will always think you need nothing else
- 58. Questions? * Any question that is not: “Why don’t you use Postgres? Postgres can do anything”