Kafka in practice
6 likes650 views
The document discusses how Apache Kafka is used at Easy Taxi, a ride-hailing company. It describes Easy Taxi's Kafka architecture with over 600 million unique events per day distributed across 3 brokers and 10 topics. It provides recommendations for using Kafka, such as employing Rsyslog as a simple producer and BigQuery for SQL queries. Potential pitfalls are also outlined, such as not performing stress tests or adding unnecessary complexity.
Related topics:1 of 15
Download to read offline
More Related Content
What's hot (20)
Ad
Viewers also liked (10)
Ad
Similar to Kafka in practice (20)
Recently uploaded (20)
Kafka in practice
- 1. KAFKA in practice Caique Lima @ Easy Taxi
- 2. Apache Kafka distributed, partitioned, replicated commit log service
- 3. Apache Kafka
- 4. Apache Kafka
- 5. Apache Kafka
- 6. Kafka at Easy Taxi • +600 mi unique events/day • Currently on 0.9.0 • 3 brokers • 10 topics • X consumers
- 7. Architecture Api Kafka Consumer 1 Consumer 2 Consumer 3 S3 Redis Consumer N BQ Api … Redash
- 8. Do’s • Rsyslog as a scalable simple producer • Use BigQuery to allow SQL • Start building the event based culture • Right-time before real-time
- 9. Dont’s • Doesn't run stress tests • Increase architecture’s complexity • Produce useless data • Go live with default configs
- 10. Easy Explorer
- 11. Easy Taxi Point
- 12. Supply demand analysis • Allow us to understand the area behaviour • Generate a “health indicator” based on predefined parameters • Take actions without any human interaction
- 13. References • Confluent blog • Kafka Summit • Jay Kreps interview • Jepsen: Kafka • Documentation • Dados Brasil Slack
- 14. THANKS!
- 15. QUESTIONS