Jitney, Kafka at Airbnb
- 1. Jitney, Kafka at Airbnb ALEXIS MIDON & KRISHNA PUTTASWAMY / 2016-02-23 / KAFKA MEETUP
- 2. Jitney ?! a bus carrying passengers for a low fare
- 3. Some Kafka Facts • 1 Production cluster • v0.8.2 • 90 “small” brokers, d2.2xlarge • 70 topics • Replication Factor of 3 • 5 Billions events / day • IN: 80MB / second • OUT: 1.5GB / second • Network bound • Super stable
- 4. Why Jitney?
- 7. What Use Cases ?
- 8. Classic Message Bus Jitney MySQL Monorail MySQL Elasticsearch Jitney Client & Schemas
- 9. • Decouple Services • Standard Events • At-least once delivery • Standard clients, for Java and Ruby • Easy to use • Conventions over configuration Message Bus
- 10. • Site and image load times, OOM events • Searches, requests, bookings, etc. • Experiment assignments • Event data is critical for building data products • Data ingestion should be reliable: timely and complete User Activity Logging
- 11. • JSON events without schemas • Easy to break events during evolution/code changes • One topic overall for 800+ event types • Improper producer configs • Lack of monitoring • Lead to: • Too many data outages, data loss incidents • Lack of trust on data systems Challenges
- 12. Data Stability CEO dashboard and Magical booking dashboards were regularly broken. A Year Ago
- 13. Data Stability ERF was unstable and experimentation culture was weak Hi team, This is partly a PSA to let you know ERF dashboard data hasn't been up to date/ accurate for several weeks now. Do not rely on the ERF dashboard for information about your experiment. A Year Ago
- 14. Join Forces! Data Infrastructure &Production Infrastructure
- 17. Thrift Schema Repository Why Thrift? • Easy syntax • Good performance in Ruby • Ubiquitous Advantages of schema repo? • Great Catalyst for communication, documentation, etc • it ships jar and gems • Will developers hate you for this? no
- 18. • Standard Field in the event schema • Managed Explicitly • use Semantic Versioning: 1.0.0 = MODEL . REVISION . ADDITION MODEL is a change which breaks the rules of backward compatibility. Example: changing the type of a field. REVISION is a change which is backward compatible but not forward compatible. Example: adding a new field to a union type. ADDITION is a change which is both backward compatible and forward compatible. Example: adding a new optional field. Schema Evolution
- 19. Example of Thrift Event because the event is your API
- 20. Jitney Components Schema Repository Topic Repository
- 21. Topic Repository • Declare all Jitney topics • Aggregate all characteristics of a topic: name ordering (partitioning function) white list of accepted schemas • Great for documentation purposes • DRY
- 22. Example of a Topic
- 23. Jitney Components Schema Repository Topic Repository Clients
- 24. Jitney Clients • Kafka clients are hard to use correctly • it’s better with 0.9 • Committing offsets is tricky, someone will get it wrong • even with 0.9 • Configuration is a mess
- 25. Jitney Clients it provides: • metrics reporting: github.com/airbnb/kafka-statsd-metrics2 • configuration for default clusters • built-in support for Schema Repository and Topic Repository Consumer: • offset management to implement at-least once delivery • polymorphic dispatching to event handler
- 26. Example of a Java Producer
- 27. Example of a Java Consumer
- 28. Jitney Components Schema Repository Topic Repository Clients HTTP Proxy
- 29. Jitney Components Schema Repository Topic Repository Clients HTTP Proxy Warehouse Integration
- 30. Data Ingestion Pipeline • Stack: Jitney, Spark Streaming, HBase, HDFS • Spark Streaming 1.5 with Kafka “direct” connect • Process 1 minute batches • Write to HBase after deserializing with the right schema • Dump data to HDFS every hour (with dedup) and add a Hive partition • But live data can be queried via “current” partition
- 31. Data Ingestion Pipeline end to end
- 32. Audit 124 124 3
- 33. Event Schema for Audit Metadata
- 34. How is Jitney used in the org?
- 35. DB change ingestion Payment processing via pub/sub Experimentation User activity ingestion Cache invalidation Use cases currently powered
- 36. Key take aways 1 2 3 Standardization! Auditing Pipeline Huge Advantage for the organization
- 37. Thank You!