Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Every Piece of Content Ever Published in Kafka
1 like2,986 views
The document discusses how The New York Times stores all published content in Apache Kafka. It describes how the Times previously used an API-based architecture with different data schemas for each service, which led to inconsistencies. The new approach establishes a single source of truth (Kafka) with a normalized schema. All content is stored as a log in Kafka, then denormalized logs and special-purpose logs are replicated from the main log. This allows content to be efficiently accessed and ensures consistency across services. The implementation uses Google Cloud infrastructure and passive replication across regions. Key lessons are that log-based architectures are new, managed Kafka would help, and it will take time to transition all services.
Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Every Piece of Content Ever Published in Kafka
- 1. The Source of Truth 2017-05-08 The New York Times Why The New York Times Stores Every Piece of Content Ever Published in Kafka
- 2. Boerge Svingen Director of Engineering at the New York Times, working on backend systems.
- 3. Topic: How is published content made available to the front-end applications?
- 5. Agenda 1. A little history 2. How things used to work 3. The source of truth 4. Implementation 5. Lessons so far
- 6. Agenda 1. A little history 2. How things used to work 3. The source of truth 4. Implementation 5. Lessons so far
- 10. 20 years on the web
- 11. The New York Times Company Archives
- 12. The New York Times Company Archives
- 13. The New York Times Company Archives
- 14. The New York Times Company Archives
- 15. The New York Times Company Archives
- 16. The New York Times Company Archives
- 17. The New York Times Company Archives
- 18. Agenda 1. A little history 2. How things used to work 3. The source of truth 4. Implementation 5. Lessons so far
- 20. A rather typical API-based architecture.
- 21. Disadvantages with this approach … The consumers have to know about all the producers of content.
- 22. Disadvantages with this approach … Every API tends to be different.
- 23. Disadvantages with this approach … Every API tends to return data with a different (no) schema.
- 24. Disadvantages with this approach … We have no efficient way of reading old content in bulk, so it’s hard to replace service stores.
- 25. Disadvantages with this approach … Most services have to manage permanent state.
- 26. Disadvantages with this approach … It is difficult to change the (non-existent) schema, leading to inconsistencies and duplication.
- 27. Disadvantages with this approach … We get monoliths that try to be everything for everyone.
- 28. Disadvantages with this approach … It’s hard to develop new products and change current ones.
- 29. Agenda 1. A little history 2. How things used to work 3. The source of truth 3. Implementation 4. Lessons so far
- 32. We have a schema.
- 35. Article 1 Dateline 1 Credit 1 Section 1 Image 2 Image 1 Credit 2 Article 2 Section 2 Image 3
- 36. The schema. The Gateway validates all assets before they go on the log.
- 37. The schema. All assets are identified by a URI: nyt://article/186faf12-24a0-4dda-b737-018cee0b81cd
- 38. The schema. Custom linter to check for forwards and backward compatibility.
- 39. The schema. GraphQL schema is automatically generated from the protobuf schema.
- 40. Monolog.
- 41. The Monolog Single partition, totally ordered, infinite retention.
- 42. The Monolog The Source of Truth for published content.
- 43. The Monolog Contains everything published since 1851.
- 44. Article 1 Dateline 1 Credit 1 Section 1 Image 2 Image 1 Credit 2 Article 2 Section 2 Image 3
- 45. Article 1 Dateline 1 Credit 1Section 1 Image 2 Image 1Credit 2 Article 2 Section 2 Image 3 Topological sort
- 48. The denormalized log Replicated from the monolog.
- 49. The denormalized log Updates the full asset every time a dependency is updated.
- 50. The denormalized log Makes it easier for consumers that need all the dependencies.
- 51. The denormalized log Partitioned by asset ID.
- 54. Special-purpose logs All replicated from the monolog.
- 55. Agenda 1. A little history 2. How things used to work 3. The source of truth 4. Implementation 5. Lessons so far
- 56. Runs on Google Cloud.
- 57. Producersofcontent Consumersofcontent Kafka broker Kafka broker Kafka broker Kafka broker Kafka broker ZooKeeper ZooKeeper ZooKeeper ZooKeeper ZooKeeper Gateway Gateway Gateway Gateway ReplicatorsReplicatorsReplicatorsReplicators GKE (Kubernetes) GKE (Kubernetes) Google Compute Engine gRPC over Cloud Endpoint Kafka Consumer over SSL
- 60. Agenda 1. A little history 2. How things used to work 3. The source of truth 4. Implementation 5. Lessons so far
- 61. Managed Kafka would be very nice.
- 62. (Assuming it would run on Google Cloud.)
- 63. Log-based architectures are still very new.
- 64. Google PubSub/SNS/SQS/Kinesis are not replacements for Kafka.
- 65. It’s still early days.
- 66. It will take us a while to move all services over to the new architecture.
- 67. Questions?