Running a Massively Parallel Self-serve Distributed Data System At Scale
Download as PPTX, PDF0 likes578 views
The document discusses Netflix's real-time data infrastructure, highlighting the need for a scalable and self-serve distributed data system called the Keystone streaming platform. It outlines the challenges faced, including scalability and multi-tenancy, and the solutions implemented, such as event sourcing and separation of concerns. Key principles include treating failure as a core aspect and utilizing declarative reconciliation for system management.
Running a Massively Parallel Self-serve Distributed Data System At Scale
- 1. Running a Massively Parallel Self-serve Distributed Data System At Scale Zhenzhong Xu | @ZhenzhongXu Real-time Data Infrastructure
- 5. Running a Massively Parallel Self-serve Distributed Data System at Scale ● What is Netflix Real-time Data Infrastructure ● Challenges ● Solutions and Principles
- 6. … Real-time Data Infrastructure at Netflix What is ...
- 7. Business/Product Driven Analytics ● Recommendations / Personalization Algorithms ● Customer Experience ● Content Operation ● A/B Testing ● Marketing ● etc
- 8. Rise of Event Driven Architecture ● Notification ● Event Sourcing ● CQRS ● etc
- 9. We need a data platform that’s both scalable and real-time.
- 11. So what exactly is Keystone Streaming Platform?
- 12. Publish, Collect, Move & Compute event data in near real time @ Cloud Scale
- 13. Keystone is ... … a collection of microservices & components Stream Processing Service Pub/Sub Queuing Service Producer API Control Plane Consumer API Self Service UI
- 14. Keystone is ... … a single self-contained logical PaaS Event Processing Pipeline
- 15. Keystone is ... … a multi-tenants, self-serving tool
- 16. Keystone is ... … a self healing, cloud failure tolerant service, guarantees at-least-once delivery semantics … adaptable to changing environments
- 17. Stream Consumers Batch System Fronting Kafka Event Producer Consumer Kafka Control Plane Self Service UI Putting together ... mantis
- 18. … the decisions we made to build scalable, reliable and maintainable systems. Challenges Solutions & Principles
- 20. A Single Stream
- 21. Anatomy of a Single Stream Stream Processing Job Queuing Service Event Producer Sink
- 23. Separation of Concerns ● Separation of Messaging and Stream Processing services. ● Each service scales individually. ● Each service manages its own states. ● Independently manage service dependencies. ○ Kafka brokers on EC2 ○ Streaming Service job on Titus Container Runtime
- 24. Titus Container Runtime ● Resource Provisioning ● Scheduling and bin-packing ● Capacity Guarantees ● Resource isolation ● Per container IP address (underlay via VPN)
- 25. Delivery/Processing Semantics ● At-most once ● At-least once ● Exactly once
- 26. At-least Once Processing Checkpointing ● Synchronous checkpointing through event loop
- 27. Exactly Once* Processing Semantics ● Lightweight Asynchronous Snapshot (Async Barrier Checkpointing)
- 28. Per Stream Monitoring & Alerting
- 31. Logical Isolation ● Streams Level ● Deployments Level ● What about regional Island Isolation?
- 32. Total Bytes Out = (num Of Consumers + replication factor - 1) * Bytes In
- 33. Solve Consumer Fanout with Hierarchies
- 34. Total Infrastructure Scale ● 500+ Billion events generated per day ● 1+ Trillian events processed per day ● ~800 Topics ● ~1,800 Streams ● 4000+ Kafka Instances ● ~9,000 Stream Processing Containers
- 39. Pet vs Cattle
- 41. Principles: Failure as a First Class Citizen Separation of Concerns Embrace Immutability
- 42. Challenge 2: Self-serv & Multi-tenants
- 43. Diverse Customer Requirements ● Diverse combination of features ● Diverse platform tradeoffs
- 44. “Change is the Only Constant”
- 45. Change Is the Only Constant ● New streams deployed in a few mins ● Customer changing needs ● Scaling activity ● Infrastructure Upgrades
- 46. Failure Modes ● Infrastructure Disaster ● Any component can become temporarily unavailable
- 49. Declarative Reconciliation ● “Declarative” is a communication pattern ● “Reconciliation” to drive the entire system towards goal
- 50. Declarative Reconciliation ● Goal States ● Current States
- 52. Declarative Reconciliation ● Goal States ● Current States ● State Machine Driven Reconciliation
- 53. State Machine Goal State Driver
- 55. Single Source of Truth Allows Eventual Consistency Convergence on Goal State
- 56. Single Source of Truth Allows Eventual Consistency Convergence on Goal State
- 57. Principle: Leverage Reusable Building Blocks Declarative Reconciliation Single Source of Truth
- 58. Thought Experiment: Kitchen Management vs Distributed Architecture?
- 59. Thank you. References: https://medium.com/netflix-techblog https://www.confluent.io/kafka-summit-sf17/multitenant-multicluster-and-hieracrchical-kafka- messaging-service We’re hiring - http://bit.ly/NetflixSPaaS @ZhenzhongXu (tweet me questions!)
Editor's Notes
- #3: Hook: kitchen management vs distributed system architecture
- #5: Microservice - separation of concern Communication style - declarative reconciliation
- #8: With a large scale microservice architecture supporting more than 100MM subscribers worldwide. Almost any internet-connected screens are capable of generating events. These amazing amount of events are used to ...
- #11: Microservice, empowers local decisions without having to consult to a centralized decision making process. Connect silos, break technical, organizational barriers
- #13: Data backbone for the company to ensure event data collection, movement and computation. Describe what is event data.
- #14: Control plane vs data plane Chef example, management/orchestration vs heavy lifting/make the work happen.
- #18: Describe router and stream processing platform. High level non-engine specific api, vs implement job on apis. Talk about two different product offerings, and why fronting kafka need to be protected. Typical latencies.
- #20: Cloud abstraction. Network partition.
- #21: Data pipeline routing. High level DSL. Define source, transformation (projection, filtering) and sink location. Briefly mention embarrassingly parallel processing vs more complex processing with shuffling. (locality, operator chaining vs large local states, managing large checkpoints). And we ARE only focusing on embarrassingly parallel case.
- #22: Implemented on Apache Flink. Scaling. At least once delivery. Tolerate failure.
- #23: Parallelism/Partitioning isolated across components. Back pressure handling in parallel job. Checkpoint, delivery semantics. Fined grained recovery.
- #24: One service does one thing only. Do it really well. Expand on states. Bug fixes Platform Upgrades Kafka Stream Processing Engine Runtime Provisioning & Deployment Failover Scaling
- #25: Immutable infrastructure on container. Talk about actual scheduling, bin-packing.
- #28: Side effect of processing is identical to exactly once. Not apply to external systems.
- #29: Monitoring, mention external monitor of committed offsets.
- #30: Fanout sinks. Support both batch and custom stream processing jobs.
- #32: Each stream is deployed as separate clusters. Trade offs.
- #36: Cluster differs in size, names, and recoverability. Pet vs Cattle.
- #37: From Failover to failback to failover only.
- #38: From Failover to failback to failover only.
- #39: From Failover to failback to failover only.
- #40: Cluster differs in size, names, and recoverability. Pet vs Cattle.
- #41: From Failover to failback to failover only.
- #43: Cloud abstraction. Network partition.
- #46: Constant Changes => Constant Infrastructure out of sync with requirements Requirement differs => Give customer option for different tradeoffs Topic move
- #47: Customer new streams - deployed in a few mins Customer requested scaling activity - involves topic move. Infrastructure failure Detected scale change Infrastructure upgrades Constant Changes => Constant Infrastructure out of sync with requirements Requirement differs => Give customer option for different tradeoffs
- #49: Why kitchen orders are immutable/append only? talk about benefits: scale, eventual consistent, less human interaction with declarative reconciliation approach
- #52: What customer need to know, what they don’t (topic move example) Separation of concern, break source of truth into lower level goal states that each individual component is authoritative for. Goal State Current State exchange. Goal states are typically expressed in a way that’s “idempotent”.
- #53: Why orders and contracts are append only (immutable)?
- #58: Building blocks - scale your operation. Declarative reconciliation allows eventual convergence on customer goals in a fast changing environment. Single source of truth to drive reconciliation flow and tolerate to total infrastructure failures by persisting only the essential information.