Keystone - Leverage Big Data 2016
- 1. The Netflix way to deal with real-time data How we built a 1t/day stream processing cloud platform in a year
- 2. What should I expect Keystone Season 1 - Who, What, How and Why Keystone Season 2 - Preview Trailer
- 3. 1. The Who?
- 4. hello! I am Peter Bakas I lead the Real-Time Data Infrastructure team @ Netflix You can find me at @peter_bakas
- 5. 2. The What?
- 6. Publish + Collect + Process + Move Data
- 7. 1,000,000,000,000 Whoa! That’s a big number. Events Processed Every Day
- 8. Daily Averages 700B unique events ingested 1T events processed 1.4 PB By the numbers Peak 1T unique events ingested/day 12.5M/sec 35 GB/sec
- 11. In the beginning - Chukwa
- 12. Q4 2014 - Chukwa/Suro
- 13. Q4 2015 - Keystone Internal Routing Service EMR Fronting Kafka Event Producer Consumer Kafka Control Plane HTTP PROXY Stream Consumers
- 14. Kafka
- 15. Q4 2015 - Keystone Internal Routing Service EMR Fronting Kafka Event Producer Consumer Kafka Control Plane HTTP PROXY Stream Consumers
- 16. Keystone Kafka Footprint Fronting Kafka Consumer Kafka Number of Clusters 24 8 Number of Instances 3000+ 900+ Retention Period 8 to 24 hrs 2 to 4 hrs
- 17. Q4 2015 - Keystone Internal Routing Service EMR Fronting Kafka Event Producer Consumer Kafka Control Plane HTTP PROXY Stream Consumers
- 18. Keystone Internal Routing Service + Checkpointing Cluster +
- 19. Keystone Internal Routing Service
- 20. Keystone Internal Routing Service Footprint S3 ElasticSearch Consumer Kafka Number of containers 7000 1500 4500
- 21. Keystone avg end-to-end metrics S3 ElasticSearch Consumer Kafka 1 sec 13 sec 800 ms
- 22. 3. The How?
- 23. Netflix Culture Freedom and Responsibility
- 24. “It may well be the most important document to ever come out of the Valley Sheryl Sandberg, COO @ Facebook
- 25. What does culture have to do with how?
- 27. A true story Keystone went live 10/27/15 2 days later...
- 28. Place your screenshot here 80% “loss” over 6 hour period A true story
- 29. Lessons learned There are times when things can go wrong… and no turning back Reduce complexity Minimize blast radius Find a way to start over fresh
- 30. Failover Cold standby Kafka cluster with different instance type Different ZooKeeper cluster with no state Fully automated
- 31. Place your screenshot here Time is of the essence Failover as fast as 5 minutes Fully Automated Failover
- 32. Best Practices Full Automation Self Healing Kafka Kong
- 33. 4. The Why?
- 34. “We didn’t do anything wrong, but somehow, we lost... Stephen Elop, Nokia CEO
- 36. 125,000,000 hrs/day That’s a lot of hours! 37 % of North America internet traffic @ peak! 81,000,000 members and a lot of members
- 37. If you don’t change, you will be eliminated from the competition
- 38. 5. Coming Soon
- 39. Our philosophy Create Duplo R Blocks : Let reusability drive new value
- 41. Keystone Unified event publishing, collection, routing for batch and stream processing 85% of data volume Keystone Messaging Ad-hoc Messaging 15% of data volume
- 42. Consumers weary of Complexity of self-managed infrastructure Multiple runtimes across different platforms Keystone Stream Processing Consumers want Simple unified model/API/UI/system
- 43. Simple and intuitive interface to manage all Keystone services Keystone Management
- 44. thanks! Any questions? You can find me at @peter_bakas pbakas@netflix.com
- 45. Credits Special thanks to all the people who made and released these awesome resources for free: Presentation template by SlidesCarnival Photographs by Unsplash