Big Data Pipeline and Analytics Platform Using NetflixOSS and Other Open Source Libraries
Download as PPTX, PDF23 likes8,122 views
Netflix collects over 100 billion events per day from over 1000 device types and 500 apps/services. They built a big data pipeline using open source tools like NetflixOSS, Hadoop, Druid, Elasticsearch, and RxJava to ingest, process, store, and query this data in real-time and perform tasks like intelligent alerts, distributed tracing, and guided debugging. The system is designed for high throughput and fault tolerance to support a variety of use cases while being simple for message producing and consumption. Developers are encouraged to contribute to improving the open source tools that power Netflix's data platform.
Big Data Pipeline and Analytics Platform Using NetflixOSS and Other Open Source Libraries
- 1. Sudhir Tonse (@stonse) Danny Yuan (@g9yuayon) Big Data Pipeline and Analytics Platform Using NetflixOSS and Other Open Source Software
- 3. Data Is the most important asset at Netflix
- 4. If all the data is easily available to all teams, it can be leveraged in new and exciting ways
- 5. ~1000 Device Types ~500 Apps/Web Services ~100 Billion Events/Day 3.2M messages per second at peak time 3GB per second at peak time Dashboard
- 6. Type of Events • User Interface Events • Search Event (‘Matrix’ using PS3 …) • Star Rating Event (HoC : 5 stars, Xbox, US, …) • Infrastructural Events • RPC Call (API -> Billing Service, ‘/bill/..’, 200, …) • Log Errors (NPE, “Movie is null”, …, …) • Other Events …
- 7. Making Sense of Billions of Events
- 16. We Want to Process App Data in Hadoop
- 23. @NetflixOSS Big Data Tools
- 24. Hadoop as a Service
- 25. Pig Scripting on Steroids
- 26. Pig Married to Clojure “Map-Reduce for Clojure”
- 27. S3MPER S3mper is a library that provides an additional layer of consistency checking on top of Amazon's S3 index through use of a consistent, secondary index. S3mper is a library that provides an additional layer of consistency checking on top of Amazon's S3 index through use of a consistent, secondary index.
- 28. Efficient ETL with Cassandra Cassandra
- 29. Offline Analysis
- 31. We Want to Aggregate, Index, and Query Data in Real Time
- 33. Let’s walk through some use cases
- 34. client activity event * /name = “movieStarts”
- 35. Pipeline Challenges • App owners: send and forget • Data scientists: validation, ETL, batch processing • DevOps: stream processing, targeted search
- 36. Message Routing
- 38. We Want to Consume Data Selectively in Different Ways
- 40. • Message broker • High-throughput • Persistent and replicated
- 41. There Is More
- 44. Guided Debugging in the Right Context
- 45. Guided Debugging in the Right Context
- 46. Guided Debugging in the Right Context
- 47. • Ad-hoc query with different dimensions • Quick aggregations and Top-N queries • Time series with flexible filters • Quick access to raw data using boolean queries What We Need
- 48. Druid • Rapid exploration of high dimensional data • Fast ingestion and querying • Time series
- 49. • Real-time indexing of event streams • Killer feature: boolean search • Great UI: Kibana
- 50. The Old Pipeline
- 51. The New Pipeline
- 53. There Is More
- 54. It’s Not All About Counters and Time Series
- 56. RequestId Parent Id Node Id Service Name Status 4965-4a74 0 123 Edge Service 200 4965-4a74 123 456 Gateway 200 4965-4a74 456 789 Service A 200 4965-4a74e 456 abc Service B 200 Status:200
- 60. A System that Supports All These
- 61. A Data Pipeline To Glue Them All
- 62. Make It Simple
- 63. Message Producing • Simple and Uniform API • messageBus.publish(event)
- 64. Consumption Is Simple Too consumer.observe().subscribe(new Subscriber<>() { @Override public void onNext(Ackable<IncomingMessage> ackable) { process(ackable.getEntity(MyEventType.class)); ackable.ack(); } }); consumer.pause(); consumer.resume()
- 65. RxJava • Functional reactive programming model • Powerful streaming API • Separation of logic and threading model
- 66. Design Decisions • Top Priority: app stability and throughput • Asynchronous operations • Aggressive buffering • Drops messages if necessary
- 68. Cloud Resiliency
- 69. Fault Tolerance Features • Write and forward with auto-reattached EBS (Amazon’s Elastic Block Storage) • disk-backed queue: big-queue • Customized scaling down
- 71. There’s More to Do • Contribute to @NetflixOSS • Join us :-)
- 73. You can build your own web-scale data pipeline using open source components
- 74. Thank You! Sudhir Tonse http://www.linkedin.com/in/sudhirtonse Twitter: @stonse Danny Yuan http://www.linkedin.com/pub/danny- yuan/4/374/862 Twitter: @g9yuayon
Editor's Notes
- #16: Something happened. Our traffic turned into a hockey stick, and the number of applications exploded. So, log traffic also exploded. Simple log scraping wouldn’t cut it any more.
- #33: For one thing: interactive exploration. Sometimes we want to get data in real time so we can act quickly. Some data is only useful in a small time window after all. Sometimes we want to perform lots of experimental queries just to find the right insights. If we wait too long for a query back, we won’t be able to iterate fast enough. Either way, we need to get query results back in seconds.
- #38: Here is one example: we process more than 150 thousand events per second about user activities. What if we’d like to know the geographically how many users started playing videos in the past 5 minutes? So I submit my query, and in a few seconds.... But this is an aggregated view. What if I want to drill down the data immediately along different dimensions? In this particular case, to find out failed attempts on our SilverLight players that run on PCs and Macs?
- #43: Note this is different from alerting based on monitoring metrics. Monitoring metrics are great and versatile. But it doesn’t help us catch unexpected errors. When we build an application, we instrument our code diligently, yet it’s very likely we miss some critical instrumentation points. There’s one thing that we always catch, though: logged errors and unhandled exceptions. It’s about The alert provides a precise entrypoint and the right context for people to drill down the right problems
- #44: Note this is different from alerting based on monitoring metrics. Monitoring metrics are great and versatile. But it doesn’t help us catch unexpected errors. When we build an application, we instrument our code diligently, yet it’s very likely we miss some critical instrumentation points. There’s one thing that we always catch, though: logged errors and unhandled exceptions. It’s about The alert provides a precise entrypoint and the right context for people to drill down the right problems