Discovery & Consumption of Analytics Data @Twitter
- 1. Discovery & Consumption of Analytics Data @Twitter @sirkamran32 ADP Team November 17, 2016
- 2. Sriram Krishnan Arash Aghevli Joseph Boyd @sluicing@aaghevli@krishnansriram Dave Marwick Kamran Munshi Alex Maranka @sometext42@sirkamran32@dmarwick
- 3. 1.Analytics Data @Twitter 2.Analytics Data Platform (ADP) Stack Overview 3.Deeper Dive (EagleEye Demo) 4.Deeper Dive (DAL Design & Concepts) 5.Observations & Lessons learned 6.Future Work Agenda
- 4. Analytics Data @Twitter Mostly resides in Hadoop •We manage and run several LARGE Hadoop clusters •Some of Twitter’s Hadoop clusters are the largest of their kind •> 10K nodes storing more 100s of pb on HDFS Includes important business data •e.g. logs, user data, recommendations data, publicly reported metrics, A/B testing data, ads targeting and much more Lots of it processed and analyzed in BATCH fashion using… •Scalding (for ETL, data science, and general analytics) •Presto (for more interactive querying) •Vertica, Tableau, Zeppelin, MySQL (for analysis) All in all there are more than 100K daily jobs processing 10s of pbs/day.
- 5. Analytics Data Software Goals Data Discovery • Important datasets • Owners • Semantics & relevant metadata Data Abstraction • Logical datasets to abstract away physical data location and format 1 3 Data Auditing • Creators and consumers • Dependencies • Alerts 2 Data Consumption • How to consume data via various clients (scalding, presto, etc) ? 4
- 6. How our tools fit into data platform ADP Stack Data Explorer (EagleEye) Discovery & Consumption Data Abstraction Layer (DAL) Data Abstraction Alerting Service Alerting 1 2 3 State Management Service Scheduling 4 Source
- 7. Data Explorer (aka Eagle Eye)
- 8. An internal web UI to make it easy to search, discover, track and manage batch applications AND analytics datasets. EagleEye
- 9. View dataset schemas, notes, health, physical locations, & owners EagleEye Cont’d
- 10. Set up alerts and view lineage & dependencies EagleEye Cont’d
- 11. Data Abstraction Layer (aka DAL)
- 12. DAL helps you… Discover and consume data easily • DAL stores and exposes metadata, schemas, and consumption information for datasets (in HDFS, Vertica, and MySQL) Understand your dependencies •DAL tracks which applications produced or consumed datasets, allowing you to visualize your up and downstream dependencies in EagleEye (and set up alerts) Future proof your data and avoid lengthy migration work • DAL abstracts away physical data locations/formats from consumers by informing them about datasets at runtime, allowing jobs to read data that has changed locations without a redeploy. Enable error rate resiliency without redeploys •DAL centrally manages error thresholds which avoids hardcoded values that may require rebuild/redeploy of jobs Easily explore your data - DAL CLI allows users to easily view data without writing jobs/scripts
- 13. DAL Models Data & Applications Data Logical Dataset - a symbolic name (a tuple of role, simple name, and environment) with a schema and segment type associated with it Physical Dataset - describes the serialization of a Logical Dataset to a specific physical Location (i.e. hadoop-cluster, vertica-cluster, etc) Segments - where the data actually resides. Segments model all the information necessary to deserialize them like their physical URL, schema, and format used upon serialization. They also maintain state to indicate if the segment is ready for consumption, rolled-back, deleted, etc. •Partitioned Segment - a subset of a dataset for a given time interval •Snapshots Segment - an entire physical dataset up to a point in time Batch Applications DAL can be used by all batch applications, including ad-hoc jobs not managed by Statebird •Statebird apps have a PhysicalRunContext (the batch application, batch increment and statebird run id)
- 14. DAL Architecture Thrift service on top of a MySQL db. •Written in Scala using Slick as the ORM •Deployed on Mesos (like most Twitter services) DAL Clients (e.g. Scalding jobs) register the production and consumption of data segments via DAL APIs (PUSH MODEL): • DAL.read(TweetEventsScalaDataset) • DAL.writeDAL(DalWriteAuditTestDataset, D.Hourly, D.Suffix(args("output")), D.Parquet) Scalding jobs also communicate with Statebird to: • Poll for dataset updates and find out when the job is supposed to run • Find out if dependent jobs have run
- 15. DAL Auditing Flow Listener Registers reads & writes by jobs that don't use the DAL API directly. • A custom flow listener is built into all scalding jobs at Twitter by default • This is limited but useful for those wanting view data via EagleEye • NOTE: This audit information doesn't include full details of data format and physical location, and is used only to surface lineage/provenance in EagleEye
- 16. Why not just Hive Metastore? Value-added features such as… •A single service for both HDFS and non-HDFS data •Auditing and lineage for datasets •Example usage •A place to add additional future features Single DAL service VS. multiple metastores… •DAL is a singular service with one back-end, avoiding the need for multiple metastores in each DC
- 17. •Simplifying data consumption across ALL data formats and storage systems is critical as the size of your organization and datasets grow •A metadata service is something that every data platform needs and should be built early on •DAL-integration is now the first order of business for every new processing tool @Twitter Observations & Lessons
- 18. •Integrate retention and replication information into DAL •Helps to further EagleEye/DAL a ‘one-stop-shop’ for managing datasets •Expose Hive compatible API •Currently focused on BATCH applications. Investigate STREAMING applications as well •“Janitor monkey” to poll DAL and clean up datasets that are not being used •Open source? Future Work