Democratizing data science Using spark, hive and druid
Download as PPTX, PDF3 likes982 views
The document outlines a data science framework utilizing Apache Spark, Hive, and Druid to enhance marketing strategies for a game studio. It details processes for data ingestion, normalization, model generation, and optimization for ad campaigns, significantly improving operational efficiency and performance metrics. The framework incorporates machine learning pipelines and real-time data processing to support large-scale ad marketing across multiple channels globally.
Democratizing data science Using spark, hive and druid
- 1. Democratizing Data Science using Apache Spark, Hive and Druid
- 2. ● Pushkar Priyadarshi ● Igor Yurinok ● Michael Dreibelbis Intro
- 3. ● Game studio produces massive mobile games that break down linguistic and geographic barriers by uniting an unprecedented number of global players in one gaming world. Games are played in 180+ countries. ● Performance marketing platform Cognant enables marketing for our internal games as well as external businesses over 250+ channels. It merges extensive mobile ad buying expertise with a live data platform to deliver not only true ROI on mobile marketing spend but eliminate endless fraud and tiresome make-goods in the process. Machine Zone(mz.com)
- 4. ● 40 billion messages/day ● kafka cluster handling 250+ topics over 4k partitions ● 3 hadoop clusters largest one spanning 300 nodes ● 5 PB of unreplicated data in hadoop eco system ● Ads published on 100k apps in nearly 200 countries serving average 750 millions impression a day peaking at 1B/day ● Data from 300 distinct sources ● Druid cluster containing 30+ data sources holding 50 TBs of data Data @ MZ
- 5. ● Data Ingestion ○ Ingest raw data from external entities ● Data Normalization ○ Normalize data using transformation framework ● Model Generation ○ Create Model using model generation framework ● Generate predictions ● Second layer of Intelligence ○ Campaign Initialization ○ Campaign Optimization ● Data Service Framework Overview
- 7. Data Ingestion (cont’d) ● DataReaders extract data from various types of sources ○ S3 - Amazon S3 bucket accessed reporting data ○ REST - HTTP endpoint reporting data ○ FTP Similar to S3, loads from FileSystem ○ Email - Scan inbox and extract valid reports ● DataWriters output data to HDFS ○ HIVE external tables
- 8. Data Normalization RT RAW Rules Loader Rules Store HDFS Rules Parser Rules Applier Druid Rule Based Transformation Engine
- 9. ● Streaming Real time data source ○ Kafka + Spark Streaming => Tranquility => Druid ● Batch historical backfill raw data source ○ Spark => Druid ● Rule based transformation engine (normalizer) ○ Built using Apache Spark ○ Custom DDL for defining column transformation rules Data Normalization (cont’d)
- 10. ● Machine Learning Pipeline based on Apache Spark ML ○ Feature Engineering ○ Model Training ○ Predictions ○ Model Testing/Tuning ○ Model Deployment MLPlatform
- 11. ● Feature Engineering extensions ○ Aggregator => NumericAggregator ● Perform aggregate transformation on input Dataset MLPlatform (cont’d)
- 12. ● Feature Engineering extensions ○ ParallelCountVectorizer ■ Compute CountVectorizer per input column ○ ParallelIDF ■ Compute IDF per input column MLPlatform (cont’d)
- 13. ● Feature Engineering extensions ○ DAGPipeline ■ Support multi-input dataset DAG based feature extraction MLPlatform (cont’d) n1 n4 n3 n2 DAGModel generated:
- 14. ● Model Testing/Tuning ○ Feature Store ■ Rapid iterative model testing ○ Configurable Split-Testing ○ Model Store ■ Based on SparkML MLWritable ● Predictions ○ Can be generated using any version of model ○ Compared across model implementations MLPlatform (cont’d)
- 15. ● Predictions using Apache Zeppelin based visualization layer ○ Notebooks allow for rapid testing and model iteration ○ Graphing library allows for instant visual feedback MLPlatform (cont’d)
- 16. What is output from ML Models? ● Predictions What is business value of it? ● Not much What does business need? ● Translate predictions in ad partner instructions Second Layer of Intelligence
- 17. Partner instruction is a command which partner can/should execute: ● Create a new campaign ● Update Budget ● Update Bid ● Update Targeting ● Update Creative Asset What is Partner Instructions?
- 18. Campaign Initialization: ● Bid ○ Finds the best possible bid to create campaigns ● Budget ○ Splits total budget between partners ● Targeting ○ Generates sets of possible targeting groups (Gender, Age, GEO) ● Creative ○ Generates and assign creatives Campaign Initialization Process
- 19. Campaign Optimization: ● Bid ○ Increase, Decrease bids per campaign based on performance prediction ● Budget ○ Increase, Decrease and Reshuffle budget across partners/campaigns ● Targeting ○ Update targeting based on performance ● Creative ○ Reassign creatives based on performance Campaign Optimization Process
- 20. Campaign InitializationOptimization Process ML Output (Predictions) Historical Data Initializer Optimizer Ad Partner Instructions Ad Partner
- 21. Where to store metadata for Data Pipelines? Where to store Ad Partner Instructions? How to deliver Ad Partner Instructions? Data Service Framework
- 22. Possible Microservices: ● Ad Partner Data Service ● Campaign Data Service ● ASP Data Service ● Ad Partner Instruction Service Data Service Framework (cont’d)
- 23. Technologies: ● REST API ● Spring Boot ● Openshift Kubernetis ● Gradle + Jenkins Pipelines for CI/CD Data Service Framework (cont’d)
- 24. Connect All Components Together Data Ingestion Data Normalization MLPlatform Ad PartnerData Services
- 25. Questions???