Real-Time Attribution with Structured Streaming and Databricks Delta with Caryl Yuhas
2 likes596 views
Caryl Yuhas discusses real-time attribution using Databricks Delta, highlighting tools for creating low-latency attribution pipelines. The document covers the challenges of attribution, including handling massive data volumes and complexities, and demonstrates an architecture that combines structured streaming and data optimization techniques. In conclusion, it emphasizes a unified approach to batch and streaming data analytics for scalable and flexible performance.
Real-Time Attribution with Structured Streaming and Databricks Delta with Caryl Yuhas
- 1. Caryl Yuhas, Databricks Real-Time Attribution with Structured Streaming and Databricks Delta #ExpSAIS13
- 2. Introduction 2#ExpSAIS13 • Goal: Provide tools and information that can help you build more real-time / lower latency attribution pipelines • Crawl, Walk, Run: Pull Model Carylpreviously MediaMath / SE / PM for Attribution, SA for Databricks
- 3. Getting Started 3#ExpSAIS13 • What is Attribution? Image Source: www.mediamath.com
- 4. Introduction What is Databricks Delta? Delta is a data management capability that brings data reliability and performance optimizations to the cloud data lake. 4#ExpSAIS13
- 5. Stream-to-Sink BEFORE 5#ExpSAIS13 Reprocessing Data Lake λ-arch λ-arch Streaming Analytics Reporting Events Validation λ-arch Validation Reprocessing Compaction Partitioned Compact Small Files Scheduled to Avoid Compaction 1 2 3 1 1 3 4 4 4 2
- 7. Attribution in Practice 7 impressions conversionsJOIN #ExpSAIS13 attributed impressions
- 8. Attribution Challenges Scale • Often dealing with millions to billions of data points per attribution window Complexity • Simple, last-click model is still common • MTA and more sophisticated attribution on rise 8#ExpSAIS13
- 9. High Level Attribution Pipeline 9#ExpSAIS13
- 10. Attribution in Practice 10 impressions conversionsJOIN #ExpSAIS13 attributed impressions
- 11. Data Architecture 11#ExpSAIS13 impression stream conversion stream conversions table impressions table attributed table last touch attributed table weighted attribution views (filters, logic, etc.)
- 13. Unification of Streaming + Batch DEMO 13#ExpSAIS13
- 14. • How can we optimize performance? • Levers: – Delta Tools • Optimize • ZOrder • Caching • Data Skipping – Join on Stream – Cluster Size Managing Performance 14#ExpSAIS13
- 15. Handling Complexity • Flexibility with Complex Logic – Forking streams – Logic on query vs. in-stream • Late or Corrected Data – Upserts – Views automatically update when raw data changed 15#ExpSAIS13
- 16. Conclusion • Unification of Batch & Streaming • Easy APIs for Managing Performance • Flexible and Scalable Analytics on Near Real-Time Data 16#ExpSAIS13