Real time pipeline at terabyte sacle
Download as PPTX, PDF2 likes1,550 views
The document outlines ShareThis's capabilities in processing over 1 billion social signals monthly to provide actionable insights regarding brands and industries. It discusses the need for real-time data access, addressing issues with previous architectures and emphasizing improved data quality and accessibility through centralized systems. Future developments include dynamic stream filters to tailor data offerings based on customer requirements.
Real time pipeline at terabyte sacle
- 1. Real-Time Applications At Terabyte Scale Isaac Mosquera VP Engineering, Data & Insights
- 2. You’ve probably seen our sharing tools... But that’s not all we do...
- 3. WE MAKE SOCIAL DATA ACTIONABLE Over 1B social signals are processed monthly by the ShareThis Social Intelligence Platform™ to generate insights about your brand, industry and events. ENGAGEMENT Users consume and share content across web and mobile TARGETING Desktop and mobile targeting at scale INSIGHTS Actionable cross-device insights DATA 1B+ first party Social Actions Monthly
- 4. ENGAGEMENT TARGETING INSIGHTS DATA • Lookalike Audiences • Audience Segments “Wow small SUVs are fuel efficient!” User #12345 • Automotive Study • Car Buying Infographic
- 5. Why Is Real-Time Important? Time Sharing Interest Decays With Time
- 7. Previous Architecture Problems Duplicated Data Query Engine Share Data Insights Query Engine Ad Tech Query Engine Consumer Engagement Query Engine Data Science
- 8. Fragmented & Siloed Data Sources Query Engine Share Data Insights Query Engine Ad Tech Query Engine Consumer Engagement Query Engine Data Science Campaign RTB Conversion Summarization 3rd Party Trends Studies
- 9. Generating Reports From Old Platform Raw Data Pre Aggregation Staged Data Results Consumers Query Rest API New Report Type
- 10. Why Focus On These Problems? Faster Iterations Data Science New Applications Business Value Targeting
- 11. The Birth of a New Team
- 12. Data Team’s Mission Making our data easily accessible Our Data
- 13. Vision Centralize Data sources Data Quality & Trust Reliable Infrastructure
- 14. Real Time All The Things Raw Social Data DLX Geo Device Mappings SentimentSocial Keywords Downstream Applications
- 15. Kafka Architecture Data ScienceApplication Data ScienceLogs Data ScienceProducers Data ScienceApplication Data ScienceLogs Data ScienceProducers Brokers Data ScienceConsumers Data Loaders Data ScienceAnnotations Data ScienceFilters Destinations Big QuerySocial Ad Tech
- 16. Integrate Campaign Social Data DLX Geo Device Mappings SentimentSocial Keywords RTB Bid Data Campaign Data Downstream Applications
- 17. Build An Active Warehouse 3 Trillion Row Interactive Query Engine Share Data Data Science Ad Tech Consumer Engagement Sales Strategy Insights RTB ImpressionS & Clicks + RT External Data Science Data Science ATDs Data Scienc e DMPs DSPs Internal Google Big Query
- 18. Add in redundancy and robustness into our data pipeline that protects us against data loss. Reliability
- 19. Unified Monitoring Centralizing monitoring allows us to have a singular definition of “data quality”
- 20. Monitoring Infrastructure Consumer App Metrics Library Producer App Metrics Library Graphite Slack Dev Team Seyren Dashboards
- 21. Defining Data Quality Expected Field Distribution Data Loss Business KPIs
- 22. What’s Next?
- 23. Dynamic Stream Filter You Want This But You Get This
- 24. Stream Sources Filter Application Data Filter UI Filter Definitions
- 25. Data Stream Filter Prototype Real Time Pipeline shares from top 100 domains user actions in north east region users who recently bought car user likely to buy a car soon actions from user ids in (1234, 5432, 9999) Data Science External Customers Data ScienceInternal Teams Predictive Algorithms Dynamically create filters based on customer’s needs. These can be created instantly on-demand.
- 26. Questions? Isaac Mosquera twitter: imosquera e-mail: isaac@sharethis.com