Prassnitha Sampath - Real Time Big Data Analytics with Kafka, Storm & HBase - NoSQL matters Dublin 2015
0 likes1,765 views
The document discusses Groupon's real-time data infrastructure and its use of big data analytics for deal personalization. It outlines challenges and solutions in scaling data storage and processing, particularly focusing on technologies like HBase, Storm, and Redis. The final design emphasizes the need for real-time analytics, data de-duplication with bloom filters, and robust backup/recovery strategies to manage user data effectively.
Prassnitha Sampath - Real Time Big Data Analytics with Kafka, Storm & HBase - NoSQL matters Dublin 2015
- 1. Relevance - Deal Personalization and Real Time Big Data Analytics Prassnitha Sampath psampath@groupon.com
- 2. About Me • Lead Engineer working on Real Time Data Infrastructure @ Groupon • Graduate of Portland State and Madras University
- 3. What are Groupon Deals?
- 5. Scaling: Keeping Up With a Changing Business 2014 2011 2012 Growing Number of deals Growing Users • 100 Million+ subscribers • We need to store data like, user click history, email records, service logs etc. This is billions of data points and TB’s of data
- 6. Changing Business: Shift from Email to Mobile • Growth in Mobile Business • Reducing dependence on email markeOng 100 Million+ App Downloads
- 7. Deal Personalization Infrastructure Use Cases Deliver Personalized Emails Deliver Personalized Website & Mobile Experience Offline System Online System Email Personalize billions of emails for hundreds of millions of users Personalize one of the most popular e-‐commerce mobile & web app for hundreds of millions of users & page views
- 8. Deal Personalization Infrastructure Use Cases Deliver Personalized Website, Mobile and Email Experience Deal Performance Understand User Behavior Deliver Relevant Experience with High Quality Deals
- 9. Earlier System Offline PersonalizaOon Map/Reduce Data Pipeline (User Logs, Email Records, User History etc) Online Deal PersonalizaOon API MySQL Store Email
- 10. Earlier System Email Offline PersonalizaOon Map/Reduce Data Pipeline Online Deal PersonalizaOon API MySQL Store • Scaling MySQL for data such as user click history, email records was painful unless we shard data • Data Pipeline is not “Real Time”
- 11. Email Offline PersonalizaOon Map/Reduce Real Time Data Pipeline Online Deal PersonalizaOon API Ideal Data Store • Common data store that serves data to both online and offline systems • Data store that scales to hundreds of millions of records • Data store that works well with our exisOng Hadoop based systems • Real Time pipeline that scales and can process about 100,000 messages/ second Ideal System
- 12. Email Offline PersonalizaOon Map/Reduce Web Site Logs Online Deal PersonalizaOon API HBase Final Design Mobile Logs Ka`a Message Broker Storm
- 13. Two Challenges With HBase HBase How to scale 100,000 writes/ second? HBase • How to run Map Reduce Programs over HBase without affecOng read latency? • How to batch load data in HBase without affecOng read latencies?
- 14. Final Hbase Design Real Time HBase Batch HBase Bulk Load data via HFiles ReplicaOon Map Reduce Over HBase
- 15. Leveraging System for Real Time Analytics Various requirements from relevance algorithms to pre-‐ compute real 6me analy6cs for be9er targe6ng Category Level MulOdimensional Performance Metrics Deal Level Performance Metrics How do women in Dublin convert for Pizza deals? How do women in Dublin convert for a parOcular pizza deal?
- 16. Leveraging System for Real Time Analytics More Complex Examples Category Level MulOdimensional Performance Metrics Deal Level Performance Metrics How do women in Dublin from the Dundrum area aged 30-‐35 convert for New York Style Pizza, when deal is located within 2 miles, and when deal is priced between €10-‐€20? How do women in Dublin from Dundrum area aged 30-‐35 convert for a parOcular deal?
- 17. Leveraging System for Real Time Analytics Even More Complex Examples How do women in Dublin from the Dundrum area aged 30-‐35 who also like acOviOes like Biking and are acOve customers on our mobile plahorm convert when deal is located within 2 miles, and when deal is priced between €10-‐€20? How do women in Dublin from the Dundrum area aged 30-‐35 who also like acOviOes such as biking and are acOve customers of Groupon deals on mobile plahorm convert for this parOcular deal?
- 18. Power of Simple Counting Turns out all earlier quesOons can be answered if we could count appropriate events in appropriate bucket No Deal Impressions by Women in Dublin for Pizza Deals No of Purchases by Women in Dublin for Pizza Deals Conversion rate for pizza deals for women in Dublin =
- 19. Real Time Analytics Infrastructure Ka`a Topic – With Real Time User events Storm – Running AnalyOcs Topology Real Time infrastructure processing 100,000 requests/ second Redis 1 … Storm Topology calculaOng various dimensions/ buckets and updates appropriate Redis bucket. Redis is sharded from client side Redis cluster handles over 3 Million events per second. Stores over 14 Billion unique keys Redis 2 Redis N
- 20. Real Time Analytics Infrastructure - Explained Ka`a Topic – With Real Time User events Read user event Data from Ka`a Find out which all buckets this event falls Increase event counter for appropriate bucket in Redis Redis Shards Storm
- 21. Scaling Challenges - Kafka - Storm • Storm was hard to scale. We had to try various number of combinaOons to finalize how many bolts of each type are required for steady state operaOons and overall how many workers are needed. • Use “topology.max.spout.pending” senng in Storm topologies. We found it to be very useful to shield your topologies from sudden surge in traffic. • Build your enOre infrastructure – where data duplicates are allowed
- 22. Scaling Challenges - Redis • Reduce memory footprint – use hashes. Very memory efficient compared to normal Redis keys • In order to support high write operaOons turned off AOF, turned on RDB backups Easiest of all other infrastructure pieces – Ka`a, Storm, HBase
- 23. When Small is Big – Bloom Filters • Since both Ka`a and Storm can send same data twice specially at scale, it was important to build downstream infrastructure that can handle duplicate data. • However, by very nature AnalyOcs Topology (CounOng Topology) cannot handle duplicates • Storing individual messages for billions of messages is way too expensive and would take lot more memory • So we used bloom filters. At a very small % error rate, we could effecOvely de-‐dupe data with a very small memory footprint.
- 24. Avoiding Errors – Backups/ Recovery Strategy For a high volume system, which also drives so much revenue for the company good backup/recovery strategy is necessary Redis RDB Backups every few hours. RDB backups are stored in HDFS for later use HBase HBase Snapshot funcOonality is used. Snapshot are taken every few hours. Ka`a/ Storm All input into Ka`a topic is stored in HDFS for 30 days. So any hour/ day can be replayed from HDFS if necessary.
- 25. Monitoring Overall end-to-end monitoring to test the complete flow of data Ka`a -‐> Storm -‐> HBase Pipeline Crawler crawls the page and monitoring looks for corresponding data in HBase
- 26. psampath@groupon.com www.groupon.com/techjobs Ques6ons? Thank you! Slides prepared in collabora/on with Ameya Kanitkar