Big Data
- 1. Big Data Use Cases in the cloud Peter Sirota, GM Elastic MapReduce @petersirota
- 2. What is Big Data?
- 3. Computer generated data ď§ Application server logs (web sites, games) ď§ Sensor data (weather, water, smart grids) ď§ Images/videos (traffic, security cameras)
- 4. Human generated data ď§ Twitter âFirehoseâ (50 mil tweets/day 1,400% growth per year) ď§ Blogs/Reviews/Emails/Pictures Social graphs ď§ Facebook, linked-in, contacts
- 5. Big Data is full of valuable, unanswered questions!
- 6. Why is Big Data Hard (and Getting Harder)?
- 7. Data Volume ď§ Unconstrained growth ď§ Current systems donât scale Why is Big Data Hard (and Getting Harder)?
- 8. Why is Big Data Hard (and Getting Harder)? Data Structure ď§ Need to consolidate data from multiple data sources in multiple formats across multiple businesses
- 9. Why is Big Data Hard (and Getting Harder)? Changing Data Requirements ď§ Faster response time of fresher data ď§ Sampling is not good enough and history is important ď§ Increasing complexity of analytics ď§ Users demand inexpensive experimentation
- 10. We need tools built specifically for Big Data!
- 11. Innovation #1: Apache Hadoop ď§The MapReduce computational paradigm ď§Open source, scalable, fault tolerant, distributed systemâ Hadoop lowers the cost of developing a distributed system for data processing
- 12. Innovation #2: Amazon Elastic Compute Cloud (EC2) âprovides resizable compute capacity in the cloud.â Amazon EC2 lowers the cost of operating a distributed system for data processing
- 13. Amazon Elastic MapReduce = Amazon EC2 + Hadoop
- 14. Elastic MapReduce applications Targeted advertising / Clickstream analysis Security: anti-virus, fraud detection, image recognition Pattern matching / Recommendations Data warehousing / BI Bio-informatics (Genome analysis) Financial simulation (Monte Carlo simulation) File processing (resize jpegs, video encoding) Web indexing
- 15. Clickstream Analysis â Big Box Retailer came to Razorfish ď§ 3.5 billion records ď§ 71 million unique cookies ď§ 1.7 million targeted ads required per day Problem: Improve Return on Ad Spend (ROAS)
- 16. Clickstream Analysis â Targeted Ad User recently purchased a sports movie and is searching for video games (1.7 Million per day)
- 17. Clickstream Analysis â Lots of experimentation but final design: ď§ 100 node on-demand Elastic MapReduce cluster running Hadoop
- 18. Clickstream Analysis â Processing time dropped from 2+ days to 8 hours (with lots more data)
- 19. Clickstream Analysis â Increased Return On Ad Spend by 500%
- 20. Worldâs largest handmade marketplace ď§ 8.9 million items ď§ 1 billion page view per month ď§ $320MM 2010 GMS
- 21. ⢠Easy to âbackfillâ and run experiments just boot up a cluster with 100, 500, or 1000 nodes Production DB snapshots Production DB snapshots Web event logs Web event logs ETL â Step 1 ETL â Step 1 ETL â Step 2 ETL â Step 2 JobJob JobJob JobJob
- 22. Recommendations The Taste Test http://www.etsy.com/tastetest
- 23. Recommendations etsy.com/gifts Gift Ideas for Facebook Friends
- 24. ⢠⢠Yelp generates close to 400GB of logs per day Yelp
- 25. ⢠Yelp does not have a physical MapReduce cluster ⢠Running 250 production clusters per week ⢠All of those run on Elastic MapReduce MapReduce at Yelp
- 26. Features driven by MapReduce
- 27. Features driven by MapReduce
- 28. ⢠Analyze ad stats (reporting, billing, algorithm inputs) ⢠Analyze A/B test results ⢠Detect duplicate business listings ⢠Email bounce processing ⢠Identify bots based on traffic patterns More MapReduce uses
- 29. 9/23/2011 Amazon EMR Strata Justin Moore - @injust Big Data @ foursquare
- 30. 9/23/2011 Amazon EMR Strata Justin Moore - @injust How do we use EMR? ⢠Map-Reduce â Run algorithms on our entire dataset â Streaming jobs, complex analyses ⢠Hive â Business intelligence â Exploratory analyses â Infographics!
- 31. 9/23/2011 Amazon EMR Strata Justin Moore - @injust How big is our data? ⢠Global reach (North Pole, Space) ⢠Native app for almost every smartphone, SMS, web, mobile-web ⢠10M+ users, 15M+ venues, ~1B check-ins ⢠Terabytes of log data
- 32. 9/23/2011 Amazon EMR Strata Justin Moore - @injust Our Stack
- 33. 9/23/2011 Amazon EMR Strata Justin Moore - @injust Computing venue-to-venue similarity ⢠Spin up 40 node cluster ⢠Submit Ruby streaming job â Invert User x Venue matrix â Grab Co-occurrences â Compute similarity ⢠Spin down cluster ⢠Load data to app server
- 34. 9/23/2011 Amazon EMR Strata Justin Moore - @injust Who is checking in?
- 35. 9/23/2011 Amazon EMR Strata Justin Moore - @injust What are people doing?
- 36. 9/23/2011 Amazon EMR Strata Justin Moore - @injust Where are our users?
- 37. 9/23/2011 Amazon EMR Strata Justin Moore - @injust When do people go to a place? Thursday Friday Saturday Sunday
- 38. 9/23/2011 Amazon EMR Strata Justin Moore - @injust Why are people checking in? ⢠Explore their city, discover new places ⢠Find friends, meet up ⢠Save with local deals ⢠Get insider tips on venues ⢠Personal analytics, diary ⢠Follow brands and celebrities ⢠Earn points, badges, gamification of life ⢠The list growsâŚ
- 39. 9/23/2011 Amazon EMR Strata Justin Moore - @injust How can we leverage these insights?
- 40. 9/23/2011 Amazon EMR Strata Justin Moore - @injust Join us! foursquare is hiring www.foursquare.com/jobs Justin Moore @injust justin@foursquare.com