Apache Hadoop India Summit 2011 talk "Online Content Optimization using Hadoop" by Shail Aditya
Download as PPTX, PDF1 like718 views
This document summarizes a presentation about using Hadoop for online content optimization at Yahoo. It discusses using machine learning and large-scale data from user interactions to build models that learn users' interests and content attributes to deliver personalized recommendations. Key points include: - Collecting hundreds of GB of user interaction data daily to build user profiles and content models - Storing models and metadata in HBase for fast lookup and updating models every 5-30 minutes - Using Pig and Hadoop jobs to generate features, build recommendation models, and analyze results - A service architecture with HBase, Pig, Hive, and edge services to power large-scale personalized recommendations.
Apache Hadoop India Summit 2011 talk "Online Content Optimization using Hadoop" by Shail Aditya
- 1. Hadoop Summit 2011Online Content Optimization using HadoopShail Adityashailg@yahoo-inc.com
- 2. What do we do ?Deliver right CONTENT to the right USER at the right TIME”
- 3. Effectively and “pro-actively” learn from user interactions with content that are displayed to maximize our objectives
- 4. A new scientific discipline at the interface of
- 5. Large scale Machine Learning and Statistics
- 6. Multi-objective optimization in the presence of uncertainty
- 8. Content understandingContent Relevance at Yahoo!Editorial10s of ItemsImportantEditorsPopularPersonal / SocialScienceMillions of Items
- 9. Content Ranking ProblemsXYMost PopularMost engaging overall based on objective metricsMost Popular + Per User HistoryRotate stories I’ve already seenLight PersonalizationMore relevant to me based on my age, gender, location, and property usageDeep PersonalizationMost relevant to me based on my deep interests (entities, sources, categories, keywords)Related Items and Context-Sensitive ModelsBehavioral Affinity: People who did X, did YMost engaging in this page/section/property/device/referral context?Layout OptimizationWhich modules/ad units should be shown to this user in this context?Revenue OptimizationVoice and Business RulesReal-time Dashboard
- 10. Yahoo FrontpageTrending Now (Most popular)Today Module(Light personalization)PersonalAssistant(LightPersonalization)National News(Most Popular + User History bucket)Deals (most popular)
- 11. Recommendation: A Match-making Problem Recommendation problems
- 12. Search: Web, Vertical
- 14. …Item InventoryArticles, web page, ads, …Use an automated algorithm to select item(s) to showGet feedback (click, time spent,..) Refine the modelsRepeat (large number of times)Measure metric(s) of interest(Total clicks, Total revenue,…)OpportunityUsers, queries, pages, …
- 15. Problem Characteristics : Today moduleTraffic obtained from a controlled randomized experimentThings to note: a) Short lifetimes b) temporal effects c) often breaking news story
- 16. Scale: Why use Hadoop?Million events per second (user view/click, content update)Hundreds of GB data collected and modeled per run Millions of items in poolMillions of user profilesTens of thousands of Features (Content and/or User)
- 17. Data FlowOptimization EngineContent feed with biz rulesRules EngineContent MetadataExploit~99%Explore~1%Near Real-timeFeedbackReal-timeInsightsDashboardOptimized Module
- 18. How it happens ?Additional Content & UserFeature GenerationFeatureGenerationITEM ModelSTORE: HBASE5 minlatencyRequestUserEventsModelingRankingB-RulesSLA 50 ms – 200 ms5 – 30 minlatencyAt time ‘t’ User ‘u’ (user attr: age, gen, loc) interacted withContent ‘id’ atPosition ‘o’Property/Site ‘p’ Section - sModule – mInternational - i’STORE: PNUTSUSER ModelContent ‘id’Has associated metadata ‘meta’meta = {entity, keyword, geo, topic, category}ItemMetadata
- 19. ModelsUSER x CONTENT FEATURES USER MODEL : Tracks User interest in terms of Content FeaturesITEM x USER FEATURES ITEM MODEL : Tracks behavior of Item across user featuresUSER FEATURES x CONTENT FEATURESPRIORS : Tracks interactions of user features with content featuresUSER x USER CLUSTERING : Looks at User-User Affinity based on the feature vectorsITEM x ITEM CLUSTERING : Looks at Item-Item Affinity based on item feature vectors
- 20. Technology StackIngestAnalytics and Debugging
- 21. Modeling FrameworkGlobal state provided by HBase
- 22. Hadoop processing via a collection of PIG UDFs
- 23. Different flows for modeling or stages assembled in PIG
- 24. OLR, Clustering, Affinity, Regression Models, Decompositions (Cholesky…)
- 25. Timeseries models (generally trends – extract of user activity on content)
- 26. Configuration based behavior for various stages of modeling
- 27. Type of Features to be generated
- 28. Type of joins to perform – User / Item / Feature
- 29. Input : DFS and/or HBase
- 30. Output: DFS and/or HBaseHBase ITEM Model
- 31. Stores item related features
- 32. Stores ITEM x USER FEATURES model
- 33. Stores parameters about item like view count, click count, unique user count.
- 34. 10 of Millions of Items
- 35. Updated every 5 minutes
- 36. USER Model
- 37. Store USER x CONTENT FEATURES model for each individual user by either a Unique ID
- 38. Stores summarized user history – Essential for Modeling in terms of item decay
- 40. Updated every 5 to 30 minutes
- 41. TERM Model
- 42. Inverts the Item Table and stores statistics for the terms.
- 43. Used to find the trending features and provide baselines for user features
- 44. Millions of terms and hundreds of parameters tracked
- 45. Updates every 5 minutesGrid Edge Services Keeps MR jobs lean and mean
- 46. Provides ability to control non-gridifyable solutions to be deployed easily
- 47. Have different scaling characteristics (E.g. Memory, CPU)
- 48. Provide gateway for accessing external data sources in M/R
- 49. Map and/or Reduce step interact with Edge Services using standard client
- 50. Examples
- 51. Categorization
- 52. Geo Tagging
- 53. Feature TransformationAnalytics and Debugging Provides ability to debug modeling issues near-real time
- 54. Run complex queries for analysis
- 55. Easy to use interface
- 56. PM, Engineers, Research use this cluster to get near-real time insights
- 57. 10s of Modeling monitoring and Reporting queries every 5 minute
- 58. We use HIVELearnings PIG & HBase has been best combination so far
- 59. Made it simple to build different kind of science models
- 60. Point lookup using HBase has proven to be very useful
- 61. Modeling = Matrices
- 62. HBase provides a natural way to represent and access them
- 63. Edge Services
- 64. Have provided simplicity to whole stack
- 65. Management (Upgrades, Outage) has been easy
- 66. HIVE has provided us a great way for analyzing the results
- 67. PIG was also consideredThank youDeliver right CONTENT to the right USER at the right TIME”Shail Adityashailg@yahoo-inc.com
Editor's Notes
- #2: This is the Title slide.Please use the name of the presentation that was used in the abstract submission.
- #9: This is the agenda slide. There is only one of these in the deck.NOTES:What does X stories to run mean ? Can we be more clear on thatAlso – This should be a more a punch line of what we do. This slide to me is very broad and not clear. Following are the things that I would describeProblem of matching the best content to the interest of a userScale Millions of content slicesMillions of users
- #12: This is the agenda slide. There is only one of these in the deck.NOTES:What does X stories to run mean ? Can we be more clear on thatAlso – This should be a more a punch line of what we do. This slide to me is very broad and not clear. Following are the things that I would describeProblem of matching the best content to the interest of a userScale Millions of content slicesMillions of users
- #13: This is the agenda slide. There is only one of these in the deck.NOTES:What does X stories to run mean ? Can we be more clear on thatAlso – This should be a more a punch line of what we do. This slide to me is very broad and not clear. Following are the things that I would describeProblem of matching the best content to the interest of a userScale Millions of content slicesMillions of users
- #14: This is the agenda slide. There is only one of these in the deck.NOTES:What does X stories to run mean ? Can we be more clear on thatAlso – This should be a more a punch line of what we do. This slide to me is very broad and not clear. Following are the things that I would describeProblem of matching the best content to the interest of a userScale Millions of content slicesMillions of users
- #15: This is the agenda slide. There is only one of these in the deck.NOTES:What does X stories to run mean ? Can we be more clear on thatAlso – This should be a more a punch line of what we do. This slide to me is very broad and not clear. Following are the things that I would describeProblem of matching the best content to the interest of a userScale Millions of content slicesMillions of users
- #16: This is the agenda slide. There is only one of these in the deck.NOTES:What does X stories to run mean ? Can we be more clear on thatAlso – This should be a more a punch line of what we do. This slide to me is very broad and not clear. Following are the things that I would describeProblem of matching the best content to the interest of a userScale Millions of content slicesMillions of users
- #17: This is the agenda slide. There is only one of these in the deck.NOTES:What does X stories to run mean ? Can we be more clear on thatAlso – This should be a more a punch line of what we do. This slide to me is very broad and not clear. Following are the things that I would describeProblem of matching the best content to the interest of a userScale Millions of content slicesMillions of users
- #18: This is the agenda slide. There is only one of these in the deck.NOTES:What does X stories to run mean ? Can we be more clear on thatAlso – This should be a more a punch line of what we do. This slide to me is very broad and not clear. Following are the things that I would describeProblem of matching the best content to the interest of a userScale Millions of content slicesMillions of users
- #19: This is the final slide; generally for questions at the end of the talk.Please post your contact information here.