天下武功唯快不破：利用串流資料實做出即時分類器和即時推薦系統
72 likes8,248 views
The document outlines the challenges of static content in e-commerce and presents a solution called 'pinball', which utilizes real-time data streaming to enhance product recommendations and ads personalization. Key features of pinball include a real-time classifier that detects buyer preferences and improves engagement based on user behavior. Additionally, the document discusses the architecture of storm for buying intention detection and highlights the importance of combining batch and real-time processing for efficient data insights.
![Lambda Architecture
Yahoo Confidential & Proprietary
42
[REF] http://lambda-architecture.net/](https://image.slidesharecdn.com/pinball-dscv2yahoo-140831025242-phpapp01/85/-42-320.jpg)
![Lambda Architecture
Yahoo Confidential & Proprietary
43
[REF] http://lambda-architecture.net/](https://image.slidesharecdn.com/pinball-dscv2yahoo-140831025242-phpapp01/85/-43-320.jpg)
![Lambda Architecture
Yahoo Confidential & Proprietary
44
Storm Streaming
[REF] http://lambda-architecture.net/](https://image.slidesharecdn.com/pinball-dscv2yahoo-140831025242-phpapp01/85/-44-320.jpg)
![Lambda Architecture
Summingbird
Yahoo Confidential & Proprietary
45
[REF] http://lambda-architecture.net/](https://image.slidesharecdn.com/pinball-dscv2yahoo-140831025242-phpapp01/85/-45-320.jpg)
天下武功唯快不破：利用串流資料實做出即時分類器和即時推薦系統
- 1. 天下武功唯快不破： 利用串流資料實做出即時分類器和即時推薦系統 Yahoo! Taiwan EC Data Team
- 2. Who I am ▪ Norman Huang (normany@yahoo-inc.com) ▪ Software & Data Engineer of Yahoo! Taiwan ▪ Aims to retrieve and deliver data insights via BI platform and data mining algorithms. 2
- 3. Who I am ▪ Jason Lin (jasonysl@yahoo-inc.com) ▪ Software & Data Engineer of Yahoo! Taiwan ▪ Responsible for recommendation system personalization mechanisms and cloud computing developer. 3
- 4. Agenda ▪ Challenges ▪ Solution: Pinball ▪ Q&A 4
- 5. Challenges ! ! ! ! ! ! ▪ Static content until next batch job. ! ! ! 5 Processing
- 6. Challenges ! ! ! ! ! ! ▪ Static content until next batch job. ▪ Batched product recommendation algorithms have become common features among e-commerce platforms. ! 6 Processing
- 7. Challenges ! ! ! ! ! ! ▪ Nearly 72% of visitors made their decision at the same day. 7 Absorbed into batch views Not yet absorbed Time Several hours of data
- 8. Challenges ! ! ! ! ! ! ▪ Nearly 72% of visitors made their decision at the same day. ▪ Real-time solution to interact with potential buyers. 8 Absorbed into batch views Not yet absorbed Time Several hours of data
- 10. Pinball 10 Classifier Classifier User Profile A Profile B Profile C
- 11. Pinball ! ▪ Real-time classifier ▪ Detect buyers’ preferences by streaming data processing ▪ Deliver personalized ads and product recommendations on the fly 11
- 12. Pinball ! ▪ Real-time classifier ▪ Detect buyers’ preferences by streaming data processing ▪ Deliver personalized ads and product recommendations on the fly ! ▪ Challenges › How do to it in real-time? 12
- 13. Pinball ! ▪ Real-time classifier ▪ Detect buyers’ preferences by streaming data processing ▪ Deliver personalized ads and product recommendations on the fly ! ▪ Challenges › How do to it in real-time? › Storm 13
- 14. Pinball ! ▪ Real-time classifier ▪ Detect buyers’ preferences by streaming data processing ▪ Deliver personalized ads and product recommendations on the fly ! ▪ Challenges › How do to it in real-time? › Storm! › How to determine customers’ purchasing desire? 14
- 15. Pinball ! ▪ Real-time classifier ▪ Detect buyers’ preferences by streaming data processing ▪ Deliver personalized ads and product recommendations on the fly ! ▪ Challenges › How do to it in real-time? › Storm! › How to determine customers’ purchasing desire? › Buying Intention Detection 15
- 16. Solution: Pinball ▪ Storm Overview ▪ Buying Intention (BI) ▪ Architecture and Design 16
- 17. Pinball 17 Storm Learning Buyer
- 18. Pinball 18 Storm Learning Buyers
- 19. Pinball 19 Learning Storm Is Potential Buyer? Buyers Visitor Promotions
- 20. Pinball Pinball 20 Learning Storm Is Potential Buyer? Buyers Visitor Promotions
- 21. Pinball Pinball 21 Learning Storm Is Potential Buyer? Buyers Buyer Promotions
- 22. Storm Concepts ▪ Tuple & Streams ▪ Spouts & Bolts ▪ Topologies Yahoo Confidential & Proprietary 22
- 23. Tuple & Streams ▪ Tuple ! ! ! ! ▪ Stream Yahoo Confidential & Proprietary 23 Field 1 Field 2 Field 3 Field 4 Field 5 Tuple Tuple 1 Tuple 2 Tuple 3 Tuple n Stream
- 24. Spouts & Bolts Yahoo Confidential & Proprietary 24 Spout T T T T T Bolt T T T
- 25. Topology 25 Spout Bolt Bolt Streams ▪ Hadoop map-reduce job vs. Storm topology
- 26. Topology 26 Spout Bolt Bolt Streams ▪ Hadoop map-reduce job vs. Storm topology
- 27. Storm Concepts Yahoo Confidential & Proprietary 27 Computational Primitives Use Case High-level! Language Hadoop Map & Reduce Batch Processing Pig Storm Spout & Bolt Stream Processing Trident
- 28. Storm 28 Nimbus Zookeeper Zookeeper Zookeeper Supervisor Supervisor Supervisor Supervisor Supervisor Master node, similar to the Hadoop JobTracker
- 29. Storm 29 Nimbus Zookeeper Zookeeper Zookeeper Supervisor Supervisor Supervisor Supervisor Supervisor Coordinates the Storm cluster
- 30. Storm 30 Nimbus Zookeeper Zookeeper Zookeeper Supervisor Supervisor Supervisor Supervisor Supervisor Run worker processes
- 31. Buying Intention ▪ Based on our findings: › The more page views, the higher the chance a visitor will buy it. › BUT, the buying intension value of each category will vary. 31 2 6
- 32. How to leverage Storm with Buying Intention (BI)?
- 33. Data Flow Diagram 33
- 35. Learning & Classifier ▪ Online Binary Classification › Simple and computationally efficient ▪ e.g. › assumptions: γ=0.1, BI = 3 › scenario: a user makes 6 page views before purchasing • BI’ = 3 + (6-3) x 0.1 • BI’ = 3.3 35 BI ' = BI +(PV − BI )×γ
- 38. Lambda Architecture ▪ Term created by Nathan Marz (Creator of Apache Storm) ! ▪ Batch Real-time processing Yahoo Confidential & Proprietary 38
- 39. Lambda Architecture ▪ Term created by Nathan Marz (Creator of Apache Storm) ! ▪ Batch Real-time processing Yahoo Confidential & Proprietary 39
- 40. Lambda Architecture ▪ Term created by Nathan Marz (Creator of Apache Storm) ! ▪ Batch + Real-time processing › Hybrid batch and real-time processing Yahoo Confidential & Proprietary 40
- 41. Lambda Architecture ▪ Term created by Nathan Marz (Creator of Apache Storm) ! ▪ Batch + Real-time processing › Hybrid batch and real-time processing › Batch processing is treated as source of truth, and real-time updates models/insights between batches. Yahoo Confidential & Proprietary 41
- 42. Lambda Architecture Yahoo Confidential & Proprietary 42 [REF] http://lambda-architecture.net/
- 43. Lambda Architecture Yahoo Confidential & Proprietary 43 [REF] http://lambda-architecture.net/
- 44. Lambda Architecture Yahoo Confidential & Proprietary 44 Storm Streaming [REF] http://lambda-architecture.net/
- 45. Lambda Architecture Summingbird Yahoo Confidential & Proprietary 45 [REF] http://lambda-architecture.net/
- 47. 47
- 48. How to keep it generic and flexible? ▪ to add more signals ▪ to add more online learning algorithms ▪ to add more channels
- 49. How to keep it generic and flexible? Signals Algorithms Channels 49 Click Login Buy View Bounce Time Spent Buying Intention Email Y! Webpages Mobile Apps Messenger Fraud Detection Webpage Sequence
- 50. Summary ▪ Scalable to process real-time data ▪ Supports online learning algorithms ▪ Flexible interactions with visitors ▪ Increase user's engagement ▪ Increase the conversion rate ▪ To create synergy by combining batched recommender and Pinball Yahoo Confidential & Proprietary 50
- 51. Simple Hands-on -> Find out the heavy users!
- 52. Find out the heavy users! ▪ Memorize the numbers of page views for each user ▪ If the numbers are great than 3, it’s a heavy user Yahoo Confidential & Proprietary 52
- 53. Find out the heavy users! Yahoo Confidential & Proprietary 53 User Log Spout Learning Bolt userid, type, catlv1, catlv2, timestamp
- 54. Find out the heavy users! Yahoo Confidential & Proprietary 54 User Log Spout Learning Bolt userid, type, catlv1, catlv2, timestamp Learning Bolt shuffleGroup userA, xxxxx userB, xxxxx userD, xxxxx userB, xxxxx userE, xxxxx userC, xxxxx userB, xxxxx userC, xxxxx
- 55. Find out the heavy users! Yahoo Confidential & Proprietary 55 User Log Spout Learning Bolt userid, type, catlv1, catlv2, timestamp Learning Bolt fieldGroup userA, xxxxx userD, xxxxx userF, xxxxx userF, xxxxx userE, xxxxx userC, xxxxx userB, xxxxx userB, xxxxx userB, xxxxx userC, xxxxx
- 56. Find out the heavy users! Yahoo Confidential & Proprietary 56 User Log Spout Learning Bolt Learning Bolt fieldGroup userA, xxxxx userD, xxxxx userF, xxxxx userF, xxxxx userE, xxxxx userC, xxxxx userB, xxxxx userB, xxxxx userB, xxxxx userC, xxxxx Qualification Bolt userA, totalPV userB, totalPV userC, totalPV userF, totalPV
- 57. Questions? Norman! @normanyhuang! www.linkedin.com/in/normany Jason! @kalijason! www.linkedin.com/pub/jason-lin/67/93/743