![API
item2item(id: ItemId): List[ItemId]
user2item(id: UserId): List[ItemId]
popular : List[ItemId]](https://image.slidesharecdn.com/hikalabo-160331004915/85/DMM-com-Spark-24-320.jpg)
![[ ]](https://image.slidesharecdn.com/hikalabo-160331004915/85/DMM-com-Spark-31-320.jpg)
DMM.comラボはなぜSparkを採用したのか？レコメンドエンジン開発の裏側をお話します！
- 2. 2015/4 - DMM labo API scalamatsuri
- 4. DMM
- 5. DMM
- 8. SPARK UC Berkekey Apache Scala, Python, Java, SQL, R API
- 13. Hadoop
- 14. item to item user to item popular
- 15. 1. (Tracking API) 2. (Hive on Spark) 3. (Spark) 4. (Sqoop) 5. API(Play)
- 18. (SPARK)
- 19. val itemToItems = userProducts.join(userProducts).filter { case (user, ((item1, keyword1, score1), (item2, keyword2, score2))) => item1 }.map { case (user, ((item1, keyword1, score1), (item2, keyword2, score2))) => ((item }.reduceByKey(_ + _).mapValues(math.sqrt(_)).map { case ((item1, keyword1, item2), score) => ((item1, keyword1), (item2, score)) }.groupByKey().mapValues(_.toList.sortBy(_._2).reverse.take(config.numDisplayIt case ((item1, keyword1), items) => items.size >= config.numDisplayItems }.cache()
- 21. val model = ALS.train(ratings.map(_._1), config.alsRank, config.alsNumIterations, config.alsLambda) val predictions = model.predict(candidates).groupBy(_.user).map { case (user, ratings) => (user, ratings.toList.sortBy(_.rating) .reverse.take(config.numDisplayItems)) }.cache()
- 23. Sqoop MariaDB
- 24. API item2item(id: ItemId): List[ItemId] user2item(id: UserId): List[ItemId] popular : List[ItemId]
- 25. DEPLOY AND EXECUTE Jenkins + Build Pipeline + BuildFlow
- 26. (2015/09) Jenkins + Build Pipeline + BuildFlow Job Script + Git Hive Spark Sqoop Recommend API(Node.js) MariaDB(Galera Cluster)
- 27. Jenkins + Build Pipeline + BuildFlow Job Script + Management API Hive on Spark Spark Sqoop Recommend API(Play) MariaDB(Galera Cluster)
- 28. Management API File Hive on Spark Hive 3 Play Spark, Hive UDF Util
- 30. AB PDCA
- 31. [ ]
- 32. 701
- 33. 75 % ↑ 97% ↑
- 34. TIPS use dataframes or datasets hive
- 35. executor memoryOverhead cheat sheet Top 5 Mistakes to Avoid When Writing Apache Spark Applications
- 38. DATAFRAMES DATASETS (1.3 - ) Dataframes (1.6 - ) Datasets Project Tungsten(1.5 - )
- 40. Realtime Recommend Dataframes & Datasets Graphframes