クックパッドでのemr利用事例
17 likes4,635 views
The document discusses the history and features of Amazon's Elastic MapReduce (EMR) service. It mentions that in 2009 Hadoop was used with MySQL for analyzing large datasets, and that in 2010 Amazon launched EMR to make Hadoop easier to use on EC2. EMR supports the open-source Hadoop framework and makes it simple to set up Hadoop clusters without having to acquire and configure hardware. The document also provides a brief comparison of EMR to other Hadoop distributions like Cloudera and discusses how EMR can be used to run MapReduce jobs on large datasets stored in databases.
![[PR] NoSQL](https://image.slidesharecdn.com/111215emr-111215052342-phpapp01/85/emr-5-320.jpg)
![[13930, 29011, 39291, ...] # 50000
1000
{
‘139’ => [13930, 13989, 13991, ...], # 50
‘290’ => [29011, 29098, 29076, ...], # 50
‘392’ => [39291, 39244, 39251, ...], # 50
...
}](https://image.slidesharecdn.com/111215emr-111215052342-phpapp01/85/emr-30-320.jpg)
クックパッドでのemr利用事例
- 1. EMR
- 2. • (@sasata299) • Hadoop • • • Rails, Hadoop, NoSQL
- 5. [PR] NoSQL
- 7. 1. Hadoop 2. EMR 3. EMR 4.
- 8. 1. Hadoop
- 10. 2009/9 • • MySQL • GROUP BY … • 7000 • Hadoop
- 12. 2009/10 • EC2 Hadoop • Cloudera CDH1 • Ruby Hadoop Streaming • 7000 →30 • Hadoop
- 13. Hadoop++ ←Hadoop ↓MySQL
- 14. 2. EMR
- 15. 2010/7 • Hadoop • Hadoop • SocketTimeoutException … • CDH2 • EMR
- 17. EMR vs CDH2 AMI (Amazon Machine Image) UP EMR CDH2
- 18. EMR vs CDH2 AMI (Amazon Machine Image) UP EMR CDH2
- 24. 3. EMR
- 25. DB • xx UU • UU • • , etc...
- 26. • MySQL MySQL • MySQL EMR - UU - -
- 27. EMR • - ○○ xx • Ruby • •
- 28. 4.
- 29. • • - 1 • 5 …
- 30. [13930, 29011, 39291, ...] # 50000 1000 { ‘139’ => [13930, 13989, 13991, ...], # 50 ‘290’ => [29011, 29098, 29076, ...], # 50 ‘392’ => [39291, 39244, 39251, ...], # 50 ... }
- 31. • … • mapper → reducer → finalize • script-runner.jar •
- 32. • • IF • • EMR
- 33. • EMR • • • Hadoop Streaming • :-)
- 38. @sasata299
Editor's Notes
- #2: \n
- #3: \n
- #4: \n
- #5: \n
- #6: \n
- #7: \n
- #8: \n
- #9: \n
- #10: クックパッドの検索ログを、地域別や季節別に見られるサービスです。\n
- #11: \n
- #12: \n
- #13: \n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n
- #29: \n
- #30: \n
- #31: \n
- #32: \n
- #33: \n
- #34: \n
- #35: \n
- #36: \n
- #37: \n
- #38: \n