Emphemeral hadoop clusters in the cloud
1 like894 views
This document discusses how Etsy uses ephemeral Hadoop clusters in the cloud to process and analyze their large amounts of data. They move data from databases and logs to S3, then use Cascading to run jobs that perform joins, grouping, etc. on the data in Hadoop. They also leverage Hadoop streaming to run MATLAB scripts for more complex analysis, and build a system called Barnum to coordinate jobs and return results. This approach allows them to flexibly scale processing from zero to thousands of nodes as needed in a cost effective and isolated manner.
![Ephemeral
Hadoop
Clusters
in
the
Cloud
[1]
Greg
Fodor,
Etsy
gfodor@etsy.com](https://image.slidesharecdn.com/emphemeralhadoopclustersinthecloud-110915115022-phpapp01/85/Emphemeral-hadoop-clusters-in-the-cloud-1-320.jpg)
![[5]](https://image.slidesharecdn.com/emphemeralhadoopclustersinthecloud-110915115022-phpapp01/85/Emphemeral-hadoop-clusters-in-the-cloud-9-320.jpg)
![visit
logs
input:
akamai
access
logs
(event
beacons)
output:
[visit_id,
[event]]](https://image.slidesharecdn.com/emphemeralhadoopclustersinthecloud-110915115022-phpapp01/85/Emphemeral-hadoop-clusters-in-the-cloud-18-320.jpg)
![[2]](https://image.slidesharecdn.com/emphemeralhadoopclustersinthecloud-110915115022-phpapp01/85/Emphemeral-hadoop-clusters-in-the-cloud-20-320.jpg)
![Java
syntax
[10]](https://image.slidesharecdn.com/emphemeralhadoopclustersinthecloud-110915115022-phpapp01/85/Emphemeral-hadoop-clusters-in-the-cloud-24-320.jpg)
![Swiss
army
knife
Full
dataset
analysis
Matlab,
Ruby
scripts
‘ArHfact’
outputs
Tokyo
Cabinet,
Lucene,
SQLite
Side-‐effects
MySQL,
CloudFront
[3]](https://image.slidesharecdn.com/emphemeralhadoopclustersinthecloud-110915115022-phpapp01/85/Emphemeral-hadoop-clusters-in-the-cloud-35-320.jpg)
![Job
2
Job
1
[4]](https://image.slidesharecdn.com/emphemeralhadoopclustersinthecloud-110915115022-phpapp01/85/Emphemeral-hadoop-clusters-in-the-cloud-37-320.jpg)
![[6]
v1](https://image.slidesharecdn.com/emphemeralhadoopclustersinthecloud-110915115022-phpapp01/85/Emphemeral-hadoop-clusters-in-the-cloud-71-320.jpg)
![ad-‐hoc
shell
scripts
TSV
into
unsharded
MySQL
not
re-‐usable
[6]](https://image.slidesharecdn.com/emphemeralhadoopclustersinthecloud-110915115022-phpapp01/85/Emphemeral-hadoop-clusters-in-the-cloud-72-320.jpg)
![[5]](https://image.slidesharecdn.com/emphemeralhadoopclustersinthecloud-110915115022-phpapp01/85/Emphemeral-hadoop-clusters-in-the-cloud-95-320.jpg)
![PHENOMENAL
COSMIC
POWERS
[7]](https://image.slidesharecdn.com/emphemeralhadoopclustersinthecloud-110915115022-phpapp01/85/Emphemeral-hadoop-clusters-in-the-cloud-104-320.jpg)
![[9]](https://image.slidesharecdn.com/emphemeralhadoopclustersinthecloud-110915115022-phpapp01/85/Emphemeral-hadoop-clusters-in-the-cloud-107-320.jpg)
![photo
credits
[1]
by
elfike
hhp://www.flickr.com/photos/elfike/157439707/
[2]
by
Dan4th
hhp://www.flickr.com/photos/43264265@N00/5371557240/
[3]
by
mandolux
hhp://www.flickr.com/photos/73935252@N00/34418046/
[4]
by
The
Suss-‐Man
hhp://www.flickr.com/photos/8692813@N06/4580254188/
[5]
by
Stephen
Rees
hhp://www.flickr.com/photos/60142746@N00/214461223/
[6]
by
Let
Ideas
Compete
hhp://www.flickr.com/photos/quesHon_everything/3414827746/
[7]
by
funkandjazz
hhp://www.flickr.com/photos/phunk/2484159004/
[8]
by
ViaMoi
hhp://www.flickr.com/photos/12187843@N07/3343619603/
[9]
by
kreg.steppe
hhp://www.flickr.com/photos/spyndle/500305000/
[10]
clipart
(really)
[11]
by
Chris
Pirillo
hhp://www.flickr.com/photos/49503157467@N01/34588230/](https://image.slidesharecdn.com/emphemeralhadoopclustersinthecloud-110915115022-phpapp01/85/Emphemeral-hadoop-clusters-in-the-cloud-109-320.jpg)
Emphemeral hadoop clusters in the cloud
- 1. Ephemeral Hadoop Clusters in the Cloud [1] Greg Fodor, Etsy gfodor@etsy.com
- 2. about me gfodor@etsy.com @gfodor Data Wrangler
- 3. about etsy
- 4. the world’s handmade marketplace
- 5. total members: 9,000,000 total acHve shops: 800,000 items listed: 9.5M page views per month: >1B 2010 sales: $314.3M
- 6. lots of data
- 8. ephemeral?
- 9. [5]
- 10. “elasHc” to the extreme
- 14. how did we get here?
- 15. wanted to dip our toes stop hiWng the database stop grepping log files
- 16. 2 data sources -‐> S3
- 17. database snapshots input: nightly diffs (SELECT * FROM <table> WHERE update_date > 1 day ago) output: full tables as sequence files
- 18. visit logs input: akamai access logs (event beacons) output: [visit_id, [event]]
- 20. [2]
- 21. data flow joins, group bys, etc.
- 22. cascading Chris Wensel hhp://www.cascading.org/
- 24. Java syntax [10]
- 25. cascading.jruby Grégoire Marabout (Qualtera), Mah Walker (Etsy), Stefan Karpinski (Etsy), Steve Mardenfeld (Etsy) github: hhp://bit.ly/o3DNtC blog: hhp://etsy.me/cFytuL
- 27. “push” job binaries to S3 run on ElasHc Map/Reduce starts cluster, runs, shuts down access results on S3
- 28. next project: shop recommendaHons
- 29. 3 steps: ✔ data preparaHon -‐ Cascading ✖ analysis/training ✖ predicHon
- 30. sparse implementaHon of SVD
- 31. 3 steps: ✔ data preparaHon -‐ Cascading ✖ analysis/training -‐ MATLAB ✖ predicHon -‐ MATLAB
- 32. “MATLAB, in my Hadoop cluster?”
- 33. hadoop streaming
- 34. arbitrary scripts for map & reduce
- 35. Swiss army knife Full dataset analysis Matlab, Ruby scripts ‘ArHfact’ outputs Tokyo Cabinet, Lucene, SQLite Side-‐effects MySQL, CloudFront [3]
- 36. 3 steps: ✔ data preparaHon -‐ Cascading ✔ analysis/training -‐ MATLAB ✔ predicHon -‐ MATLAB
- 37. Job 2 Job 1 [4]
- 38. Barnum
- 39. Sinatra web service on EC2
- 40. barnum starts job and passes callback URL when job finishes, hadoop hits callback URL to barnum to proceed
- 54. 3 steps: ✔ data preparaHon -‐ Cascading ✔ analysis/training -‐ MATLAB ✔ predicHon -‐ MATLAB
- 70. geWng data back to web stack?
- 71. [6] v1
- 72. ad-‐hoc shell scripts TSV into unsharded MySQL not re-‐usable [6]
- 73. v2
- 79. datasets are versioned based upon job execuHon Hme
- 81. MySQL Tables: Memcache Cluster:
- 82. Output dataset <-‐> ORM Model
- 83. PHP:
- 84. PHP: Cascading:
- 86. Old tables regularly dropped
- 87. how we’re using this stack analyHcs products (internal) (external)
- 88. analyHcs
- 91. products
- 94. May 2011: 4,926 successful job runs
- 95. [5]
- 96. scale up from zero
- 97. isolaHon
- 98. isolaHon across runs fresh machine each Hme
- 99. isolaHon between developers no toe-‐stepping
- 101. big RAM when you need it (but not when you don’t)
- 102. need one machine? use one machine.
- 103. wriHng jobs
- 104. PHENOMENAL COSMIC POWERS [7]
- 105. prototyping run slow, unopHmized version on 500 machine for < $100
- 106. parameter tuning Try N=1, 2, 5, 10 and see which results in best output
- 107. [9]
- 108. quesHons?
- 109. photo credits [1] by elfike hhp://www.flickr.com/photos/elfike/157439707/ [2] by Dan4th hhp://www.flickr.com/photos/43264265@N00/5371557240/ [3] by mandolux hhp://www.flickr.com/photos/73935252@N00/34418046/ [4] by The Suss-‐Man hhp://www.flickr.com/photos/8692813@N06/4580254188/ [5] by Stephen Rees hhp://www.flickr.com/photos/60142746@N00/214461223/ [6] by Let Ideas Compete hhp://www.flickr.com/photos/quesHon_everything/3414827746/ [7] by funkandjazz hhp://www.flickr.com/photos/phunk/2484159004/ [8] by ViaMoi hhp://www.flickr.com/photos/12187843@N07/3343619603/ [9] by kreg.steppe hhp://www.flickr.com/photos/spyndle/500305000/ [10] clipart (really) [11] by Chris Pirillo hhp://www.flickr.com/photos/49503157467@N01/34588230/