Thinking in MapReduce - StampedeCon 2013
1 like1,704 views
The document discusses how data is growing faster than individual machines can scale, requiring work to be spread across many machines. It introduces MapReduce as an approach to tackle large datasets by moving computation to where the data is located. MapReduce provides a way to break problems into independent parallelizable tasks by splitting data, processing it in a map phase where computation is done locally, and then shuffling and reducing the data in a parallel fashion. The document advocates building higher-level functions on top of MapReduce as a basic building block.
Thinking in MapReduce - StampedeCon 2013
- 1. Thinking in MapReduce Ryan Brush @ryanbrush
- 2. 2 We programmers have had it pre1y good
- 3. 3 Hardware has scaled up faster than our problem sets
- 4. 4
- 6. 6 But the party is ending (or at least changing)
- 7. 7 Data is growing faster than we can scale individual machines
- 8. 8 So we have to spread our work across many machines
- 9. 9 This is a big deal in health care
- 10. 9 This is a big deal in health care Fragmented InformaKon
- 11. 9 This is a big deal in health care Fragmented InformaKon Spread across many systems
- 12. 9 This is a big deal in health care Fragmented InformaKon Spread across many systems No one has the complete picture
- 13. 10 We need to put the picture back together again
- 14. 10 We need to put the picture back together again Be1er-‐informed decisions
- 15. 10 We need to put the picture back together again Be1er-‐informed decisions Reduce systemaKc fricKon
- 16. 10 We need to put the picture back together again Be1er-‐informed decisions Understand and improve the health of populaKons Reduce systemaKc fricKon
- 17. Chart Search
- 18. Chart Search
- 20. Chart Search -InformaKon extracKon -SemanKc markup of documents
- 21. Chart Search -InformaKon extracKon -SemanKc markup of documents -Related concepts in search results
- 22. Medical Alerts
- 23. Medical Alerts
- 24. Medical Alerts -Detect health risks in incoming data
- 25. Medical Alerts -Detect health risks in incoming data -NoKfy clinicians to address those risks
- 26. Medical Alerts -Detect health risks in incoming data -NoKfy clinicians to address those risks -Quickly include new knowledge
- 27. PopulaKon Health
- 28. PopulaKon Health
- 29. PopulaKon Health - Securely bring together health data
- 30. PopulaKon Health - Securely bring together health data - IdenKfy opportuniKes to improve care
- 31. PopulaKon Health - Securely bring together health data - IdenKfy opportuniKes to improve care - Support applicaKon of improvements
- 32. PopulaKon Health - Securely bring together health data - IdenKfy opportuniKes to improve care - Support applicaKon of improvements - Close the loop
- 33. 17 Peter Norvig, h1p://www.youtube.com/watch?v=yvDCzhbjYWs The Unreasonable EffecKveness of Data
- 34. 17 Peter Norvig, h1p://www.youtube.com/watch?v=yvDCzhbjYWs Simple models with lots of data almost always outperform complex models with less data The Unreasonable EffecKveness of Data
- 35. 18 So how can we tackle such large data sets?
- 36. 19 Can we adapt what has worked historically?
- 37. Rela%onal Databases are Awesome Acer all,
- 38. Rela%onal Databases are Awesome Atomic, transacKonal updates DeclaraKve queries Guaranteed consistency Easy to reason about Long track record of success
- 39. Rela%onal Databases are Awesome …so use them!
- 40. Rela%onal Databases are Awesome …so use them! But…
- 41. Those advantages have a cost Global, atomic, consistent state means global coordinaKon
- 42. Those advantages have a cost Global, atomic, consistent state means global coordinaKon CoordinaKon does not scale linearly
- 43. The costs of coordinaKon Remember the network effect?
- 44. The costs of coordinaKon 2 nodes = 1 channel 5 nodes = 10 channels 12 nodes = 66 channels 25 nodes = 300 channels
- 45. The result is we don’t scale linearly as we add nodes
- 48. “Shared Nothing” architectures are the most scalable…
- 49. “Shared Nothing” architectures are the most scalable… …but most real-‐world problems require us to share something…
- 50. “Shared Nothing” architectures are the most scalable… …but most real-‐world problems require us to share something… …so our designs usually have a parallel part and a serial part
- 51. The key is to make sure the vast majority of our work in the cloud is independent and parallelizable.
- 52. Amdahl’s Law S : speed improvement P : raKo of the problem that can be parallelized N: number of processors
- 53. MapReduce Primer Input Data Split 1 Split 2 Split 3 Split N . . . Mapper 1 Mapper 2 Mapper 3 Mapper N . . . Map Phase Reducer 1 Reducer 2 Reducer N . . Reduce Phase Shuffle
- 54. MapReduce Example: Word Count Books Count words per book . . . Map Phase Sum words A-‐C . . Reduce Phase Shuffle Sum words D-‐E Sum words W-‐Z Count words per book Count words per book
- 55. The network is a shared resource
- 56. The network is a shared resource Too much data to move to computaKon
- 57. The network is a shared resource So move computa3on to data Too much data to move to computaKon
- 58. MapReduce Data Locality Input Data Split 1 Split 2 Split 3 Split N . . . Mapper 1 Mapper 2 Mapper 3 Mapper N . . . Map Phase Reducer 1 Reducer 2 Reducer N . . Reduce Phase Shuffle = a physical machine
- 59. Data locality only guaranteed in the Map phase
- 60. Data locality only guaranteed in the Map phase So do as much work as possible there
- 61. Data locality only guaranteed in the Map phase So do as much work as possible there Some jobs have no reducer at all!
- 62. 38 MapReduce is a building block
- 63. 39 So let’s build higher-‐level funcKons
- 64. Grouping and AggregaKng Books Count words per book . . . Map Phase Sum words A-‐C . . Reduce Phase Shuffle Sum words D-‐E Sum words W-‐Z Count words per book Count words per book
- 65. Joins Data Set 1 Split 1 Split 2 Split 3 Group by key Map Phase Reducer 1 Reducer 2 Reducer N . . Reduce Phase Shuffle Group by key Group by key Data Set 2 Split 1 Split 2 Split 3 Group by key Group by key Group by key
- 66. Persons Split 1 Split 2 Split 3 Group by person id Map Phase Reducer 1 Reducer 2 Reducer N . . Reduce Phase Shuffle Group by person id Group by person id Visits Split 1 Split 2 Split 3 Group by person id Group by person id Group by person id Joins
- 67. Map-‐Side Joins Data Set 1 Split 3 Mapper 3 Map Phase Reducer 1 Reducer 2 . . Reduce Phase Shuffle Data set 2 Split 1 Mapper 1 Data set 2 Split 2 Mapper 2 Data set 2
- 68. 44 Filtering Map or reduce funcKons can simply discard data we’re not interested in
- 69. 45 And Others More sophisKcated pa1erns composable DisKnct Sort Binning Top N ...
- 70. 46 Chain Jobs Together Large-‐scale joins must have a reduce phase MulKple joins or group-‐by operaKons mean mulKple jobs Normalize Data Join Related Items Compute Summary Output
- 71. Codified in High-‐Level Libraries Hive, Pig, Cascading, and Crunch provide simple means to use these pa1erns Apache Crunch The era of wriKng MapReduce by hand is over
- 72. 48 How do we use these tools?
- 73. 49 Start with the ques3on you want to ask, then transform the data to answer it.
- 74. 50 output = transform (input)
- 75. 50 output = transform (input) FuncKonal over Place-‐Oriented Programming
- 76. 51 Work with data holisKcally
- 77. 51 Work with data holisKcally Re-‐running funcKons simpler to reason about than updaKng state
- 78. 51 Work with data holisKcally Re-‐running funcKons simpler to reason about than updaKng state Hadoop makes this possible at scale
- 79. 52 Don’t be afraid to re-‐process the world
- 80. 52 Don’t be afraid to re-‐process the world Something’s wrong, we’re above 95% usage! -‐TradiKonal System Administrator
- 81. 52 Don’t be afraid to re-‐process the world Something’s wrong, we’re above 95% usage! -‐TradiKonal System Administrator Something’s wrong, we’re below 95% usage! -‐Hadoop System Administrator
- 83. 54 Franklin, Halevy, Maier, h1p://homes.cs.washington.edu/~alon/files/dataspacesDec05.pdf From Databases to Dataspaces
- 84. 54 Franklin, Halevy, Maier, h1p://homes.cs.washington.edu/~alon/files/dataspacesDec05.pdf From Databases to Dataspaces (Also referred to as Data Lakes)
- 85. 55 Franklin, Halevy, Maier, h1p://homes.cs.washington.edu/~alon/files/dataspacesDec05.pdf Bring all of your data together...
- 86. 55 Franklin, Halevy, Maier, h1p://homes.cs.washington.edu/~alon/files/dataspacesDec05.pdf Bring all of your data together... ..structured or unstructured...
- 87. 55 Franklin, Halevy, Maier, h1p://homes.cs.washington.edu/~alon/files/dataspacesDec05.pdf Bring all of your data together... ...transform it with unlimited computaKon... ..structured or unstructured...
- 88. 55 Franklin, Halevy, Maier, h1p://homes.cs.washington.edu/~alon/files/dataspacesDec05.pdf Bring all of your data together... ...transform it with unlimited computaKon... ...at any Kme for any new need. ..structured or unstructured...
- 89. 56 And offer a variety of interacKve access pa1erns.
- 90. 56 And offer a variety of interacKve access pa1erns. SQL, Search, Domain-‐Specific Apps
- 91. 57 Hadoop is becoming an adapKve, mulK-‐purpose plasorm.
- 92. 58 The gap between asking novel quesKons and our ability to answer them is closing.