Introduction to hadoop V2
- 2. • Tarjei Romtveit • Co-founder of Monokkel AS • Former CTO – Integrasco AS • My story with Hadoop www.monokkel.io
- 3. • Daglig leder i Monokkel AS • Tidligere COO i Integrasco AS • Persistering, Prosessering og Presentasjon av data Persistering – Prosessering – Presentasjon
- 4. Bombshell If you work with data today and not start to learn the Hadoop ecosystem: You may be unemployed soon
- 5. Agenda • Context – Big Data and how to handle it • What is Hadoop? • Demo • Distributions and/or demo • “Deepdive” into Hadoop - Architecure – HDFS – YARN – MapReduce • Languages and ecosystem
- 6. What we not will cover • Security • Integrations with database X or system Y • Running Hadoop in production
- 7. Big Data
- 8. Big Data – hype and hipsters
- 9. Big Data
- 10. Big Data – Let’s add some letters • Volume • Variety • Velocity • Variability • Veracity / Data quality and the step-brother • Complexity
- 11. Big Data – Example
- 12. The Nordic Hotel Tycoon 1600 Hotels in 5 countries
- 13. I am a digital champion: The website
- 14. I am a digital champion: The desk
- 15. I am a digital champion: The external provider
- 16. I am a digital champion The IoT case
- 17. I am a digital champion Social
- 18. Houston we have a problem • Sales is declining and my stock price is tumbling
- 19. The CEO
- 20. How can the CEO manage his problem? • Get control over the data • Implement analytical processes to aid sales
- 22. The data he need to handle • Volume – Gigabytes/Terabyte • Variety – Click stream, Voice, emails, sensor data, social data, different languages, timestamp data, transactional data, third party data • Variability – Various quality • Velocity – MB per second
- 23. The data he need to handle • Veracity / Data quality – Inconsistent data quality • Complexity – Many legacy domain models
- 24. How to handle ? Web Emails Sensors Social Processing RDBMS Search
- 25. How to understand ? Web Emails Sensors Social Processing RDBMS Search
- 26. So what do Hadoop solve? Processing
- 27. What is Hadoop?
- 28. What is Hadoop? An operating system for data
- 29. An OS need software on top
- 30. Distributions '
- 31. Distributions • ”Stable” compilation of the Hadoop Ecosystem • Operational tools • Integration tools and frameworks • Data governance and data management tools • Security
- 32. Distributions
- 33. HADOOP An operating system for data Layman’s terms • Store huge files (unstructured) on many machines • Query and modify data • Can run sophisticated analytics on top
- 34. How to start: Alt 1 • https://hadoop.apache.org/ • Getting Started • Download • Unzip • bin/hadoop <commandline arguments> Alt 2 • http://hortonworks.com/products/hortonworks-sandbox/#install • Install VMWare Player or VirtualBox • Download image (6 GB) • Install and run (give it lots of memory)
- 35. DEMO – Transform and modify data – Machine learning with Spark – Integrate with ElasticSearch NEXT: ARCHITECHTURE AND HOW IT WORKS
- 36. DEMO • Hortonworks Sandbox • Hortonworks Ambari • Hortonworks Hue
- 38. 2.X.X • Hadoop Distributed File System (HDFS) • YARN (Yet Another Resource Negotiator) • MapReduce
- 40. HDFS Block index D1 D2 D3 Data Nodes B: 1, D1 B: 2, D2 B: 3, D3 B: 4, D1 B: 5, D2 B: 6, D3 Name node
- 41. HDFS Block index D1 D2 D3 Data Nodes B: 1, D1 B: 2, D2 B: 3, D3 B: 4, D1 B: 5, D2 B: 6, D3 Name node
- 42. HDFS Write Client /path/to/document1, R:2, B:{1,2} Name Node I need to write a document!
- 43. Client /path/to/document1, R:2, B:{1,2} Name Node I need to write /path/to/document1, R:2, B:{3,4} /path/to/document1, R:2, B:{5,6} HDFS Write
- 44. Client Name Node You can write to : D1,D2,D3 D1 D2 D3 Data Nodes HDFS Write
- 48. • HDFS blocks are immutable you can not change them! • Deletes and updates are written as new blocks • The node name takes care of overwriting deleted blocks • Small files are consuming a lot of name node memory HDFS Delete/Update
- 50. YARN HOW DOES HADOOP PROCESS THE DATA STORED IN HDFS?
- 51. YARN Client Resource Manager Scheduler Applications manager I want to process file “docuemt1” with my-app.jar?
- 52. YARN Resource Manager Scheduler Applications manager You can process on D1!
- 53. YARN D1 D2 Node Manager Node Manager Resource Manager Scheduler Applications manager Start my-app.jar Application Master
- 54. YARN D1 D2 Node Manager Node Manager Resource Manager Scheduler Applications manager Application Master AM to RM: “document1” is located on d1 and d2 and I need X Gb RAM
- 55. YARN D1 D2 Node Manager Node Manager Application Master Container Resource Manager Scheduler Applications manager my-app.jar is running here! Start my-app.jar
- 56. YARN + HDFS D1 D2 D3 Name Node Client Client Client • YARN will try to make sure data is processed where it is stored • ….. data locality
- 57. YARN + HDFS • Blocks are immutable. This enables high write speeds • Data is schema free! You can store any data you want • Data locality is what differentiates HDFS from other data storage • You can read massive amounts of data only limited by disk read speeds
- 58. MapReduce and others OK… BUT HOW DO I PROCESS ?
- 59. YARN Tez MapReduce <Name here> Libraries: Mahout, MLib, GraphX, Oryx Languages: Hive, Pig, R, Spark SQL, Stinger
- 60. YARN Tez <Name here> Languages: Hive, Pig, R, Spark SQL, Stinger Libraries: Mahout, Crunch, Mlib, GraphX, Oryx MapReduce
- 61. MapReduce
- 62. Document Deer Bear River Car Car River Deer Car BearDocument stored in HDFS
- 63. Splitting Deer Bear River Deer Car Bear Deer Bear River Car Car River Car Car River Deer Car Bear
- 64. Mapping Deer Bear River Car Car River Deer Car Bear Deer 1 Bear 1 River 1 Car 1 Car 1 River 1 Deer 1 Car 1 Bear 1
- 65. Shuffling Deer 1 Bear 1 River 1 Deer 1 Car 1 Bear 1 Car 1 Car 1 River 1 Deer 1 Deer 1 Deer 1 Bear 1 Bear 1 Car 1 Car 1 River 1 River 1
- 66. Reduce Deer 1 Deer 1 Deer 1 Bear 1 Bear 1 Car 1 Car 1 River 1 River 1 Deer 3 Bear 2 Car 2 River 2 Deer 3 Bear 2 Car 2 River 2 HDFS
- 69. API: Main
- 70. How to run $ bin/hadoop jar wc.jar WordCount /hdfs/dir/in /hdfs/dir/out
- 71. MapReduce • Mappers and reducers are distributed in YARN containers • Chaining of MapReduce jobs make them slow • Easy to scale but difficult to code • … use the data DSL languages instead
- 72. Languages
- 73. YARN Tez MapReduce <Name here> Languages: Hive, Pig, R, Spark SQL, Stinger Libraries: Mahout, Crunch, MLib, GraphX, Oryx
- 74. ”Languages”
- 75. PIG • Procedural language • Execute on YARN • Great for • Structuring • Moving • Transforming
- 76. Hive/Drill/Spark SQL • Declarative / SQL-like languages • Great for • Column data / Database dumps • Aggregations • Connect BI tools and Dashboards • Data Warehouse for Hadoop++
- 77. Spark • Core language (runs in YARN or standalone) • Great for • Anything that MapReduce can do • Analytics, Machine Learning • In memory and languages in Java, Scala and Python
- 78. Summary • Hadoop is designed to handle/process massive amounts of data through HDFS and/or YARN • The data do not need to be structured before it is stored in HDFS • Hadoop is an ecosystem and have languages/frameworks for data extraction, data management, data analysis and data integration • It is most convenient to begin with Hadoop by testing distributions. E.g. Hortonworks, Cloudera, MapR etc. • Learn MapReduce and learn to understand languages and a few integration tools
- 79. Is it a fad?
Editor's Notes
- #3: - Startet med å bygge distribuerte systemer for store mengder data Hoppet på Hadoop da det skulle løse alle problemer ca 2009/2010 Hoppet av igjen Hopper på igjen nå
- #4: Hvor mange jobber med data Hvor mange har jobbet med Hadoop Hvor mange har jobbet med ElasticSearch Hvor mange er konsulenter Hvor mange konsuleter/ansatt I industrien/olje/manufacturing Hvor mange konsuleter/ansatt I merkantile/handel/service/IT Hvor mange konsuleter/ansatt I statlig
- #6: Noen med erfaring med Hadoop?
- #9: Detter er det jeg forbinder med Big Data akkurat nå Veldig mye buzz… men la oss se hva det er i kjærnen og hvor Hadoop kommer inn i dette bilde
- #10: Doug Laney the inventor of big data back in 2001
- #11: MASSE KJEDELIGE ORD… LA OSS PRØVE Å SE PÅ ET EKSEMPEL Volume – Variety – Many datasets Velocity – The speed of generation of data Variability – Data can be inconsise and come in various form Veracity – Quality of data Complexity
- #12: Doug Laney the inventor of big data back in 2001
- #13: Doug Laney the inventor of big data back in 2001
- #14: Clickstream data Ratings
- #15: Clickstream data Ratings
- #16: External agreements on ratings and traffic
- #17: -Stuepiken is registering all activities -IoT
- #18: -Stuepiken is registering all activities -IoT
- #20: Doug Laney the inventor of big data back in 2001
- #29: An OpenSource operationg system for data
- #38: 2002: Open source crawler Nutch by Dough Cutting and Mike Cafarella: The internet crawler. Web was maximumily 1 billion pages large. Limited scalability capabilities. 2003: Google releases their GFS paper for massively distributed filesystem.. Cutting and Cafarella incorporates the filesystem into Nutch 2004: Google releases their Map Reduce paper for massively parallell computing. This is incorporated into Nutch as well 2006: Yahoo hires Dough Cutting and the filesystem and Map Reduce component is extracted into the Hadoop project from the Nutch project.
- #39: 2002: Open source crawler Nutch by Dough Cutting and Mike Cafarella: The internet crawler. Web was maximumily 1 billion pages large. Limited scalability capabilities. 2003: Google releases their GFS paper for massively distributed filesystem.. Cutting and Cafarella incorporates the filesystem into Nutch 2004: Google releases their Map Reduce paper for massively parallell computing. This is incorporated into Nutch as well 2006: Yahoo hires Dough Cutting and the filesystem and Map Reduce component is extracted into the Hadoop project from the Nutch project.
- #40: 2008: Hadoop was storing all data. Even financial data was trusted to Hadoop 2008: Cloudera was the first commercial company that supported Hadoop 2011: 42 000 nodes storing petabytes of data 2011: Hortonworks was spun out of Yahoo as hadoop company. This company only focuses on the open source software from with its origin @ yahoo
- #41: 2011 – First feature complete 1.0 version of Hadoop. MapReduce and HDFS is tighly integrated in 1.0 and pre versions 2013 – First large refactor of the operating system. Map Reduce is detached and Hadoop is more generalized to handle different processing paradigms
- #45: Data Nodes contains disks only
- #46: Data Nodes contains disks only
- #57: Scheduler is allocating based on information available from the node ApplicationsManager track the state of all applications (managers) in the cluster
- #58: Node Managers constantly updates the ResourceManager with the current resource situatuon Node Managers start the ApplicationMaster and Container Application Masters are negotiating resources and allocates more containers if allowed
- #59: Node Managers constantly updates the ResourceManager with the current resource situatuon Node Managers start the ApplicationMaster and Container Application Masters are negotiating resources and allocates more containers if allowed
- #60: Application Masters are negotiating resources and allocates more containers if allowed. CPU cores, and Memory is requested, and that my file is located on D2 The application started by the Node Manager does not need to be Java.
- #84: De store selskapene: Spotify, Google, Netflix, === disruptorene