2014 feb 24_big_datacongress_hadoopsession1_hadoop101
0 likes764 views
This document provides an introduction to Hadoop using the Hortonworks Sandbox virtual machine. It discusses how Hadoop was created to address the limitations of traditional data architectures for handling large datasets. It then describes the key components of Hadoop like HDFS, MapReduce, YARN and Hadoop distributions like Hortonworks Data Platform. The document concludes by explaining how to get started with the Hortonworks Sandbox VM which contains a single node Hadoop cluster within a virtual machine, avoiding the need to install Hadoop locally.
2014 feb 24_big_datacongress_hadoopsession1_hadoop101
- 1. Adam Muise – Solu/on Architect, Hortonworks HADOOP 101: AN INTRODUCTION TO HADOOP WITH THE HORTONWORKS SANDBOX
- 2. Who are we?
- 3. Who is ?
- 4. 100% Open Source – Democra/zed Access to Data The leaders of Hadoop’s development We do Hadoop Drive Innova/on in the plaForm – We lead the roadmap Community driven, Enterprise Focused
- 5. We do Hadoop successfully. Support Training Professional Services
- 6. Enter the Hadoop. ……… hOp://www.fabulouslybroke.com/2011/05/ninja-‐elephants-‐and-‐other-‐awesome-‐stories/
- 7. Hadoop was created because tradi/onal technologies never cut it for the Internet proper/es like Google, Yahoo, Facebook, TwiOer, and LinkedIn
- 8. Tradi/onal architecture didn’t scale enough… App App App App App App App App DB DB DB SAN App App App App DB DB DB SAN DB DB DB SAN
- 9. Databases can become bloated and useless
- 10. $upercompu/ng Tradi/onal architectures cost too much at that volume… $/TB $pecial Hardware
- 11. So what is the answer?
- 12. If you could design a system that would handle this, what would it look like?
- 13. It would probably need a highly resilient, self-‐healing, cost-‐efficient, distributed file system… Storage Storage Storage Storage Storage Storage Storage Storage Storage
- 14. It would probably need a completely parallel processing framework that took tasks to the data… Processing Processing Processing Storage Storage Storage Processing Processing Processing Storage Storage Storage Processing Processing Processing Storage Storage Storage
- 15. It would probably run on commodity hardware, virtualized machines, and common OS plaForms Processing Processing Processing Storage Storage Storage Processing Processing Processing Storage Storage Storage Processing Processing Processing Storage Storage Storage
- 16. It would probably be open source so innova/on could happen as quickly as possible
- 17. It would need a cri/cal mass of users
- 18. Tez Storm YARN Pig HDFS MapReduce Apache Hadoop HCatalog Hive HBase Ambari Knox Sqoop Falcon Flume
- 19. Storm Tez Pig YARN HDFS MapReduce Hortonworks Data PlaForm HCatalog Hive HBase Ambari Knox Sqoop Falcon Flume
- 20. We are going to learn how to work with Hadoop in less than an hour.
- 21. To do this, we need to install Hadoop right?
- 22. Nope.
- 23. Enter the Sandbox.
- 24. The Sandbox is ‘Hadoop in a Can’. It contains one copy of each of the Master and Worker node processes used in a cluster, only in a single virtual node. Processing Processing Processing Storage Storage Storage Processing Processing Processing Storage Storage Storage Processing Storage Linux VM Processing Processing Processing Storage Storage Storage
- 25. Gefng started with Sandbox VM: -‐ Pick your flavor of VM at… hOp://www.hortonworks.com/sandbox -‐ Start the sandbox VM -‐ find the IP displayed -‐ go to… hOp://172.16.130.131 -‐ Register -‐ Click on ‘Start Tutorials’ -‐ On the lek hand nav, click on ‘HCatalog, Basic Pig & Hive Commands’
- 26. In this tutorial we will: -‐ Land files in HDFS -‐ Assign metadata with HCatalog -‐ Use SQL with Hive -‐ Learn to process data with Pig
- 27. Try the other tutorials.
- 28. Hadoop is the new Modern Data Architecture for the Enterprise
- 29. There is NO second place Hortonworks …the Bull Elephant of Hadoop InnovaGon © Hortonworks Inc. 2012: DO NOT SHARE. CONTAINS HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION Page 29