Riding the Elephant - Hadoop 2.0
Download as PPT, PDF0 likes754 views
The document discusses advancements in Hadoop, particularly Hadoop 2.0, highlighting the new YARN architecture that enhances resource management and job scheduling. It emphasizes the shift from traditional MapReduce to more scalable frameworks, enabling the integration of various data processing applications and languages. Additionally, it showcases the potential for improved performance in big data applications through better resource utilization and support for real-time data processing.
Riding the Elephant - Hadoop 2.0
- 1. Simon Elliston Ball Head of Big Data - Red Gate Ventures @sireb Riding the Elephant: Hadoop 2.0 http://bit.ly/RidingElephants
- 2. Append only distributed file-system In the beginning… Map Reduce Java.
- 3. JVM Based (scala, groovy, jython, clojure) More languages Streaming (python, whatever) HDP for Windows and .NET SDK
- 5. SQL on Hadoop Learning to share the toys HBase Solr on Hadoop Sharing HDFS…
- 6. Map Reduce v1 JobTracker Job Head Node TaskTracker Task (Map / Reduce) Data Node m slot 1 m slot 2 … m slot n Task r slot 1 r slot 2 … r slot n TaskTracker Task (Map / Reduce) Data Node m slot 1 m slot 2 … m slot n r slot 1 r slot 2 … r slot n TaskTracker Task (Map / Reduce) Data Node m slot 1 m slot 2 … m slot n r slot 1 r slot 2 … r slot n
- 7. Map Reduce v1 JobTracker Job Head Node TaskTracker Task (Map / Reduce) Data Node m slot 1 m slot 2 … m slot n MR Status r slot 1 r slot 2 … r slot n TaskTracker Task (Map / Reduce) Data Node m slot 1 m slot 2 … m slot n r slot 1 r slot 2 … r slot n TaskTracker Task (Map / Reduce) Data Node m slot 1 m slot 2 … m slot n r slot 1 r slot 2 … r slot n
- 8. Typical Hadoop 1.x setup HBaseProductionAdhoc
- 9. Typical Hadoop 1.x setup HBaseProductionAdhoc
- 10. YARN architecture Container Application Master Container Data Node Node Manager Container ContainerContainer Data Node Node Manager Application Master Container Free Slot Data Node Node Manager Resource Manager YARN Client
- 11. YARN architecture Container Application Master Container Data Node Node Manager Container ContainerContainer Data Node Node Manager Application Master Container Free Slot Data Node Node Manager Resource Manager YARN Client
- 12. YARN architecture Container Application Master Container Data Node Node Manager Container ContainerContainer Data Node Node Manager Application Master Container Free Slot Data Node Node Manager Resource Manager YARN Client
- 13. YARN architecture Container Application Master Container Data Node Node Manager Container ContainerContainer Data Node Node Manager Application Master Container Free Slot Data Node Node Manager Resource Manager YARN Client
- 14. Removing the choke point Advantages 60%-150% better usage Long running applications
- 15. Not quite… Operating system for Big Data? Security …but a framework for Big Data Apps Data Access abstraction
- 16. Storm on YARN A whole batch of new applications HOYA Tez (Stinger)MapReduce 2 Giraph <Insert your application here>
- 17. Batch applications Spinning YARNs with Spring Services Direct to YARN APIs Spring Data Hadoop abstraction
- 19. Spark A higher abstraction Hadoop based? … but can run on YARN In Memory Distributed Fault tolerant Real-time ✓ ✓ ✓ ✓ ❌ RRDs ✓
- 21. Hadoop is more than MapReduce The new world YARN opens up new paradigms Infrastructure maturing: better sharing
- 23. Thank you
- 24. Questions? Simon Elliston Ball Head of Big Data - Red Gate Ventures @sireb simon@simonellistonball.com http://bit.ly/RidingElephants