Introducing the hadoop ecosystem
2 likes890 views
The document introduces the Hadoop ecosystem, which provides an approach for handling large amounts of data across commodity hardware. Hadoop is an open source software framework that uses MapReduce and HDFS to allow distributed storage and processing of large datasets across clusters of computers. It has been adopted by many large companies as a standard for batch processing of big data. The document describes how Hadoop is used by organizations to combine different datasets, remove data silos, and enable new types of experiments and analyses.
Introducing the hadoop ecosystem
- 1. Introducing The Hadoop Ecosystem The Hadoop Ecosystem
- 2. Context: Performance Gap Trend Introduction to the Hadoop Ecosystem 2
- 3. Context: Exponential for Decades Abundance of - computing & storage - generated data (estimated 8ZB in ’15) - things More data provides greater value Traditional data doesn’t scale well It’s time for a new approach! Introduction to the Hadoop Ecosystem 3
- 4. New Hardware Approach Traditional Big Data Exotic HW Commodity HW - big central servers -racks of pizza boxes - SAN -Ethernet - RAID -JBOD Hardware reliability Unreliable HW Scales further Limited scalability Cost effective Expensive Introduction to the Hadoop Ecosystem 4
- 5. New Software Approach Traditional Big Data Monolotic Distributed - Centralized -storage & compute nodes - RDBMS Raw data Schema first Open source Proprietary Introduction to the Hadoop Ecosystem 5
- 6. Hadoop De facto big data industry standard (batch) Vendor adoption - IBM, Microsoft, Oracle, EMC, ... A collection of projects at Apache - HDFS, MapReduce, Hive, Pig, Hbase, Flume, Oozie, ... Main components - HDFS - MapReduce Cluster Set of machines running HDFS and MapReduce Introduction to the Hadoop Ecosystem 6
- 7. HDFS Introduction to the Hadoop Ecosystem 7
- 8. MapReduce Introduction to the Hadoop Ecosystem 8
- 9. MapReduce Introduction to the Hadoop Ecosystem 9
- 10. MapReduce Introduction to the Hadoop Ecosystem 10
- 11. Typical Adoption Pattern An idea that’s impractical without Hadoop Build Hadoop-based POC Move initial application to production Add more datasets and users - removing data silos in organizations - permitting easy experiments on real data Snowballs into institution’s central repository for - analysis data processing data service layer Introduction to the Hadoop Ecosystem 11
- 12. Use Case 1: Truvo Introduction to the Hadoop Ecosystem 12
- 13. Use Case 2: UZ Brussel Introduction to the Hadoop Ecosystem 13
- 14. How can you use Hadoop? What data are you ignoring? - How can you use it? How can you combine internal and external data? - Business partners - Feedback from you customers through social media - End your data silos - ... Introduction to the Hadoop Ecosystem 14
- 15. DataCrunchers - Big Data Enablers Introduction to the Hadoop Ecosystem 15
- 16. Introduction to the Hadoop Ecosystem 16