Hadoop Infrastructure (Oct. 3rd, 2012)
0 likes416 views
This document provides an overview of Hadoop infrastructure and related technologies: - Hadoop is based on Apache's implementation of Google's BigTable and uses Java VMs to parse instructions. It allows reading, writing, and manipulating very large datasets using sequential writes and column-based file structures in HDFS. - HDFS is the backend file system for Hadoop that allows for easy node management and operability. Technologies like HBase can augment or replace HDFS. - Middleware like Hive, Pig, and Cassandra help connect to and utilize Hadoop. Each has different uses - Hive is a data warehouse, Pig uses its own query language, and Sqoop connects databases and datasets.
Hadoop Infrastructure (Oct. 3rd, 2012)
- 1. Infrastructure and Stack Presented by John Dougherty, Viriton 10/03/2012 john.dougherty@viriton.com
- 2. What is Hadoop? • Apache’s implementation of Google’s BigTable • Uses a Java VM in order to parse instructions • Uses sequential writes & column based file structures with HDFS • Grants the ability to read/write/manipulate very large data sets/structures.
- 3. What is Hadoop? (cont.) VS.
- 4. What is BigTable • Contains the framework that was based on, and is used in, hadoop • Uses a commodity approach to hardware • Extreme scalability and redundancy
- 5. Commodity Perspective • Commercial Hardware cost vs. failure rate – Roughly double the cost of commodity – Roughly 5% failure rate • Commodity Hardware cost vs. failure rate – Roughly half the cost of commodity – Rougly 10-15% failure rate
- 6. Breaking Down the Complexity
- 7. What is HDFS • Backend file system for the Hadoop platform • Allows for easy operability/node management • Certain technologies can replace or augment – Hbase (Augments HDFS) – Cassandra (Replaces HDFS)
- 8. What works with Hadoop? • Middleware and connectivity tools improve functionality • Hive, Pig, Cassandra (all sub-projects of Apache’s Hadoop) help to connect and utilize • Each application set has different uses Pig
- 10. Schedulers/Configurators • Zookeeper – Helps you in configuring many nodes – Can be integrated easily • Oozie – A job resource/scheduler for hadoop – Open source • Flume – Concatenator/Aggregator (Dist. log collection)
- 11. Middleware • Hive – Data warehouse, connects natively to hadoop’s internals – Uses HiveQL to create queries – Easily extendable with plugins/macros • Pig – Hive-like in that it uses its own query language (pig latin) – Easily extendable, more like SQL than Hive • Sqoop – Connects databases and datasets – Limited, but powerful
- 12. How can Hadoop/Hbase/MapReduce help? • You have a very large data set(s) • You require results on your data in a timely manner • You don’t enjoy spending millions on infrastructure • Your data is large enough to cause a classic RDBMS headaches
- 13. Column Based Data • Developer woes – Extract/Transfer/Load is a concern for complicated schemas – Egress/Ingress between existing queries/results becomes complicated – Solutions are deployed with walls of functionality – Hard questions turn into hard queries
- 14. Column Based Data (cont.) • Developer joys – You can now process PB, into EB, and beyond – Your extended datasets can be aggregated, not easily; but also unlike ever before – You can extend your daily queries to include historical data, even incorporating into existing real-time data usage
- 15. Future Projects/Approaches • Cross discipline data sharing/comparisons • Complex statistical models re-constructed • Massive data set conglomeration and standardization (Public sector data, mostly)
- 16. How some software makes it easier • Alteryx – Very similar to Talend for interface, visual – Allows easy integration into reporting (Crystal Reports) • Qubole – This will be expanded on shortly – Easy to use interface and management of data • Hortonworks (Open Source) – Management utility for internal cluster deployments • Cloudera (Open, to an extent) – Management utility from Cloudera, also for internal deployments