Hadoop jon
Download as PPTX, PDF1 like364 views
The document discusses Hadoop, an open-source software framework for distributed storage and processing of large datasets across clusters of computers. It describes how Hadoop addresses the growing volume, variety and velocity of big data through its core components: HDFS for storage, and MapReduce for distributed processing. Key features of Hadoop include scalability, flexibility, reliability and economic viability for large-scale data analytics.
Hadoop jon
- 1. Hadoop Jon By HumoyunJon Lee
- 2. 90% OF THE WORLD’S DATA HAS BEEN GENERATED IN THE LAST THREE YEARS ALONE, AND IT IS GROWING AT EVEN A MORE RAPID RATE. BIG DATA The world has been exponential data growth, due to social media, mobility, E-commerce and other factors. • Volume • Variety • Velocity
- 3. “Big Data is like teenage sex; everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it” Dan Ariely, Duke University
- 5. To Address This Issue We need HadoopJon
- 6. A Shared Nothing Network or What is that Hadoop
- 7. The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.
- 10. Prerequisites : • Installing Java v1.5+ • Adding dedicated Hadoop system user. • Configuring SSH access. • Disabling IPv6. Installing HadoopJon
- 11. Configuring Hadoop : a. hadoop-env.sh b. core-site.xml c. mapred-site.xml d. hdfs-site.xml
- 12. Hadoop comes with several web interfaces which are by default available at these locations: • http://localhost:50070/ – web UI of the NameNode daemon • http://localhost:50030/ – web UI of the JobTracker daemon • http://localhost:50060/ – web UI of the TaskTracker daemon Hadoop Web Interfaces
- 13. Reliable Hadoop Features Flexible Economical Scalable Hadoop Key Characteristics:
- 14. • Scalable – New nodes can be added as needed, and added without needing to change data formats, how data is loaded, how jobs are written, or the applications on top. • Economical – Hadoop brings massively parallel computing to commodity servers. The result is a sizeable decrease in the cost per terabyte of storage, which in turn makes it affordable to model all your data.
- 15. • Flexible – Hadoop is schema-less, and can absorb any type of data, structured or not, from any number of sources. Data from multiple sources can be joined and aggregated in arbitrary ways enabling deeper analyses than any one system can provide. • Reliable – When you lose a node, the system redirects work to another location of the data and continues processing without missing a beat
- 16. Hadoop Ecosystem
- 18. • HDFS is designed to store a very large amount of information (terabytes or petabytes). This requires spreading the data across a large number of machines. • HDFS stores data reliably. If individual machines in the cluster fail, data is still being available with data redundancy. Hadoop Distributed File System (HDFS):
- 19. • HDFS provides fast, scalable access to the information loaded on the clusters. It is possible to serve a larger number of clients by simply adding more machines to the cluster. • HDFS integrate well with Hadoop MapReduce, allowing data to be read and computed upon locally whenever needed. • HDFS was originally built as infrastructure for the Apache Nutch web search engine project
- 20. Hadoop does not require expensive, highly reliable hardware. It is designed to run on clusters of commodity hardware, an HDFS instance may consist of hundreds or thousands of server machines, each storing part of the file system’s data. The fact that there are a huge number of components and that each component has a non-trivial probability of failure means that some component of HDFS is always non-functional. Therefore, detection of faults and quick, automatic recovery from them is a core architectural goal of HDFS. Commodity Hardware Failure:
- 21. Applications that run on HDFS need continuous access to their data sets. HDFS is designed more for batch processing rather than interactive use by users. The emphasis is on high throughput of data access rather than low latency of data access. Continuous Data Access:
- 22. Applications that run on HDFS have large data sets. A typical file in HDFS is gigabytes to terabytes in size. So, HDFS is tuned to support large files. It is also worth examining the applications for which using HDFS does not work so well. While this may change in the future, these are areas where HDFS is not a good fit today: Very Large Data Files:
- 23. • Low-latency data access • Lots of small files • Multiple writers, arbitrary file modifications
- 24. • Pig is an open-source high-level dataflow system. • It provides a simple language for queries and data manipulation Pig Latin, that is compiled into MapReduce jobs that are run on Hadoop. • Why is it important? - Companies like Yahoo, Google and Microsoft are collecting vast sets in the form of click steams, search logs, and web crawls. - Some form of ad-hoc processing and analysis of all of this information is required. What is Pig
- 25. • An ad-hoc way of creating and executing MapReduce jobs on very large data sets • Rapid Development • No Java is required • Developed byYahoo! Why was Pig created?
- 27. • Pig is a data flow language. It is at the top of Hadoop and makes it possible to create complex jobs to process large volumes of data quicly and efficiently. • It will consume any data that you feed it: Structured, semi-structured, or unstructured. • Pig provides the common data operations (filters, joins, ordering) and nested data types (tuple, bags, and maps) which are missing in MapReduce. • PIG scripts are easier and faster to write than standard Java Hadoop jobs and PIG has lot of clever optimizations like multi query execution, which can make your complex queries execute quiker. Where I should Use PIG
- 28. • Hive is a data warehouse infrastructure built on top of Hadoop. • It facilitates querying large datasets residing on a distributed storage. • It provides a mechanism to project structure on to the data and query the data using a SQL-like query language called “HiveQL”. What is Hive
- 29. • Hive was developed by Facebook and was open sourced in 2008 . • Data stored in Hadoop is inaccessible to business users. • High level languages like Pig, Cascading etc are geared towards developers. • SQL is a common language that is known to many. Hive was developed to give access to data stored in HadoopJon, translating SQL like queries into map reduce jobs. Why hive was developed
- 30. Hammaga rahmat Nahorgi Presentatsiya over