Hadoop installation with an example
Download as PPTX, PDF0 likes383 views
This document provides an overview of Apache Hadoop, an open-source framework for distributed storage and processing of large datasets across clusters of computers. It discusses what Hadoop is, why it is useful for big data problems, examples of companies using Hadoop, the core Hadoop components like HDFS and MapReduce, and how to install and run Hadoop in pseudo-distributed mode on a single node. It also includes an example of running a word count MapReduce job to count word frequencies in input files.
Hadoop installation with an example
- 1. A new way to store and analyze data An Elephant can't jump. But can carry heavy load.
- 2. Overview… What is Hadoop? Why Hadoop? Famous Hadoop users. Core-Components of Hadoop. Hadoop installation and configuration. Starting your Single-node cluster. Stopping your Single-node cluster. Running a single-node cluster program.
- 3. What is Apache Hadoop? The most well known technology used for Big Data is Hadoop. Apache Hadoop is a Framework that allows for the distributed processing of large data sets across clusters of commodity computers using a simple programming model.
- 4. Why Hadoop? Need to process 100TB datasets – On 1 node Scanning @50MB/s = 23days On 1000 node cluster : Scanning @50MB/s = 33 mins
- 5. Famous Hadoop Users.. Two cluster of 8000 and 3000 Nodes 100 Nodes 4500 Nodes 532 Nodes
- 6. Core-Components of Hadoop.. The Cluster is the set of host machines (nodes). Nodes may be partitioned in racks. This is the hardware part of the infrastructure.
- 7. The YARN Infrastructure (Yet Another Resource Negotiator) is the framework responsible for providing the computational resources (e.g., CPUs, memory, etc.) needed for application executions. Two important elements are: Resource Manager Node Manager The HDFS (Hadoop Distributed File System) Inspired by Google File System is a primary distributed storage used by Hadoop. The MapReduce is the original processing model for Hadoop clusters. It distributes work within the cluster or map, then organizes and reduces the result from the nodes into a response to a query..
- 8. Mapreduce Example Two input files: file1: “hello world hello moon” file2: “goodbye world goodnight moon” Three operations: map combine • reduce we’ll use an example: WordCount Count occurrences of each word across different files
- 9. What is the output per step? MAP First map: Second map: < hello, 1 > < goodbye, 1 > < world, 1 > < world, 1 > < hello, 1 > < goodnight, 1 > < moon, 1 > < moon, 1 > COMBINE First map: Second map: < moon, 1 > < goodbye, 1 > < world, 1 > < world, 1 > < hello, 2 > < goodnight, 1 > < moon, 1 > REDUCE < goodbye, 1 > < goodnight, 1 > < moon, 2 > < world, 2 > < hello, 2 >
- 10. Hadoop Installation and Configuration... 1. Installing Java:- Java is the primary requirement for run Hadoop on any system. $ java -version
- 11. Cont... 2. Configuring SSH:-Hadoop requires SSH access to manage its nodes, i.e. remote machines plus your local machine. $ ssh-keygen -t rsa -P ""
- 12. Cont... 3. The next step is to test the SSH setup by connecting to your local machine with the user.
- 13. Cont... 4. Disabling IPv6 : To disable IPv6 on Ubuntu, open /etc/sysctl.conf in the editor of your choice and add the following lines to the end of the file: /etc/sysctl.conf # disable ipv6 net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 – You have to reboot your machine in order to make the changes take effect.
- 14. Cont... You can check whether IPv6 is enabled on your machine with the following command: $ cat /proc/sys/net/ipv6/conf/all/disable_ipv6 A return value of 0 means IPv6 is enabled, a value of 1 means disabled (that’s what we want).
- 16. Cont... Let’s unpack the Hadoop gz: $ tar xvzf hadoop 2.8.1.tar.gz
- 17. Cont... 6. Configure Hadoop Pseudo-Distributed Mode:- • Setup Environment Variables - First we need to set environment variable uses by hadoop. Edit ~/.bashrc file and append following values at end of file.
- 18. Cont... • Now edit $HADOOP_HOME/etc/hadoop/hadoop-env.sh file and set JAVA_HOME environment variable
- 19. Cont... 6. Edit Configuration Files :- Hadoop has many of configuration files, which need to configure as per requirements of your hadoop infrastructure. $ gedit hadoop/etc/hadoop/core-site.xml:
- 23. Cont... 7. Formatting the NameNode:- Now format the namenode using following command. $ hdfs namenode -format
- 24. Starting your Single-node cluster Now run start-dfs.sh script. $ start-dfs.sh Now run start-yarn.sh script. $ start-yarn.sh This will startup a Namenode, Datanode, Jobtracker and a Tasktracker on your machine.
- 25. A nifty tool for checking whether the expected Hadoop processes are running is jps $ /usr/lib/jvm/java-8-oracle/bin/jps
- 26. Go to the browser and open http://127.0.0.1:50070
- 28. Stopping your Single-node cluster $ Stop-all.sh
- 29. Running a single-node cluster program WordCount Program : -What is wordcount program? - Wordcount is a simple application that determines how many times different words appear in a given input set. Steps for this session: 1. Upload the input data text file in hdfs. 2. Create your own jar and put the output file. 3. Run a jar on hadoop architecture. 4. View output on command line. 5. View and download output file.
- 30. Input File..
- 31. Output File..
- 32. Download Output File.. You can download output file:
- 33. References.. •Apache Hadoop! (http://hadoop.apache.org) •Hadoop on Wikipedia (http://en.wikipedia.org/wiki/Hadoop) •Cloudera - Apache Hadoop for the Enterprise (http://www.cloudera.com