Apache hadoop 2_installation
Download as DOC, PDF0 likes457 views
This document provides steps to install Apache Hadoop, Hive, HBase, Pig, and Zookeeper in pseudo-distributed mode on a single node. It includes downloading and extracting the necessary files, configuring configuration files, starting daemons and services, and verifying successful installation and operation.
Apache hadoop 2_installation
- 1. Apache Hadoop 2 Installation in Pseudo Mode Download URL 1. Hadoop: https://archive.apache.org/dist/hadoop/core/stable/ 2. Hive: http://people.apache.org/~hashutosh/hive-0.10.0-rc0/ 3. Pig: http://ftp.udc.es/apache/pig/pig-0.12.0/ 4. Hbase: http://archive.apache.org/dist/hbase/hbase-0.94.10/ Step 1: Generate ssh key $ssh-keygen -t rsa -P “” Step 2: Copy id_rsa.pub to authorized_keys $cd .ssh $cp id_rsa.pub authorized_keys $chmod 644 authorized_keys Step 3: Passwordless ssh to localhost $cd ~ $ssh localhost Step 4: Untar tarballs
- 2. $tar -xvzf hadoop-2.2.0.tar.gz Step 5: Configuration files $cd hadoop-2.2.0/etc/hadoop/ $vim core-site.xml Add following properties in core-site.xml <property> <name>fs.defaultFS</name> <value>hdfs://172.17.196.14</value> </property> <property> <name>io.native.lib.available</name> <value>true</value> </property> $vim hdfs-site.xml Add following property in hdfs-site.xml <property> <name>dfs.datanode.data.dir</name> <value>/home/hadoop/hadoop-2.2.0/pseudo/dfs/data</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/home/hadoop/hadoop-2.2.0/pseudo/dfs/name</value> </property> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> $vim mapred-site.xml Add following property in mapred-site.xml
- 3. <property> <name>mapreduce.cluster.temp.dir</name> <value>/home/hadoop/hadoop-2.2.0/temp</value> <final>true</final> </property> <property> <name>mapreduce.cluster.local.dir</name> <value>/home/hadoop/hadoop-2.2.0/local</value> <final>true</final> </property> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> $vim yarn-site.xml Add following property in yarn-site.xml <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>localhost:6000</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value> localhost:6001</value> </property> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler< /value> </property> <property> <name>yarn.resourcemanager.address</name> <value> localhost:6002</value> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value>/home/hadoop/hadoop-2.2.0/yarn_nodemanager</value> </property> <property>
- 4. <name>yarn.nodemanager.address</name> <value>0.0.0.0:6003</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>10240</value> </property> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/home/hadoop/hadoop-2.2.0/app-logs</value> </property> <property> <name>yarn.nodemanager.log-dirs</name> <value>/home/hadoop/hadoop-2.2.0/logs</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> $vim slaves Add localhost in masters file Step 6: set .bashrc $cd ~ $vim .bashrc export JAVA_HOME=/usr/ export HADOOP_HOME=/home/ahadoop2/hadoop-2.2.0 export HADOOP_CONF_DIR=/home/ahadoop2/hadoop-2.2.0/etc/hadoop export PIG_HOME=/home/ahadoop2/pig-0.12.0 export HBASE_HOME=/home/ahadoop2/hbase-0.96.0-hadoop2 export HIVE_HOME=/home/ahadoop2/hive-0.11.0 export PIG_CLASSPATH=/home/ahadoop2/hadoop-2.2.0/etc/hadoop export CLASSPATH=$PIG_HOME/pig-withouthadoop.jar: $HADOOP_HOME/share/hadoop/common/hadoop-common-2.2.0.jar: $HADOOP_HOME/share/hadoop/hdfs/hadoop-hdfs-2.2.0.jar:$HBASE_HOME/lib/hbase-client- 0.96.0-hadoop2.jar:$HBASE_HOME/lib/hbase-common-0.96.0-hadoop2.jar: $HBASE_HOME/lib/hbase-server-0.96.0-hadoop2.jar:$HBASE_HOME/lib/commons-httpclient- 3.1.jar:$HBASE_HOME/lib/commons-collections-3.2.1.jar:$HBASE_HOME/lib/commons-lang- 2.6.jar:$HBASE_HOME/lib/jackson-mapper-asl-1.8.8.jar:$HBASE_HOME/lib/jackson-core-asl- 1.8.8.jar:$HBASE_HOME/lib/guava-12.0.1.jar:$HBASE_HOME/lib/protobuf-java-2.5.0.jar:
- 5. $HBASE_HOME/lib/commons-codec-1.7.jar:$HBASE_HOME/lib/zookeeper-3.4.5.jar: $HIVE_HOME/lib/hive-jdbc-0.11.0.jar:$HIVE_HOME/lib/hive-metastore-0.11.0.jar: $HIVE_HOME/lib/hive-serde-0.11.0.jar:$HIVE_HOME/lib/hive-common-0.11.0.jar: $HIVE_HOME/lib/hive-service-0.11.0.jar:$HIVE_HOME/lib/libfb303-0.9.0.jar: $HIVE_HOME/lib/postgresql-9.2-1003.jdbc3.jar:$HIVE_HOME/lib/libthrift-0.9.0.jar: $HIVE_HOME/lib/slf4j-api-1.6.1.jar:$HIVE_HOME/lib/commons-logging- 1.0.4.jar:/home/ahadoop2/Hadoop2Training.jar export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$PIG_HOME/bin:$HBASE_HOME/bin: $HIVE_HOME/bin:/bin:/usr/lib64/qt- 3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin: Step 7: Load .bashrc $cd ~ $. .bashrc Step 8: Formatting the name node $cd ~ $hadoop namenode -format Step 9: Starting Cluster $cd ~/hadoop-2.2.0/sbin $ ./start-all.sh To view the started daemons $ jps This should show the started daemons. NameNode DataNode SecondaryNamenode Nodemanager ResourceManager Apache Hbase Installation in Pseudo Mode Step 1: Untar the tarballs $tar -xvzf hbase-0.96.0-hadoop2.tar.gz Step 2: Configuration files $cd hbase-0.96.0-hadoop2/conf
- 6. $vim hbase-site.xml Copy following properties in hbase-site.xml <property> <name>hbase.rootdir</name> <value>hdfs://localhost:8020/hbase</value> <description>The directory shared by RegionServers</description> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>dfs.replication</name> <value>1</value> </property> $vim regionservers Add localhost in regionservers file Step 3: Add hadoop jars from hadoop directory to hbase lib directory $cd /home/hadoop/hadoop-2.2.0/share/hadoop/common/ $cp hadoop-common-2.2.0.jar /home/hadoop/hbase-0.96.0-hadoop2/lib/ Step 4: start hbase $cd ~ $start-hbase.sh Step 5: To view the started daemons $ jps Hmaster Hregionserver Hquorumpeer Step 6: To view hbase shell $hbase shell Step 7: Before connecting to hbase using java Start hbase rest service by executing following command
- 7. $hbase-daemon.sh start rest -p 8090 Apache Hive Installation Step 1: Untar the tarballs $tar -xvzf hive-0.11.0.tar.gz Step 2: Configuring a remote PostgreSQL database for the Hive Metastore Before you can run the Hive metastore with a remote PostgreSQL database, you must configure a connector to the remote PostgreSQL database, set up the initial database schema, and configure the PostgreSQL user account for the Hive user. Install and start PostgreSQL if you have not already done so you need to edit the postgresql.conf file. Set the listen property to * to make sure that the Configure authentication for your network in pg_hba.conf. Add a new line into pg_hba.con that has the following information: Start PostgreSQL Server $ su postgres $cd $postgres_home/bin $./pg_ctl start -D path_to_data_dir Install the Postgres JDBC Driver Copy postgresql-jdbc driver in $HIVE_HOME/lib/ Create the metastore database and user account Proceed as in the following example: bash# sudo –u postgres psql
- 8. bash$ psql postgres=# CREATE USER hiveuser WITH PASSWORD 'mypassword'; postgres=# CREATE DATABASE metastore; postgres=# exit; bash# sudo –u hiveuser metastore You are now connected to database 'metastore' with hiveuser. metastore=# i /home/hadoop/hive-0.11.0/scripts/metastore/upgrade/postgres/hive-schema- 0.10.0.postgres.sql Step 3: Configuration files $cd hive-0.11.0/conf $vim hive-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>hive.metastore.warehouse.dir</name> <value>/user/hive/warehouse</value> <description>location of default database for the warehouse</description> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:postgresql://<postgresql instance ip>:5432/metastore</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.postgresql.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hiveuser</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>mypassword</value> </property> <property>
- 9. <name>datanucleus.autoCreateSchema</name> <value>false</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://<namenode ip>:9083</value> <description>IP address (or fully-qualified domain name) and port of the metastore host</description> </property> <property> <name>datanucleus.autoStartMechanism</name> <value>SchemaTable</value> </property> </configuration> Step 4: Strat hive metastore $hive --service metastore Step 5: To view hive console $hive hive>show tables; OK Step 6: Before connecting to hive using java Start hiveserver by executing following command $hive --service hiveserver Apache pig installation Step 1: Untar the tarballs $tar -xvzf pig-0.12.0.tar.gz Step 2: Delete two jars (pig and pig-without hadoop jar) from pig home directory and add pig- withouthadoop.jar in pig installation directory (Uploaded in knowmax same path) Step 3: To open pig grunt $pig
- 10. <name>datanucleus.autoCreateSchema</name> <value>false</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://<namenode ip>:9083</value> <description>IP address (or fully-qualified domain name) and port of the metastore host</description> </property> <property> <name>datanucleus.autoStartMechanism</name> <value>SchemaTable</value> </property> </configuration> Step 4: Strat hive metastore $hive --service metastore Step 5: To view hive console $hive hive>show tables; OK Step 6: Before connecting to hive using java Start hiveserver by executing following command $hive --service hiveserver Apache pig installation Step 1: Untar the tarballs $tar -xvzf pig-0.12.0.tar.gz Step 2: Delete two jars (pig and pig-without hadoop jar) from pig home directory and add pig- withouthadoop.jar in pig installation directory (Uploaded in knowmax same path) Step 3: To open pig grunt $pig