POWERing your big data solution with IBM: open-source Hadoop on POWER

Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.
9.0
César Diniz Maciel
Executive IT Specialist
IBM Corporate Strategy
cmaciel@us.ibm.com
IBA4101 - POWERing your big data
solution with IBM: open-source
Hadoop on POWER

© Copyright IBM Corporation 2015
Session objectives
●
At the end of this session you should be able to understand
the options available to run open-source Hadoop on IBM
Power Systems, from building it from source code to using
packaged solutions.
●
You will also learn about integrated SW solutions and
appliances that leverage Power Systems technology to
accelerate Hadoop-based implementations.

Acknowledgements
●
This presentation is based on the work done by the IBM
Solutions Operating Environment (IBMSOE) team. I would
like to thank the contribution of the following experts from
IBM US:
– Corentin Baron
– Luke Browning
– Pascal Oliva
– Tony Reix
– Daniele Silvestre

Hadoop
●
From the hadoop website:
“The Apache Hadoop software library is a framework that allows for the
distributed processing of large data sets across clusters of computers using
simple programming models. It is designed to scale up from single servers to
thousands of machines, each offering local computation and storage. Rather
than rely on hardware to deliver high-availability, the library itself is designed to
detect and handle failures at the application layer, so delivering a highly-
available service on top of a cluster of computers, each of which may be prone
to failures.“
●
Actively developed by the open source community, and by several
vendors
– Several pre-packaged distributions of Hadoop, including from IBM – IBM
Infosphere BigInsights
– Hadoop market expected to reach 20 Billion in 2018 according to
Transparency Market Research

Hadoop components

Apache Spark – in-memory mapreduce

Open Source Hadoop on Power Systems
●
The IBM Solutions Operating Environment hosts buildable
Hadoop source trees optimized for Linux running on
POWER. These trees support both RedHat Enterprise
Linux (RHEL) v6.5 or later on big-endian with PowerVM and
Ubuntu v14.04 on little-endian with PowerKVM. It also
works with openSUSE 13.2 (big-endian). Other versions or
releases may also work, but have not been tested.
●
This build is based on the open source Hadoop tree, with
changes required for building on Power Systems, and with
IBM Java (optimized for POWER). The source is hosted on
github.

Why build from scratch?
●
Alternative for deploying Hadoop without purchasing
commercial software
●
Community-based development and support
●
Ability to run Hadoop on flexible environments (older
systems, different operating environments)
●
Desire to learn the details on the Hadoop framework

Hardware prerequisites
●
Hadoop itself does not have hardware requisites
– Designed to run in a cheap and unreliable system, and
to scale out
– Obviously, the faster and more reliable the system is,
the better the solution will be and easier to maintain
●
It can be built on POWER7 processor-based systems and
later
– Functionally equivalent in POWER7 or POWER8, little
endian or big endian

I like open-source, but I don't want to build from
scratch….
As it is with x86, there are pre-packaged solutions of Hadoop available for
Power Systems
●
IBM have the Infosphere BigInsights on Power Systems since 2012
– Version 3.2 available for POWER7 processor-based systems
– Now updated with the latest IBM Open Platform with Apache Hadoop
(IOP), for POWER8 processor-based systems
– IOP is available as a 100% open-source, free to use product, and a
value-add, fee-based additional components
●
Veristorm offers the Veristorm Data Hub, a 100% open-source Hadoop build
for POWER8, free to use.
Details on how to build from scratch on POWER at the
Backup section of the presentation

Any Power Systems server running RH 6.5
Trial version available at
https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?source=swg-beta-iibob

IBM Open Platform with Apache Hadoop
http://www-03.ibm.com/software/products/en/ibm-open-platform-with-apache-ha
doop

IBM Open Platform with Apache Hadoop
IBM® Open Platform with Apache Hadoop builds the
platform for big data projects and provides the most
current Apache Hadoop open source content. IBM offers
this open source Apache distribution as a free download
as well as a supported offering for all your Hadoop
workloads.
➔ 100% Apache Hadoop Open Source platform
➔ HDFS, YARN, MapReduce, Ambari, Hbase, Hive, Oozie,
Parquet, Parquet Format, Pig, Snappy, Solr, Sqoop,
Zookeeper, Open JDK, Knox, Slider
➔ Spark in-memory distributed compute engine

Download IBM Open Platform with Apache Hadoop

Installation and configuration
●
Installation involves adding a repository and installing Ambari
●
“The Apache Ambari project is aimed at making Hadoop management simpler
by developing software for provisioning, managing, and monitoring Apache
Hadoop clusters. Ambari provides an intuitive, easy-to-use Hadoop
management web UI backed by its RESTful APIs.“
●
After installation is complete, start the Ambari Server and perform the
installation via web browser
– ambariserver setup
– ambariserver start

23
IBM Systems Technical University, October 5-9 | Hilton Orlando
© Copyright IBM Corporation 2015. Technical
University/Symposia materials may not be

2
4
IBM BigInsights 4.1 Value Adds
IBM BigInsights
Analyst
Big SQL
Big SQL
BigSheets
BigSheets
Text Analytics
Text Analytics
Machine Learning on Big R
Machine Learning on Big R
Big R
Big R
IBM BigInsights
Data Scientist
Big SQL
Big SQL
BigSheets
BigSheets
IBM Value Adds
IOP – 100% open-source, free
Download and use for production
Optional fee-based support
IOP – 100% open-source, free
Download and use for production
Optional fee-based support

© 2014 IBM Corporation
2
5
IBM BigInsights Extends Value of IOP
 PLUS
– Big SQL for rich ANSI compliant SQL interface to Hadoop
– World class Text Analytics from IBM Research
– Distributed Frame Leverage R as a query language on Hadoop
– Scalable algorithms from IBM Research – Systemml
– GPFS-FPO for enterprise data life cycle manangement
– Browser-based excel-like analytics inteface
– Probabilistic matching engine on Hadoop
BigMatch
BigSql Text
Analytics
BigR
Machine
Learning
BigSheets

●
Veristorm Provides binaries of Hadoop for POWER8 free of charge
●
Compiled from the same IBMSOE source, in a joint work between IBM and
Veristorm
●
Supported with Ubuntu, RedHat and SuSE (all LE versions)
●
Provides the same functionality as the build compiled by source, but in a faster
and easier deployment way
●
Community support for the free package, and option of (paid) Enterprise
support
●
http://www.veristorm.com/veristorm-data-hub

Veristorm Data Hub
●
Prerequisites
– IBM JDK 1.7 or superior
– Python 2.6 or superior
– Postgresql-9.3+
– zlibc 0.9k-4.1 or greater
●
Installation can be done with the downloaded package, or via repository
– echo "deb
http://repo.veristorm.com/repos/vdh/apt/debian vdh
ppcvstore main" > /etc/apt/sources.list.d/vdh.list
– aptget update
– aptget install vdhppcambariserver

Managing the cluster with Ambari - Hortonworks

Managing the cluster with Ambari - Veristorm

Managing the cluster with Ambari - IOP

Conclusions
●
Open-source Hadoop is well regarded by customers
●
The ability to leverage the characteristics of the Power
Systems servers with the flexibility of Linux and Hadoop
makes it an interesting combination with little or no
knowledge curve
●
The free, community-supported Veristorm package
simplifies the implementation and management of Hadoop
on Power Systems.
●
IBM Open Platform for Apache Hadoop brings an IBM-
backed, 100% free open-source package for enterprise
environments.
●
As we go forward, focus on performance and management
makes the solution even more attractive.

© Copyright IBM Corporation
2015
Continue growing your IBM skills
ibm.com/training provides a
comprehensive portfolio of skills and career
accelerators that are designed to meet all
your training needs.
• Training in cities local to you - where and
when you need it, and in the format you want
– Use IBM Training Search to locate public training classes
near to you with our five Global Training Providers
– Private training is also available with our Global Training Providers
• Demanding a high standard of quality –
view the paths to success
– Browse Training Paths and Certifications to find the
course that is right for you
• If you can’t find the training that is right for you with our
Global Training Providers, we can help.
– Contact IBM Training at dpmc@us.ibm.com
33
Global Skills Initiative

Software prerequisites
●
Hadoop requires several components in order to run, and
others are required to build the binaries
●
Besides the typical development environment (C/C++
compiler, Java, build tools) there are libraries and Java build
tools that are required. Most of the tools are available with
the Linux distribution.
●
Depending on the Linux distribution, and release,
requirements do change
– Redhat 6.5 brings the snappy library precompiled.
Redhat 7 does not

Prerequisite install for Ubuntu Linux
root@cmaciel4:~# aptget install cmake automake
autoconf git openssl libssldev zlib1g fuse
libfusedev
Download and install IBM Java from here (if not already
installed)
●
Java 1.6 or superior is required
●
Set JAVA_HOME and PATH to point to the Java binaries
– Linux sources scripts upon login from /etc/profile.d
– Adding scripts for environmental variables under this directory makes it
easy to maintain
– /etc/profile.d/java.sh
●
export JAVA_HOME=/opt/ibm/ibmjavappc64le71
●
export PATH=$JAVA_HOME/bin:$PATH

Prerequisite install for Ubuntu Linux
●
Download and install gcc Advance Toolchain
– IBM Advance Toolchain for PowerLinux Documentation
– „The IBM Advance Toolchain for PowerLinux is a set of open source
development tools and runtime libraries which allows users to take
leading edge advantage of IBM's latest POWER hardware features on
Linux“
– gcc and libraries optimized for latest POWER processors
●
Add the repository per the documentation
●
aptget install advancetoolchainat8.0runtime advance
toolchainat8.0devel advancetoolchainat8.0perf
advancetoolchainat8.0mcorelibs
●
Set PATH to include the advance toolchain binaries
– export PATH=/opt/at8.0/bin:$PATH

Prerequisite install
Hadoop uses Apache Maven for SW project management, and
Apache Ant as a Java build tool.
„Apache Maven is a software project management and comprehension tool. Based on
the concept of a project object model (POM), Maven can manage a project's build,
reporting and documentation from a central piece of information.“
„Apache Ant is a Java library and command-line tool whose mission is to drive
processes described in build files as targets and extension points dependent upon
each other. The main known usage of Ant is the build of Java applications.“
Download Maven (version 3.3 or higher)
Download Ant

Hadoop can optionally use snappy, a compression library
developed by Google
– aptget install libsnappy1 libsnappydev
– Not available on Redhat – need to
compile from source

●
Set Maven and Ant environment variables
– export M2_HOME=/opt/apachemaven3.3.1
– export M2=$M2_HOME/bin
– export PATH=$M2:$PATH
– export ANT_HOME=/opt/apacheant1.9.4
– export PATH=$ANT_HOME/bin:$PATH
Install Google's Protocol Buffers
– aptget install protobufcompiler
libprotobufdev
– Not available on Redhat – need to
compile from source

Building Hadoop
●
Once all prerequisites are installed, run a final check:
– echo $JAVA_HOME
– echo $M2_HOME
– echo $ANT_HOME
– echo $M2
– echo $PATH
●
Check all commands run properly:
– gcc version
– java version
– ant version
– mvn version
– protoc version

Building Hadoop
●
If everything is working properly, retrieve the Hadoop code from
https://github.com/ibmsoe/hadoop-common (either using the zip file,
or cloning the tree using git).
●
To compile and install Hadoop in to Maven cache using JNI and
snappy use the following build command from the root of the
Hadoop tree directory (for example, hadoopcommon):
– <hadoop_common> $ mvn install Pnative DskipTests
Drequire.snappy
●
Install installs in local package repository
●
Pnative builds native code
●
DskipTests compiles without running tests in the binaries
●
Drequire.snappy uses the snappy compression libraries

Creating a distribution package
●
mvn package Pnative,dist Drequire.snappy DskipTests
Dtar
●
Tar file saved at
hadoop_common/hadoopdist/target/hadoopdist2.4.1.tar.gz
●
Archive contains all binaries required to run Hadoop

Hadoop basic commands
●
hadoop@cmaciel4:~/hadoopdist/hadoop2.4.1$ bin/hadoop version
– Hadoop 2.4.1
Subversion https://github.com/ibmsoe/hadoopcommon r
4d0782d33c2f6b91d34b44b15f09b142eab1a403
Compiled by hadoop on 20150415T21:39Z
Compiled with protoc 2.5.0
From source with checksum c38708d16c3c1bd89a3ab88f5aefa9
This command was run using /home/hadoop/hadoopdist/hadoop
2.4.1/share/hadoop/common/hadoopcommon2.4.1.jar
●
hadoop@cmaciel4:~/hadoopdist/hadoop2.4.1$ bin/hadoop fs ls
– Found 7 items
drwxrwxrx hadoop hadoop 4096 20150415 17:53 bin
drwxrwxrx hadoop hadoop 4096 20150415 17:53 etc
drwxrwxrx hadoop hadoop 4096 20150415 17:53 include
drwxrwxrx hadoop hadoop 4096 20150415 17:53 lib
drwxrwxrx hadoop hadoop 4096 20150415 17:53 libexec
drwxrwxrx hadoop hadoop 4096 20150415 17:53 sbin
drwxrwxrx hadoop hadoop 4096 20150415 17:53 share

POWERing your big data solution with IBM: open-source Hadoop on POWER

More Related Content

Similar to POWERing your big data solution with IBM: open-source Hadoop on POWER (20)

Recently uploaded (20)

POWERing your big data solution with IBM: open-source Hadoop on POWER