Hortonworks Technical Workshop: Apache Ambari

Apache Ambari Workshop
Stacks, Blueprints, Views
We Do Hadoop
Ajay Singh, Ali Bajwa, Sean Roberts
Partner Solution Engineering
March-2015

Who
Ajay Singh
- Director
- ajaysingh@hortonworks.com
Ali Bajwa
- Engineer
- abajwa@hortonworks.com
Sean Roberts
- Engineer, London & EMEA
- @seano

Schedule
● Ambari Overview
● Stacks
● Blueprints
● Stack Services
● Views

© Hortonworks Inc. 2011 – 2015. All Rights Reserved
How do you Operate a Hadoop Cluster?
Apache Ambari is a
platform to provision,
manage and monitor
Hadoop clusters

Apache Ambari Themes
Operate Hadoop at
Scale
Deliver the core operational capabilities to provision,
manage and monitor Hadoop clusters at scale.
Integrate with the
Enterprise
Robust API for integration with existing enterprise systems,
such as Teradata Viewpoint and Microsoft SCOM.
Extend for the
Ecosystem
Provide extensible platform for Customers, Partners and
the Community to, such as Stacks and Views.

Hadoop Operations Platform
Apache Ambari is a 100% open source platform for
provisioning, managing and monitoring Apache Hadoop
clusters
AMBARI WEB
Others
compute
&
storage
. . .
. . .
. .
compute
&
storage
.
.EXTEND
AMBARI REST API
AMBARI SERVER
PROVISION | MANAGE | MONITOR
Integration with Existing Tools
OPERATE
AMBARI STACKS

Innovation at the Speed of Community
An active 100% open source Community:
• 90+ Contributors / 45+ Committers
• 270+ Ambari User Group Members
2014
Dec
Graduates to Apache Top Level Project
2013
Dec
Apache Ambari 1.7 Released
Adds support for Hortonworks Data Platform 2.2
Adds new operational and extensibility capabilities

Recent Ambari Releases
Ambari 1.6.0
May 2014
Ambari 1.5.1
Apr 2014
Ambari 1.5.0
Apr 2014
Ambari 1.6.1
Jul 2014
Introduced
Ambari Stacks
Introduced
Ambari Blueprints
HDP
2.1 GA
Ambari 1.7.0
Dec 2014
HDP
2.2 GA
Introduced
Ambari Views

Extensibility Features
• To add new Services (ISV or otherwise) beyond HDP Stack
• To customize a Stack for customer specific environments
• To use Ambari for automating cluster installations
• To share best practices on layout and cluster configuration
• To extend and customize the Ambari Web UI
• Add new capabilities, customize existing capabilities
Stacks
Blueprints
Views
Goal: Extend Ambari without hard-coding in Ambari
1.5.0
1.7.0
1.6.0

Anatomy of Ambari Extension Points
Ambari
Server
Ambari
AgentAmbari
AgentAmbari
Agent
Ambari
Web
Stacks
Stacks
Stacks
javajs python
Ambari
Views
Ambari
Stacks
Ambari
Blueprints
Ambari API

Partners Integrating With Ambari

Ambari Community
Apache Ambari: http://ambari.apache.org
● wiki: https://cwiki.apache.org/confluence/display/AMBARI/Ambari
● mail lists: https://ambari.apache.org/mail-lists.html
● issues: https://issues.apache.org/jira/browse/AMBARI
Hortonworks:
● forum: http://hortonworks.com/community/forums/forum/ambari/
● social media: twitter, facebook, slideshare, linkedin, ...

Partner Resources
This Workshop:
● github: http://git.io/jmz4 (https://github.com/abajwa-hw/ambari-workshops)
● blog: http://hortonworks.com/blog/apache-ambari-technical-workshop/
Past & Future Workshops:
● http://hortonworks.com/partners/learn/
Upcoming meetups and events:
● http://hortonworks.com/events/

Stack Terminology
Term Definition Examples
STACK
Defines a set of Services, where to obtain the software
packages and how to manage the lifecycle.
HDP-2.0, HDP-1.3.3
SERVICE Defines the Components that make-up the Service. HDFS, NAGIOS, YARN
COMPONENT
The building-blocks of a Service, that adhere to a
certain lifecycle.
NAMENODE, DATANODE,
OOZIE_SERVER
CATEGORY The category of Component. MASTER, SLAVE, CLIENT

Stack Mechanics
Stacks define Services + Repos
What is in the Stack, and where to get the bits
Each Service has a definition
What Components are part of the Service
Each Service has defined lifecycle commands
start, stop, status, install, configure
Lifecycle is controlled via command scripts
Ability to define “custom” commands
AMBARI
SERVER
Stack
Command
Scripts
Service
Definitions
AMBARI
AGENT/S
AMBARI
AGENT/S
AMBARI
AGENT/S
pythonxml
Repos

Stacks Support Inheritance
HDP 2.0 Stack
HDP 2.1 Stack
• Defines a set of Service definitions
• Default service configurations and command scripts
• Overrides any Service definitions, commands and
configurations
• Adds new Services specific to this Stack

Deploying clusters of any scale with ease
Ambari Blueprints

Before Ambari
● Manual installation
● Manual configuration
● Manual distribution of configuration
● XML
● XML
● XML ...

Manual Install with Ambari Wizard

Automation with Ambari Blueprints
Stack Definition
Component
Layout
& Configuration
BLUEPRINT
BLUEPRINT INSTANTIATE CLUSTER

2 HTTP calls for a Deployed Cluster
{ "Blueprints": {
"stack_name": "HDP",
"stack_version": "2.2"
},
"host_groups": [
{ "name": "workers",
"components": [
{ "name": "DATANODE" },
{ "name": "JOURNALNODE" },
{ … … }
},
{ "name": "master-1",
"components": [
{ "name": "NAMENODE" },
{ … … }
}
]}
{ "Blueprints": {
"blueprint": "myblueprint",
"default_password": "awesome",
"configurations": [
{ "hive-site": { "properties": {
"hive.execution.engine": "tez" }}}],
"host_groups": [
{ "name": "master-1",
"hosts": [ { "fqdn": "hdp00" } ]},
{ "name": "workers"
"hosts": [ { "fqdn": "hdp01" },
{ "fqdn": "hdp01" },
{ "fqdn": "hdp.." },
{ "fqdn": "hdp99" }]}
]}
1) POST /api/v1/blueprints/myblueprint 2) POST /api/v1/clusters/mycluster

Export Cluster to Blueprint
GET /api/v1/clusters/my_cluster?format=blueprint
{ "configurations" : [
{ "cluster-env" : { "cluster_name" : "sandbox", ... }},
{ "hive-site" : { "hive.execution.engine" : "tez", ... }}
],
"host_groups" : [
{ "name" : "host_group_1",
"configurations" : [ ],
"components" : [
{ "name" : "NAMENODE" }, { "name" : "HIVE_SERVER"}, ...
],
"cardinality" : "1"
}],
"Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.2" }
}

Live Demo
Command snippets: http://nbviewer.ipython.org/github/seanorama/ambari-
bootstrap/blob/master/api-examples/ambari-workshop-blueprints.ipynb
For API examples including Python versions in an ipython notebook:
https://github.com/seanorama/ambari-bootstrap/tree/master/api-examples

Files needed to create an Ambari service
• Service definition file (metainfo.xml)
• Service configuration file (myservice-config.xml)
• Service configuration parameters (params.py)
• Service master python script (master.py)
Not always needed:
• Service status configuration file (myservice-env.xml)
• Service status configuration parameters (status_params.py)

Service Definition (metainfo.xml)
● name – the internal name of the service (must be unique)
● displayName – the public display name of the service
● version – the version of the service
● component category – is the component a MASTER or CLIENT (one or more)
● script – the script that will be executed during service install, start, stop
● package – packages to be installed via yum during install
<service>
<name>MAVEN</name>
<displayName>Maven</displayName>
<comment>Apache Maven Project Build Tool</comment>
<version>1.0.0</version>
<components>
<component>
<name>MAVEN_CLIENT</name>
<displayName>Maven</displayName>
<category>CLIENT</category>
<commandScript>
<script>scripts/client.py</script>
<scriptType>PYTHON</scriptType>

Service Configuration (myservice-config.xml)
● name – the internal name of property (must be unique)
● value – default value for property
● property-type – e.g. whether property is password (optional)
● description – tool tip help text for property
<configuration>
<property require-input="true">
<name>vnc.password</name>
<value></value>
<property-type>PASSWORD</property-type>
<description>VNC password</description>
</property>
<property>
<name>vnc.geometry</name>
<value>1400x860</value>
<description>VNC screen geometry/resolution</description>
</property>

Service Status Configuration (myservice-env.xml)
● status related parameters (e.g. location of file containing pid of service)
● name – the internal name of property (must be unique)
● value – default value for property
● description – tooltip help text for property
<configuration>
<property>
<name>pid_file</name>
<value>/var/run/ipython-notebook/notebook.pid</value>
<description>Process ID File</description>
</property>
</configuration>

Service Configuration Params (params.py)
Python script to read configuration properties from myservice–config.xml
from resource_management import *
# server configurations
config = Script.get_config()
user = config['configurations']['ipython-config']['ipython.user']
group = config['configurations']['ipython-config']['ipython.group']

Service Status Config Params (status_params.py)
Python script to read status related configuration properties from myservice–
env.xml
from resource_management import *
config = Script.get_config()
pid_file = config['configurations']['ipython-env']['pid_file']

Service master python script (master.py) - simple
If the Ambari service is a just wrapper around a Linux service,
the start/stop/status methods as as easy as…
def start(self, env):
Execute("service myservice start")
def status(self, env):
Execute("service myservice status")
def stop(self, env):
Execute("service myservice stop")

Service master python script (master.py)
● def install – called to install service
● def start – called to start service
● def stop – called to stop the service
● def status – called to get status of service
● def configure – called to configure service
def install(self, env):
#Install packages from metainfo.xml
self.install_packages(env)
self.configure(env)
#code to install service
def start(self, env):
import params
import status_params
#code to start service
def configure(self, env):
import params
env.set_params(params)
#code to re-configure service
def stop(self, env):
env.set_params(status_params)
self.configure(env)
Execute (format('kill `cat {stack_pidfile}` >/dev/null 2>&1'))
Execute (format("rm -f {stack_pidfile}"))
def status(self, env):
env.set_params(status_params)
check_process_status(status_params.stack_pidfile)

Let’s build some services!
Demo

Demo
• Build a MASTER service
• Ntpd
• Sample CLIENT service walk through
• Maven
• Use VNC service to setup a developer sandbox
• Installs Eclipse, IntelliJ, Maven, Spark
• Real world scenarios
• Data Science: iPython service to easily manage platform for data science on Hadoop
• Security: OpenLDAP, KDC, NSLCD services to easily setup security with minimal CLI work

Ambari Views Framework
Goal: enable the delivery of custom UI experiences in Ambari Web
Developers can extend the Ambari Web interface
• Views expose custom UI features for Hadoop Services
Ambari Admins can entitle Views to Ambari Web users
• Entitlements framework for controlling access to Views

Views Framework
Views Framework vs. Views
Page 38
Views
Core to Ambari
Built by
Hortonworks,
Community,
Partners

Views Framework
Views Framework vs. Views
Page 39
Views
Core to Ambari
Built by
Hortonworks,
Community,
Partners

View Components
• Serve client-side assets (such as HTML + JavaScript)
• Expose server-side resources (such as REST endpoints)
VIEW
Client-side
assets
(.js, html)
AMBARI WEB
VIEW
Server-side
resources
(java)
AMBARI SERVER
{rest}
Hadoop
and
other
systems

View Delivery
1. Develop the View (just like you would for a Web App)
2. Package as a View (basically a WAR)
3. Deploy the View into Ambari
4. Ambari Admins create + configuration view instance(s) and give
access to users + groups
Develop DeployPackage
Create
Instance(s)

Versions and Instances
• Deploy multiple versions and create multiple instances of a view
• Manage accessibility and usage

Choice of Deployment Model
• For Hadoop Operators:
Deploy Views in an Ambari Server that is managing a Hadoop
cluster
• For Data Workers:
Run Views in a “standalone” Ambari Server
Ambari
Server
HADOOP
Store & Process
Ambari
Server
Operators
manage the
cluster, may
have Views
deployed
Data
Workers use
the cluster
and use a
“standalone”
Ambari
Server for
Views

Views REST API
Resource Description
GET /api/v1/views List of available views
GET /api/v1/views/{viewName}/versions List of available view versions
GET /api/v1/views/{viewName}/versions/{version}
/instances
List of view instances in a given version
GET /api/v1/views/{viewName}/versions/{version}
/instances/{instanceName}
Details about a specific view instance.

View Deployment
Place view-package.jar on Ambari Server
/var/lib/ambari-server/resources/views
Restart Ambari Server to start deployment
ambari-server restart
Deployment registers the view with Ambari and extracts the view
/var/lib/ambari-server/resources/views/work
Once complete, the view is available to create instances
Dev Tip: UI assets in extracted view can be modified w/o restart

Views Framework Terminology
Term Description
View Definition Describes the view resources and core view properties such as name,
version and any necessary configuration properties.
View Package Packages the view client and server assets (and dependencies)
View Deployment Deploying a view into Ambari
View Version A specific version of a view
View Instance An instantiation of a specific view version
Framework Services View context, instance data, configuration properties and events

View Definition (view.xml)
name – the internal name of the view (must be unique)
label – the public display name of the view
version – the version of the view
parameter – (optional) metadata about view instance properties
resource – (optional) the names of the classes required to support a view sub-
resource
instance – (optional) property sets which define static view instances
<view>
<name>MYVIEW</name>
<label>My View</label>
<version>1.0.0</version>
<parameter>
<name>scope</name>
<description>The scope of the instance
</description>
</parameter>
</view>
Dev Tip: Use Static instances for development

Maven build file (pom.xml)
artifactId – determines name of jar
ambari-views 1.7.0.0 dependency ensures jar not required locally
<modelVersion>4.0.0</modelVersion>
<artifactId> hello-world-view</artifactId>
<packaging>jar</packaging>
<name>Hello World View</name>
<groupId>com.github.hortonworks</groupId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.apache.ambari</groupId>
<artifactId> ambari-views</artifactId>
<version> 1.7.0.0</version>
</dependency>

View Package
View Definition: view.xml
Resource / Service classes : JAX-RS annotated
UI classes : html, Servlets deployed as web app (WEB-INF/web.xml).
Application logic : Supporting classes
Dependencies : 3rd party jars or classes
├── WEB-INF
│ └── web.xml
├── org
│ └── apache
│ └── ambari
│ └── view
│ └── weather
│ ├── CityResource.class
│ ├── CityResourceProvider.class
│ ├── CityService.class
│ └── WeatherServlet.class
└── view.xml
view-
package.jar

View Context
• Available to the view server-side through injection.
• Provides access to instance configuration.
• Provides access to the view and instance attributes.
• Provides access to run time information about the current
execution context, including authenticated user principal.
public interface ViewContext {
public String getUsername();
public String getViewName();
public String getInstanceName();
public Map<String, String> getProperties();
public void putInstanceData(...);
public String getInstanceData(...);
}

Views Demo
Let’s build some views!

Demo
• Build example view with only HTML
• Hello World
• iFrame view
• Sample views with HTML and javascript w/o frameworks
• REST API explorer
• Google map
• Sample views using javascript framework
• HDFS Visualizer
• Sample servlet view (uses java)
• Hive query view

The End & Questions
This Workshop:
● github: https://github.com/abajwa-hw/ambari-workshops
● blog: http://hortonworks.com/blog/apache-ambari-technical-workshop/
Apache Ambari: http://ambari.apache.org/
● wiki: https://cwiki.apache.org/confluence/display/AMBARI/Ambari
● mail lists: https://ambari.apache.org/mail-lists.html
● issues: https://issues.apache.org/jira/browse/AMBARI
Hortonworks:
● forum: http://hortonworks.com/community/forums/forum/ambari/
● social media: twitter, facebook, slideshare, linkedin, ...

Ambari Views: Where Can I Learn More?
https://cwiki.apache.org/confluence/display/AMBARI/Views
https://github.com/apache/ambari/blob/trunk/ambari-views/docs/index.md
https://github.com/apache/ambari/tree/trunk/ambari-views/examples
https://github.com/apache/ambari/tree/trunk/contrib/views

Ambari Views: Developer resources
https://nodesource.com/blog/your-first-nodejs-package
https://docs.angularjs.org/tutorial/
http://www.jeromecukier.net/blog/2012/09/04/getting-to-hello-world-with-
d3/
http://jsbin.com (e.g. http://jsbin.com/hirirazu/3/edit?html,output)
More sample views:
https://github.com/apache/ambari/tree/trunk/ambari-views/examples

Hortonworks Technical Workshop: Apache Ambari

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Hortonworks Technical Workshop: Apache Ambari (20)

More from Hortonworks (20)

Recently uploaded (20)

Hortonworks Technical Workshop: Apache Ambari