Running Enterprise Workloads in the Cloud

1 © Hortonworks Inc. 2011–2018. All rights reserved
Running Enterprise Workloads in the
Cloud
Richard Doktorics
Peter Darvasi

Who we are?
⬢ Peter Darvasi
- Partner Engineer at Hortonworks
- @pdarvasi
⬢ Richard Doktorics
- Software Engineer
- @doktoric

Agenda
⬢ What is Cloudbreak?
⬢ Enterprise checklist for big data in the cloud
⬢ Cloudbreak in da house
⬢ Questions

What is Cloudbreak?

Cloudbreak is a tool for provisioning Hadoop
clusters on any cloud infrastructure
Simplified Cluster Provisioning - prescriptive setup,
simple automation

Deploy on Public or Private
Clouds
Dynamically configure and manage
clusters on public or private clouds
(Amazon Web Services, Microsoft
Azure, Google Cloud Platform and
OpenStack)
Automated Scaling
Seamlessly manage elasticity
requirements as cluster workloads
change (Ambari Metrics / Prometheus)
Secured Cluster Access
Supports configuration defining
network boundaries and configuring
security groups
Highly Extensible
Recipes to run custom commands
Custom images

⬢ Cloudbreak Deployer (CBD)
– Written in Go and Bash
– Compiled into single binary
⬢ Micro-service architecture
– Each service runs in a Docker
container
– Each container is replaceable
with custom ones
– Services are handled with
docker-compose
Single node deployment

Enterprise checklist for big data in cloud

✓ Control and Automation
✓ Cloudy Services
✓ Security
✓ Enterprise-Grade Support
Checklist for enterprises in the cloud

1
0
© Hortonworks Inc. 2011–2018. All rights reserved
✓ Cloudy Services
✓ Security
✓ Simple UX
✓ Powerful CLI
✓ Autoscaling

1
1
Simplified UX

1
2
Create Credential Experience

1
3
Built-In Blueprints

1
4
Basic and Advanced Cluster Creation Experiences
BASIC ADVANCED

1
5
New Network and Security Group Choices
⬢ Network
– Create new Network and new
Subnet
– Choose existing Network and
existing Subnet
⬢ Security Groups
– Create new SGs
• Choose default SGs
(minimal set of ports)
• Create customized
– Choose existing SGs

1
6
Powerful CLI

1
7
Cloudbreak CLI: Designed for DevOps

1
8
“Show cli command” for every request

1
9
Auto-scaling

2
0
Auto-Scaling
⬢ Alerts: Create metric or time-based alerts for cluster scaling
⬢ Policies: Scaling policies adjust cluster size based on activity and workload
alerts
⬢ General Configurations: Boundaries and cooldown period

2
1
Auto-Scaling Time-Based Alert
Fire at 10:15 am everyday

2
2
Auto-Scaling Metric-Based Alert
Fire after NodeManagers are in
CRITICAL state for 10 minutes

2
3
Auto-Scaling Policies
⬢ Define the Scale Adjustment (Node Count/Percentage/Exact size)
⬢ Select the HostGroup (to Scale)
⬢ Select Alert (which when fired, executes the Policy)

2
4
Auto-Scaling General Configurations
⬢ Cooldown Period (between scaling actions)
⬢ Minimum and Maximum Cluster size (boundaries)
Cluster size
boundaries
Time Interval between
two Autoscale events

2
5
✓ Cloudy Services
✓ Security
✓ Cloud Resources
✓ Hortonworks DataFlow
✓ Custom Images

2
6
Cloud Resources:
RDBMS + LDAP

2
7
Cloud Resources: RDBMS and LDAP/AD = Dynamic Blueprints
⬢ Background:
– Cluster configuration often includes external database (for Hive, Ranger, etc) and LDAP/AD configs
– It’s a challenge to know the different Blueprint configuration choices per service across the stack
⬢ Dynamic Blueprints:
– Ability to manage External Sources (e.g. RDBMS and LDAP/AD) outside of your Blueprint
– Cloudbreak will inject the configurations into your Blueprint
– Simplifies reuse of external cloud resources
– Simplifies your Blueprints -> don’t have to know all the configurations for each component

2
8
Dynamic Blueprints: RDBMS/LDAP
⬢ Built-In Components:
– Atlas, Ranger, Hadoop, Hive LLAP, Hive, Ambari, Oozie, Druid, SuperSet
JDBC/LDAP
properties in
Blueprint for the
Component?
Yes
Use Blueprint as-is,
no Component
configuration
property injection
No Inject Component
configuration
properties
Perform property
variable
replacement
S
E

2
9
At-Motion Workloads:
Hortonworks DataFlow

3
0
Hortonworks DataFlow in CloudBreak
⬢ Default blueprint: “Flow Management: Apache NiFi”
HDF 3.1: NiFi, Ambari, Ambari Metrics, ZooKeeper

3
1
HDF - cluster creation

3
2
HDF - cluster creation

3
3
Custom Images

3
4
Background: Cloudbreak
1. Cloudbreak creates VM instances using a default base images.
2. Cloudbreak installs Ambari on a VM instance.
3. Cloudbreak instructs Ambari to install an HDP Cluster on other VM instances.
Cloudbreak
RHEL 7
HDP Node
VM
HDP Node
VM
HDP Node
VM
HDP Node
VM
HDP Node
VM
HDP Node
VM
HDP Cluster

3
5
Background: Cloudbreak Default Images
⬢ By default, Cloudbreak uses default base public images when creating VM instances.
Cloud Standard Image Operating System
AWS Amazon Linux 2017
Azure CentOS 7.x
Google Cloud Platform CentOS 7.x
OpenStack CentOS 7.x
Support for Custom Images provides a way for Cloudbreak
users to leverage their own custom image (not the default
image) when creating VM instances.

3
6
Making a Custom Image: Overview
Create the
Custom Image
Register the
Custom Image
in Cloudbreak
Use the Custom
Image when
Creating a
Cluster
1 2 3

3
7
Creating the Image: Code Repository
⬢ Instructions, Packer scripts and Salt states in public GitHub repository
– https://github.com/hortonworks/cloudbreak-images
⬢ An understanding of Packer and Salt is useful
– Packer creates infrastructure
– Packer runs Salt provisioner
⬢ Customer should clone the repository and build on it

3
8
Creating the Image: Example Scenarios
SCENARIO APPROACH
For AWS: I don’t want Amazon Linux
and instead want RHEL 7
1. Setup repository and AWS environment
2. Use the repository tools to build a RHEL 7 image
make build-aws-rhel7
I don’t want OpenJDK and instead
want Oracle JDK
1. Setup repository and environment
2. Turn on Oracle optional state
3. Use the repository tools to build an image
For AWS: I don’t want Amazon Linux
and instead want MY RHEL 7
** This is an advanced scenario**
1. Setup repository and AWS environment
2. Change the source base image
3. Use the repository tools to build a RHEL 7 image
make build-aws-rhel7

3
9
Use the Custom Image: Create Cluster (UI)
⬢ Create Cluster > General Configuration > Advanced
Choose image
catalog
Adjust the Ambari +
HDP repos (if you want)
Choose image
you registered

4
0
Pre-Warmed Images
PROS CONS
Prewarmed: OS + pre-installed Ambari and
HDP
Cluster installs are faster
No internet connection is needed
Cannot change the Ambari or HDP versions,
cannot use local repositories
Base: OS only Cluster installs take longer Can change the Ambari or HDP Versions, or
use local repositories
Base Images Prewarmed Images

4
1
✓ Cloudy Services
✓ Security
✓ Kerberos support
✓ LDAP integration
✓ Proxy configuration

4
2
Cluster Security:
Kerberos

4
3
What is Kerberos
⬢ Strongly authenticating and establishing a user’s identity is the basis for secure access in
Hadoop. Users need to be able to reliably “identify” themselves and then have that
identity propagated throughout the Hadoop cluster.
⬢ Kerberos is the de-facto system for authenticating access to distributed services

4
4
Cloudbreak: Support for Enabling Kerberos
Goal
Provide a way for Cloudbreak users to create clusters that
are Kerberos enabled
Approach
Ambari exposes a lot of Kerberos options
Leverage Ambari Kerberos options and avoid re-creating
Ambari Kerberos experience
Pragmatic prescriptive options on-top

4
5
Cloudbreak: Enable Kerberos Security
⬢ Create Cluster > Security > Advanced
⬢ [ ] Enable Kerberos Security

4
6
Options: Use Existing KDC or Use Test KDC
Use Existing
KDC
Use Test KDC
Advanced
Basic
- Not for production use. For testing and
evaluation purposes only.
- Installs and configures an MIT KDC on the
master node.
- Configures the cluster to leverage that KDC.
- Provide basic information
about your existing KDC.
- Ambari Kerberos descriptors
are generated automatically.
- Provide basic information
about your existing KDC.
- Provide your own Ambari
Kerberos descriptors.

4
7
Cloudbreak + LDAP/AD

4
8
Cloudbreak User AuthN
⬢ Goal: Configure Cloudbreak to provide for external User AuthN to LDAP/AD
– CloudFoundry UAA (User Account and Authentication Server) is the foundation
https://github.com/cloudfoundry/uaa
⬢ Two parts:
1. Configure Cloudbreak to talk to external LDAP/AD
2. Configure which group(s) can access Cloudbreak

4
9
Step 1: Configure Cloudbreak to talk to LDAP/AD
⬢ On the Cloudbreak host, create:
/var/lib/cloudbreak-deployment/uaa-changes.yml
⬢ Define LDAP profile for users and groups
Cloudbreak LDAP/AD

5
0
Step 2: Configure which group(s) can access Cloudbreak
⬢ Configure which group(s) are authorized to access Cloudbreak:
cbd util execute-ldap-mapping [group]
cbd util delete-ldap-mapping [group]
⬢ To authorize users in the ”Analysts” group to access Cloudbreak:
cbd util execute-ldap-mapping cn=Analysts,ou=Groups,dc=hortonworks,dc=local

5
1
Proxy configuration

5
2
Limited Outbound Internet Access
⬢ Handle enterprise scenarios where:
– Limited (or restricted) outbound internet access, and/or
– Required use of a Proxy to obtain internet access
Cloudbreak
Cluster Hosts
Cloudbreak
• Docker Hub
• Cloudbreak dependencies
• Default Image Catalog
Cloudbreak and Cluster Hosts
• Cloud Provider APIs
• HDP or HDF platform repositories
http/sproxy
(optional)

5
3
Internet Access via Proxy
Cloudbreak
Proxy Setup
Clusters Proxy
Setup
How does Cloudbreak
communicate thru a proxy to
get to the internet (and to the
cluster hosts)?
How do the Cluster Hosts
communicate thru a proxy to
get to the Internet?

5
4
Cloudbreak: Proxy Setup
⬢ Setup Docker Environment to use Proxy
– Modify the Docker service to set HTTP_PROXY and HTTPS_PROXY (and NO_PROXY)
https://docs.docker.com/config/daemon/systemd/#httphttps-proxy
⬢ Setup Cloudbreak to use Proxy in Profile
⬢ Advanced Profile option “HTTPS_PROXYFORCLUSTERCONNECTION=true|false”
– Defaults to “false”
HTTP_PROXY_HOST=your-proxy-host
HTTPS_PROXY_HOST=your-proxy-host
PROXY_PORT=your-proxy-port
PROXY_USER=your-proxy-user
PROXY_PASSWORD=your-proxy-password

5
5
Cloudbreak: Advanced Proxy Scenarios
SCENARIO #1: Proxy for internet, not clusters SCENARIO #2: Proxy for internet and clusters

5
6
Clusters: Register Proxy Configuration
⬢ External Sources > Proxy Configurations
(optional)
if proxy requires
authentication

5
7
Clusters: Configure Proxy for Cluster Hosts
⬢ Create Cluster > Advanced > External Sources > Configure Proxy
• Configures yum with “proxy” settings
• Configures Ambari Server with “httpProxy”
settings

5
8
✓ Cloudy Services
✓ Security
✓ SmartSense integration
✓ Flex support

5
9
Cloudbreak in da house

6
0
⬢ Have an internal hosted Cloudbreak service for…
– our CI/CD pipeline
– testing and prototyping HDP and HDF services
– have self-service clusters for QE/SE/PS teams
Main use cases

6
1
⬢ Run Cloudbreak in HA (High Availability) mode
– Ability to recover flows in case of node failure
– Avoid master-slave design / leader election problems
⬢ Scale Cloudbreak as we desire
– Distribute each cluster related flow
– Cannot run 2 flows for the same cluster at the same time (e.g: 2 upscale flows)
– Flow cancellation must be handled
⬢ Scale the Web UI
– Had to introduce a Redis cluster for the session store
⬢ Scale every other service as well
⬢ Find a tool that makes it easy to deploy these services to multiple nodes
Our technical goals

6
2
⬢ Not because it’s fancy..
⬢ Evaluated Kubernetes, Swarm, Mesos, Rancher
⬢ Open source / Active community with hands-on experience
⬢ Many cloud providers already supports it
⬢ Lots of tooling behind it / API / CLI / Helm / Ansible / Salt
⬢ Integration with most of the cloud providers
– Provision Load Balancer (GCP, AWS, Azure)
– Use object stores to share data (Ceph, S3, GCP bucket, Azure Storage Account)
– Dynamic volume provisioning / Persistent disk (EBS, Azure Blob)
Why Kubernetes?

6
3
Thank you

Running Enterprise Workloads in the Cloud

More Related Content

What's hot (20)

Similar to Running Enterprise Workloads in the Cloud (20)

More from DataWorks Summit (20)

Recently uploaded (20)

Running Enterprise Workloads in the Cloud