Apache Ambari - HDP Cluster Upgrades Operational Deep Dive and Troubleshooting

1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache Ambari - HDP
Cluster Upgrades
Operational Deep Dive and Troubleshooting
DATAWORKS Summit, Munich
April 5, 2017

Presenters
• Venkatraman Poornalingam (vpoornalingam@hortonworks.com)
• Principal Automation Engineer, Technical Support Team Hortonworks
• Part of Ambari and Upgrades SME team
• Vivek Sharma (vsharma@hortonworks.com)
• Staff Software Engineer, Ambari Quality Engineering Team
• Specializing on Ambari Upgrades and Views

Agenda
• Use Case
• Prerequisites for upgrade
• Upgrades Deep Dive
• Express Vs Rolling
• Internals
• Troubleshooting
• Ambari 2.5 Upgrades new Feature
• Best Practices

Sam’s Upgrade Story
• Sam is a Hadoop Administrator working with WBC Inc.
• Manages several HDP clusters using Ambari
• Is planning to upgrade a cluster with following config:
• 300 nodes, HDP-2.3.6, Ambari-2.2.2.0
• Hive, Spark, HBase, Oozie, Kerberos-Managed by Ambari
• Interested in Hive LLAP for his applications, Oozie Workflow View

Sam reviews HDP Stack

Sam’s Upgrade Plan
• After reviewing Hortonworks current product stack
• Discusses with his CIO/Team
• Decides to upgrade to the following
• Ambari 2.5
• HDP 2.6
• Sam has to research / plan for
• A Runbook consisting of
• Prerequisites
• Upgrade Method
• Troubleshooting in case of issues
• Complete upgrade
• Downtime
• Identifies appropriate Ambari user roles for the upgrade
• New Stack registration can be done only by Ambari Administrator role
• Upgrade can be done by Ambari Administrator and Cluster Administrator

Sam in Research mode …

Ambari Upgrade Workflow
Post Ambari upgrade, complete upgrade for AMS, Infra, SmartSense and Logsearch

HDP Cluster Upgrade Workflow

Upgrade Planning
• Backup of configs, Databases - Hive, Oozie,Ranger
• Important to have DB access available to Ambari Administrator
• Check 3rd party software compatibility with newer HDP version
• Handling Tech Preview services / Custom Services
• Ensure Ambari pre-checks pass
• API:/api/v1/clusters/c1/rolling_upgrades_check?fields=*&UpgradeChecks/repository_
version=2.6.0.3-8&UpgradeChecks/upgrade_type=NON_ROLLING
• Disk space availability:
• New software installation (in /usr/hdp/)
• Backups during Upgrade (/tmp/)
• Check and ensure software dependencies are resolved
• Example, yum check dependencies; echo $?, Should return 0
• Identify list of hosts which are
• In maintenance mode
• To be decommissioned
• Has software installation failures

Sam decides to Deep Dive

Express Upgrade Orchestration
Upgrade Pack Location on Ambari server:
/var/lib/ambari-server/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml
Config pack:
/var/lib/ambari-server/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml

Magic of Symbolic Links!
● hdp-select /usr/hdp/current/$comp-name/ -> /usr/hdp/$version/$comp
Example:
● conf-select /etc/$comp/conf -> /usr/hdp/$version/$comp/conf -> /etc/$comp/$version/0
Example:
– Syntax:
– hdp-select set hive-server2-hive2 2.6.0.3-8
– conf-select create-conf-dir --package hive --stack-version 2.6.0.3-8 --conf-version 0
– conf-select set-conf-dir --package hive --stack-version 2.6.0.3-8 --conf-version 0
Pre-Upgrade /usr/hdp/current/hive-server2-hive2 -> /usr/hdp/2.5.3.0-37/hive2
Post-Upgrade /usr/hdp/current/hive-server2-hive2 -> /usr/hdp/2.6.0.3-8/hive2
Pre-upgrade /etc/hive2/conf -> /usr/hdp/current/hive-server2-hive2/conf -> /etc/hive2/2.5.3.0-37/0
Post-upgrade /etc/hive2/conf -> /usr/hdp/current/hive-server2-hive2/conf -> /etc/hive2/2.6.0.3-8/0

Rolling upgrade orchestration
Upgrade Pack Location on Ambari server:
/var/lib/ambari-server/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml
Config pack:
/var/lib/ambari-server/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml

EU Vs RU Performance (Controlled Environment)

Service Configurations - Merges
property_x property_y property_z property_x
HDP 2.3 foo (default) 120 Didn’t exist foobar
HDP 2.6 bar (default) deprecated baz bar
Post Upgrade bar Property deleted baz foobar

Sam decides to upgrade Dev
Cluster

Development Cluster Upgrade
● 50 Node cluster
● Starts a Runbook
● Completes Pre-requisites identified during planning phase; keeps a watch on the time taken
● Upgrades Ambari (yum upgrade, ambari-server upgrade; takes about 45 minutes)
● Verifies cluster is operational
● Completes registration and installation of new HDP version (ahead of time, takes about 30 minutes to
complete)
● Runs API to do pre-check
● Allocates 4 Hours for the upgrade
● Starts Express Upgrade at the scheduled time

Troubleshooting
● Checks
○ ambari-server.log
○ namenode logs
○ ambari-agent.log in Namenode
● And then…
ambari-agent.log → ambari-agent status
Troubleshooting is no different compared to any other Ambari Issues

Upgrade Completed!
• Finalize Later – for Application verification
• Suggests Application team to run basic application testing and finalizes within 2 days (including 3rd
party applications)
• If cluster isn’t finalized, the space usage on HDFS would increase and could lead to severe performance issues
• Checks for version details in Ambari UI and finds all in place!

Sam in Research mode…

Fine Tuning Upgrade parameters
• Support for auto-retry of tasks
• Fault tolerance options at the start and during Upgrade - skip service check failures, skip slave failures
• Batch size during package installation is controlled via a config in ambari.properties
• agent.package.parallel.commands.limit=100
• In the Express upgrade packs, the batch size can be modified from the default value:
<parallel-scheduler>
<max-degree-of-parallelism>100</max-degree-of-parallelism>
</parallel-scheduler>

Ambari Upgrade – Failure due to DB inconsistencies
23
• Ambari upgrade - constraint violation
• Review Ambari logs
• Identify table reporting the violation
• Restore Ambari DB
• Fix the violation
• Restart Ambari Upgrade
• DB Consistency check introduced from Ambari 2.4
• Verify if DB consistency is being skipped while starting Ambari
• In Previous versions, this could happen due to
• Failed installation / deletion using API’s

Ambari Schema Changes during HDP Upgrade

Performance issues during upgrade
25
● Save namespace takes too long
○ Older versions with large heap size
○ Attempt save namespace before upgrade and ensure it works good
○ Increase agent.task.timeout in ambari.properties if required
● Too many entries in host_role_command
○ It may be necessary to remove entries from the host_role_command table if the size of the table has grown excessively
large in order to reduce the query times for "IN_PROGRESS" requests.
○ This operation can’t be performed during upgrade

How to get summary of current upgrade status?
26
• Invoke the following Ambari API call:
• http://<ambari-server>:8080/api/v1/clusters/c1/upgrades
• From the output of above, identify the latest upgrade id
• http://<ambari-server>:8080/api/v1/clusters/c1/upgrades/441
• To get information upto upgrade_item level:
• http://<ambari-
server>:8080/api/v1/clusters/c1/upgrades/441?fields=upgrade_groups/upgrade_it
ems/UpgradeItem/status,upgrade_groups/upgrade_items/UpgradeItem/context,upgra
de_groups/UpgradeGroup/title
• To get information up to task level:
• http://<ambari-
server>:8080/api/v1/clusters/c1/upgrades/441?fields=upgrade_groups/upgrade_it
ems/tasks/Tasks/status,upgrade_groups/upgrade_items/tasks/Tasks/command_detai
l,upgrade_groups/upgrade_items/tasks/Tasks/stderr

Upgrade States
27
"upgrade_items" : [
{
"href" : "http://vpamb2010.novalocal:8080/api/v1/clusters/Ambari21/upgrades/441/upgrade_groups/106/upgrade_items/1",
"UpgradeItem" : {
"cluster_name" : "Ambari21",
"context" : "Restarting NodeManager on vpamb2012.novalocal",
"group_id" : 106,
"request_id" : 441,
"stage_id" : 1,
"status" : "HOLDING_FAILED"
}
},
Upgrade States:
●IN_PROGRESS
●HOLDING
●FAILED/HOLDING_FAILED/SKIPPED_FAILED
●TIMEDOUT/HOLDING_TIMEDOUT
●ABORTED
●PENDING/QUEUED
●COMPLETED

Service fails to start due to Circular Symlink issue
28
STDERR while starting Oozie service:
packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-
packages/resource_management/core/providers/system.py", line 177, in action_create raise Fail("Applying %s failed, looped
symbolic links found while resolving %s" % (self.resource, path))resource_management.core.exceptions.Fail: Applying
Directory'/usr/hdp/current/oozie-client/conf' failed, looped symbolic links found while resolving /usr/hdp/current/oozie-
client/conf
Fix:
conf-select create-conf-dir --package oozie-client --stack-version $version --conf-version 0
conf-select set-conf-dir --package oozie-client --stack-version $version --conf-version 0
ln -s /etc/oozie/2.3.2.0-2950/0 /usr/hdp/2.3.2.0-2950/oozie/conf

Post RU, Hive applications are failing
29
● Hive is started with port number 10010 instead of 10000 post upgrade
● Either Configurations need to be updated or HiveServer2 needs to be restarted with the older port number
● Rolling upgrade is not supported for Hive from HDP 2.6
○ Ambari 2.5 would give a warning while upgrading - “HiveServer2 does not currently support
rolling upgrades. HiveServer2 will be upgraded, however existing queries which
have not completed will fail and need to be resubmitted after HiveServer2 has
been upgraded.”

What’s new in Ambari 2.5 for upgrades?
30
● Auto Start of services
● Delete older version of the Software
● AMBARI-18435 Releases space used by older versions post upgrade. Previously this had to be done
manually. For eg,
curl 'http://c6401.ambari.apache.org:8080/api/v1/clusters/cl1/requests' -u admin:admin -H "X-Requested-By: ambari" -X POST -
d'{"RequestInfo":{"context":"remove_previous_stacks", "action" : "remove_previous_stacks", "parameters" : {"version":"2.5.0.0-
1245"}}, "Requests/resource_filters": [{"hosts":"c6403.ambari.apache.org, c6402.ambari.apache.org"}]}'
● Upgrade history
● Pulls all data about upgrades/downgrades from Ambari DB and displays in UI

Sam’s Runbook for Cluster
upgrade at WBC

Customized Upgrade Runbook
32
• Sam writes up a Runbook for WBC Inc. cluster upgrades which includes
• Upgrade Planning
• Installing packages ahead of time
• Checking disk space in hosts
• Choosing the right Upgrade method
• Deleting older versions if not required (keep the current and new one intact)
• Backup method for Databases and Configurations
• Stopping any Jobs which would restart services in the system and disable AUTO_RESTART of services in
Ambari
• Upgrading Development cluster
• Table to document issues faced during Development
• Time taken for the Upgrade activity
• Documents prerequisites including
• No changes to stack during upgrade
• No new installation / No new hosts etc
• Reviewing list of supported Databases in documentation

Thanks
Q & A

Apache Ambari - HDP Cluster Upgrades Operational Deep Dive and Troubleshooting

More Related Content

What's hot (20)

Similar to Apache Ambari - HDP Cluster Upgrades Operational Deep Dive and Troubleshooting (20)

More from DataWorks Summit/Hadoop Summit (20)

Recently uploaded (20)

Apache Ambari - HDP Cluster Upgrades Operational Deep Dive and Troubleshooting

Editor's Notes