Tuning Apache Ambari Performance for Big Data at Scale with 3,000 Agents
Download as PPTX, PDF1 like167 views
The document discusses optimizing the performance of Apache Ambari at scale with 3,000 agents, highlighting features and improvements introduced in Ambari 2.5, including service auto-restart, alert management, and enhanced database handling. It outlines strategies for testing at scale, details on upgrade processes, and enhancements in the Ambari Metrics System for better analytics and monitoring. Future enhancements for Apache Ambari 3.0 are also presented, including support for multi-instance services and anomaly detection capabilities.
Tuning Apache Ambari Performance for Big Data at Scale with 3,000 Agents
- 1. Tuning Apache Ambari Performance for Big Data at Scale with 3,000 Agents Alejandro Fernandez & Aravindan Vijayan June 13, 2017
- 2. Speakers Alejandro Fernandez Staff Software Engineer @ Hortonworks Apache Ambari PMC alejandro@apache.org Aravindan Vijayan Sr Software Engineer @ Hortonworks Apache Ambari PMC avijayan@apache.org
- 3. What is Apache Ambari? Apache Ambari is the open-source platform to deploy, manage and monitor Hadoop clusters
- 4. Poll Have heard of Ambari before? Have tried it, in sandbox or production?
- 5. 2,335 1,784 1,764 1,499 1,688 April ’15 Jul-Sep ’15 Dec ’15-Feb ’16 Aug-Nov ’16 Mar’17 20.5k commits over 4.5 years by 80 committers/contributors AND GROWING # of Jiras
- 6. Exciting Enterprise Features in Ambari 2.5 Core AMBARI-18731: Scale Testing on 2500 Agents AMBARI-18990: Self-Heal DB Inconsistencies Alerts & Log Search AMBARI-19257: Built-in SNMP Alert AMBARI-16880: Simplified Log Rotation Configs Security AMBARI-18650: Password Credential Store AMBARI-18365: API Authentication Using SPNEGO Ambari Metrics System AMBARI-17859: New Grafana dashboards AMBARI-15901: AMS High Availability AMBARI-19320: HDFS TopN User and Operation Visualization Service Features AMBARI-2330: Service Auto-Restart AMBARI-19275: Download All Client Configs AMBARI-7748: Manage JournalNode HA
- 7. Testing at Scale: 3000 Agents Agent Multiplier • Each Agent has own hostname, home dir, log dir, PID, ambari-agent.ini file • Must not have actual Hadoop daemons running since collides • Agent Multiplier can bootstrap 50+ Agents per VM • Tried Docker + Weave before and not very stable for networking Agent 1 VM Agent 1 Agent 50 VM
- 8. PERF Stack Dummy Services: • Happy: always passes • Sleepy: always timesout • Grumpy: always fails Semi-Real Services (only Configs): • Zookeeper • HDFS • YARN • HBASE Real Services: Ambari Metrics (server is an actual host with metrics for JVM, CPU, garbage collection, DB queries) PERF Stack
- 9. PERF Stack with Dummy Services Install: No-Op Configure: No-Op Start: If security is enabled => kinit Create PID file Stop: Delete PID file Status: If no PID file => raise ComponentIsNotRunning Exception
- 10. Testing at Scale: 3000 Agents Ambari Server Scale (server cannot tell the difference) Kerberos Stack Advisor Alerts Rolling & Express Upgrade UI Testing
- 12. PERF fixes in Ambari 2.5 SERVER AMBARI-17836 EU POST Call Takes More Than 3 Mins on 900 Node Cluster AMBARI-19224 Perf: Host Checks response is way too big (320MB for 1200 nodes) AMBARI-19242 Ambari Server hangs generating tasks for install+start when adding 600 hosts at a time UI AMBARI-19907 On certain actions, there's no feedback from the UI when API takes a long time to respond AMBARI- 18976 Config History request execution time depends on config versions count AMBARI- 19962 Clicking on the login button (or hitting page refresh) to see the dashboard takes a while on a 1000-node cluster
- 13. VIDEO
- 14. Optimize for Large Scale export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms2048m -Xmx8192m ambari-env.sh ambari.properties 10 Hosts 50 Hosts 100 Hosts > 500 Hosts agent.threadpool.size.max 25 35 75 100 alerts.cache.enabled true alerts.cache.size 50000 100000 alerts.execution.scheduler.maxThreads 2 4 • Dedicated database server with SSD • MySQL 5.7 and DB tuning • Purge old Ambari history: commands, alerts, BP topology, upgrades. https://community.hortonworks.com/articles/80635/optimize-ambari-performance-for-large-clusters.html
- 15. Background: Upgrade Terminology Manual Upgrade The user follows instructions to upgrade the stack Incurs downtime
- 16. Background: Upgrade Terminology Manual Upgrade The user follows instructions to upgrade the stack Incurs downtime Rolling Upgrade Automated Upgrades one component per host at a time Preserves cluster operation and minimizes service impact
- 17. Background: Upgrade Terminology Express Upgrade Automated Runs in parallel across hosts Incurs downtime Manual Upgrade The user follows instructions to upgrade the stack Incurs downtime Rolling Upgrade Automated Upgrades one component per host at a time Preserves cluster operation and minimizes service impact
- 18. Automated Upgrade: Rolling or Express Check Prerequisite s Review the prereqs to confirm your cluster configs are ready Prepare Take backups of critical cluster metadata Perform Upgrade Perform the HDP upgrade. The steps depend on upgrade method: Rolling or Express Register + Install Register the HDP repository and install the target HDP version on the cluster Finalize Finalize the upgrade, making the target version the current version
- 19. Process: Rolling Upgrade ZooKeeper Ranger/KMS Hive Spark Knox Storm Slider Flume Finalize or Downgrade Core Masters Core Slaves HDFS YARN HBase Clients HDFS, YARN, MR, Tez, HBase, Pig. Hive, etc. Oozie Kafka Falcon Accumulo On Failure, • Retry • Ignore • Downgrade NN1 NN2 DataNodes
- 20. Process: Express Upgrade Stop High-Level: Spark, Storm, etc Back up HDFS, HBase, Hive Change Stack + Configs Zookeeper Knox Storm Slider Flume Finalize or Downgrade Ranger/KMS Stop Low-Level: YARN, MR, HDFS, ZK Falcon Accumulo HDFS YARN MapReduce2 HBase Hive Oozie On Failure, • Retry • Ignore • Downgrade 1001 Hosts in Parallel 1001 Hosts in Parallel
- 21. Total Time: 2:53 13:16 26:26 Scales linearly with # of hosts
- 22. Total Time: 0:32 1:14 2:19 Scales linearly with # batches (defaults to 100 hosts at a time) 5.4 X 10.7 X 11.4 X faster than RU
- 23. Change Batch Size agent.package.parallel.commands.limit=100 /etc/ambari-server/conf/ambari.properties <group xsi:type="cluster|restart|stop" name="CUSTOM" title="CUSTOM"> <parallel-scheduler> <max-degree-of-parallelism>360 </max-degree-of-parallelism> </parallel-scheduler> nonrolling-upgrade-*.xml Number of hosts to install packages in parallel in a single batch Batch size during Express Upgrade for commands like restart Default is 100
- 24. Auto-Retry stack.upgrade.auto.retry.timeout.mins=10 stack.upgrade.auto.retry.check.interval.secs=10 /etc/ambari-server/conf/ambari.properties Detect any failure & retry every x seconds for up to y mins Unsupervised Use Cases: • Host is down • Intermittent failure • Service Checks
- 25. Future of Apache Ambari 3.0 • Service multi-instance (e.g., two ZK quorums) • Service multi-versions (Spark 2.0 & Spark 2.2) • YARN assemblies & services • Patch Upgrades: upgrade individual components in the same stack version, e.g., just DN and RM in HDP 3.0.*.* with zero downtime • Host-Ordered Upgrades: Host at a time – All components in a host in parallel OR – Swap the host • Ambari High Availability
- 26. Ambari Metrics at Scale - Agenda • Motivation • Introduction & Architecture • Finer Design Details • Feature Highlights • Grafana • Operational Statistics Laying down the building blocks Loading up on features Elasticity & Integration points Future : Fitter and Smarter • Distributed Collector Story • Sinks & Sources • Extensibility • Tackling Scale Issues • Anomaly Detection
- 27. Ambari Metrics Service - Motivation First version released in Ambari 2.0.0 Need service level aggregation as well as time based Pluggable and Extensible Alerts based on metrics system Ability to perform analytics based on a use case Allow fine grained control over aspects Ability to scale past a 1000 nodes Limited Ganglia capabilities OpenTSDB - GPL license and needs a Hadoop cluster
- 28. AMS Architecture • Metrics Collector – API daemon • Sinks – Service daemons configured to publish metrics • Monitors – lightweight daemon for system metrics • Managed HBase (embedded / distributed) • Phoenix schema designed for fast reads • Grafana support from version 2.2.2 Ambari Collector API Grafana Phoenix HDP Services System MONITORSSINKS Metrics Collector
- 29. Finer Design Details Performance improvements • Composite Metric Key • Automatic pre-defined Region splits based on cluster services Schema • Client Side aggregation (V1) • Server Side aggregation to utilize the region splits for query parallelism (V2) Aggregation • FIFO compaction policy • Normalizer • Native HBase timestamp support HBase & Phoenix
- 30. Feature Highlights AMSSimple POST API Rich GET API Aggregation and Down sampling Metadata API Highly Tunable Abstract Sink Impl. Aggr Functions Point in time & series support Top N Rate & Diff Rate of collection Granularity of data Skip aggregation Whitelisting
- 31. Features: Aggregations Precision Data30 SECOND 5 MINUTE 1 HOUR 1 DAY 5 MINUTE 1 HOUR 1 DAY Across Hosts CLUSTER AGGREGATES HOST AGGREGATES Across Time
- 32. Features: Metrics Get API Metrics GET API Metric Names Hostnames AppId Start Time End Time Precision Required Optional InstanceId
- 33. Grafana Integration • Contributed an AMS data source plugin to Grafana. • Leverages the metric collector’s GET metrics API that helps visualize – Cluster level and Host level aggregate data – Different precision levels – Aggregation Functions – sum, min, max, avg – Post Processing Functions - Rate and Diff AMS Grafana Plugin
- 34. Grafana Dashboards • Templated dashboards created by service experts • Helps identify metric based problems in the cluster much faster than manual troubleshooting
- 35. Operational Statistics Disk Usage Write Load
- 36. AMS – Distributed collectors Motivation & Requirements Horizonta l Scalabilit y High Availabilit y Distribute Heavy weight operation Automati c Failover Distributed Lock Problem Service Discovery Persistent Distributed checkpoints
- 37. AMS – Helix usage - Primitives Instance Resource Partitions / Replicas H I II Host Cluster Aggregators O M State Model Online Offline Host Cluster
- 38. Cluster Zookeeper METRICS MONITOR YARN Kafka Flume METRICS SINKS HBase Storm Hive NiFi HDFS METRICS COLLECTOR HBase Master + RS Phoenix Aggregators Collector API Helix Participant METRICS COLLECTOR HBase Master + RS Phoenix Aggregators Collector API Helix Participant AMS Distributed Collector Arch Details
- 39. Distributed Writes Sinks & Monitors Bootstrap Initial set of collectors Sharding Hostname based sharding strategy Expiry Refresh on expiry Discovery Every collector advertises list of known collectors Collector Down Retry and refresh after defined # of failures Fallback Talk to Zookeeper if no live collector known
- 40. Integrations (Sinks) Monitoring Ambari performance just became easier! Added Ambari Server as a sink to AMS. JVM, Database (EclipseLink) metrics
- 41. AMS Future Work
- 42. Extensibility • Currently, AMS hard wired to write metrics only to managed HBase • Ability to Tee to another Storage or Sink (RMP-5443) External Sink support Metrics Collector
- 43. Tackling Scale issues Metric Name App Instance 12-16 byte UUID Revamped Schema : UUID Row Key Aggregation at Scale Online On-Demand• Cluster aggregation Metric Monitor HBase RS Sink Datanode Sink Local Aggregato r 5 min data Raw data Metrics Collector • Time aggregation
- 44. Metric-Based Anomaly Detection • 10000+ metrics, 100+ graphs, 20+ dashboards • Feedback from users - Tell me when my system is in a bad shape or is going to be! • What we need - Anomaly detection system Point in Time Anomalies Trend & Correlation Anomalies Self adjusting system Continuous Event Processing Ambari Alerts Integration Historical Anomalies querying
- 45. Resources Contribute to Ambari: https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide Referenced Articles:https://community.hortonworks.com/articles/80635/optimize-ambari-performance-for- large-clusters.html Many thanks to the ASF, audience, and event organizers. Lightning-Question Round github.com/avijayan avijayan@apache.org github.com/afernandez alejandro@apache.org
Editor's Notes
- #3: Project Management Committee Committer and PMC since 2014 Co-architects for Rolling & Express Upgrade, PERF stack, Atlas integration
- #6: Single pane of glass. Services, deploy, manage, configure, lifecycle Provision on the cloud, blueprints High Availability + Wizards Metrics (dashboards) Security, using MIT Kerberos or others Alerts (SNMP, emails, etc) Host management Views framework Stack Upgrades
- #7: We just released Ambari 2.5 in March with almost 1800 Jiras (features and bug fixes), and have 2-3 major releases per year. 0.9 in Sep 2012 1.5 in April 2014 1.6 in July 2014 2.0.0=1,688 2.1.0=1,866 2.1.1=276 2.1.2=379 2.2.0=798 2.2.1=206 2.2.2=495 2.3.0 was not used 2.4.0=2,189 2.4.1=33 2.4.2=113 2.5.0=1,523 2.5.1=261 Cadence is 2-3 major releases per year, with follow up maintenance releases in the months after. http://jsfiddle.net/mp8rqq5x/2/
- #8: Services: Service Auto-Start: UI for enabling which services should restart automatically if they die or host is rebooted. Download all client configs with a single click Wizard for NameNode HA, to save namespace, format new namenode, move existing JN, more than 3 JNs Security: Passwords are now stored in the credential store by default for Hive, Oozie, Ranger, and Log Search Support Kerberos token authentication Core: Perf fixes for up to 2.5k agents, PERF stack, and simulate up to 50 agents per VM. DB Consistency Checker can now fix itself during mismatches. AMS: New Grafana dashboard for Ambari Server performance (JVM, GC, top queries) HDFS TopN User and Operation Visualization – Shows most frequent operations being performed on the NameNode, and more importantly who’s performing those operations, intervals are 1, 5, 25 min sliding window. Ambari Metrics Collector now supports active-active HA to distribute the load. Alerts & Log Search: Default SNMP alert now sends traps using an Ambari-specific MIB Log Search now has settings for max backup file size and max # of backup files.
- #14: Just mention that 6 fixes.
- #16: Dynamic availability Allow host_count to be specified instead of host_names As hosts register, they will be matched to the request host groups and provisioned according to to the requested topology When specifying a host_count, a predicate can also be specified for finer-grained control 3 Terabytes since units is in MB
- #17: Express Upgrade: fasted method to upgrade the stack since upgrades an entire component in batches of 100 hosts at a time Rolling Upgrade, one component at a time per host, which can take up to 1 min. For a 100 node cluster with
- #18: Express Upgrade: fasted method to upgrade the stack since upgrades an entire component in batches of 100 hosts at a time Rolling Upgrade, one component at a time per host, which can take up to 1 min. For a 100 node cluster with
- #19: Express Upgrade: fasted method to upgrade the stack since upgrades an entire component in batches of 100 hosts at a time Rolling Upgrade, one component at a time per host, which can take up to 1 min. For a 100 node cluster with
- #20: Can install new bits ahead of time, side-by-side with existing bits, while the cluster is running Option to downgrade
- #21: One component per host at a time.
- #22: Batches of 100 hosts in parallel. Must first stop on the current version, take backups, change version, and start on the newer stack
- #30: Motivation: Limited Ganglia capabilities OpenTSDB – GPL license and needs a Hadoop cluster Aggregation at multiple levels: Service, Time Scale, tested at 3000 nodes Fine grained control over retention, collection intervals, aggregation Pluggable and Extensible
- #31: Composite Metric Key – Metric Name, App, Host, Instance and Time Automatic region splits during cluster deploy based on deployed services Client Side Aggregation V1 & Server Side aggregation – V2 FIFO compaction policy selects only files which have all cells expired. Raw data collected by FIFO compactor after rolled up into higher aggregates. Normalizer – automatic splitting and merging of regions based on traffic Native HBase timestamp support in Phoenix - Map one column from the table to HBase cell's timestamp and to use Scan with timestamp ranges in queries with WHERE clause.
- #37: Resources used by AMS Write load 500 Sinks – 867M rows Disk Usage 500 sinks – 81GB
- #38: Motivation Horizontal Scalability – cluster size, easy addition of new collectors, restart resiliency HA for metrics service Heavy weight operations – write Automatic failover – sinks Challenges Distributed locking – leader election, partition Service discovery Distributed checkpoints Helix - A cluster management framework for partitioned and replicated distributed resources
- #40: HBase HA