Apache Ambari Meetup - AMS & Grafana
- 1. Ambari Metrics System (AMS) & Grafana Aravindan (avijayan@apache) Prajwal (prao@hortonworks)
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Intro to Grafana Grafana integration Demo AMS - Overview AMS - Features & highlights AMS – API Overview Support utilities Contributors : Sid Wagle, Dmytro Sen, Myroslav Papirkovskyy, Aravindan Vijayan & Prajwal Rao
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Grafana
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Grafana Introduction Open source dashboard builder integrated with AMS. Available from Ambari-2.2.2 Pre-defined host level and service level (HDFS, HBase, Yarn etc) dashboards. Added to Ambari through API after upgrade
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved AMS - Grafana Integration AMS is added as a Grafana datasource Service dashboards are created if not already present – System, HDFS, HBase, YARN (2.2.2) – Hive, Kafka, HBase per user and per table metrics, Storm (2.4.0) Grafana Supports Basic Auth & Https
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Demo
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved AMS Overview
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ambari Metrics Service - Motivation Limited Ganglia capabilities OpenTSDB – GPL license and needs a Hadoop cluster Need service level aggregation as well as time based Alerts based on metrics system Ability to scale past a 1000 nodes Ability to perform analytics based on a use case Allow fine grained control over aspects like: retention, collection intervals, aggregation Pluggable and Extensible First version released with Ambari 2.0.0
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved AMS Architecture Custom Sinks – HDFS, YARN, HBASE, STORM, KAFKA, FLUME, ACCUMULO Monitors – lightweight daemon for system metrics Collector – API daemon + HBase (embedded / distributed) Phoenix schema designed for fast reads Managed HBase Grafana support from version 2.2.2 High level component arch AMBARI Collector API GRAFANA HBASE PHOENIX HDP SERVICES SYSTEM MONITORSSINKS METRICS COLLECTOR
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Cluster Zookeeper METRICS MONITOR YARN KAFKA FLUME METRICS SINKS HBASE STORM HIVE NIFI HDFS METRICS COLLECTOR HBASE Master + RS PHOENIX Aggregators Collector API Helix Participant METRICS COLLECTOR HBASE Master + RS PHOENIX Aggregators Collector API Helix Participant AMS Arch Details - HA
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ambari Metrics Service Active-Active Collector configuration (HBase fully managed) Partition / Failover aggregate functions using Apache Helix Deterministic client side distribution of write load with failover Automatic discovery for Sink and Ambari (No restart needed) AMS – Distributed / HA architecture
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ambari Metrics Service Automatic pre-defined Region splits based on cluster services Aggregator v2 implemented to utilize the Region splits for query parallelism Ambari Metrics Cache to improve availability and reduce request/response footprint Major HBase/Phoenix - HDP improvements influenced by AMS: – FIFO compaction policy enabled for all AMS tables – Normalizer to do automatic region split/merge for large tables – Native HBase timestamp support to optimize date range queries in Phoenix Performance improvements - Under the hood
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Feature Highlights, API
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ambari Metrics Service Ambari native widgets for service level metrics (2.1.0) Ambari Metrics UI - Grafana integration (2.2.2) Pre-built Grafana dashboards to improve cluster debug-ability (2.2.2) Alerts based on metrics system in Ambari (2.2.2) Container metrics stored in AMS HBase with Zeppelin dashboards (2.4.0) Fully distributed mode with failover (2.5.0) Feature Highlights sorted by release version
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Aggregations ( Schema ) Request Interval Ambari Metrics Service - Aggregators Precision Data30 SECOND 5 MINUTE 1 HOUR 1 DAY 5 MINUTE 1 HOUR 1 DAY Across Hosts Across Time CLUSTER AGGREGATES HOST AGGREGATES < 2 hours 2 hrs – 1 day 1 day - 30 days > 30 days
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved GET – Timeseries & Point in Time data – Aggregate & Post Processing functions – Precision POST – Used by Monitors and Sinks Discovered metrics info All hosted applications Ambari Metrics Service - API API Metrics API Metadata API
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ambari Metrics Service - API Metrics GET API Metrics GET API <Metrics Collector Host>:6188/ws/ v1/timeline/met rics metricNames hostNames appId startTime endTime precision Required Optional instanceId
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Metadata API Ambari Metrics Service - API Metrics Metadata Hosts Metadata
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Debugging utils for AMS-Grafana Older AMS instance Export script Exported metrics Mimicked AMS endpoint (Flask Server) Import script Grafana datasource ambari/ambari-server/src/main/resources/scripts/export_ams_metrics.py
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Future AMS – External mode AMS – Tee to external storage by providing pluggable sink interface AMS – Failover with temp storage for HDFS Grafana – Support for AMS HA Grafana - More analytic dashboards Grafana – Alerts integration with Grafana 3.2.0
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Q & A
Editor's Notes
- #9: Ganglia – default metric system for Ambari. Diff to customize, no failover, single disk writes, unreliable, cannot scale past 1000 nodes without special hardware OpenTSDB – no pre-aggregations
- #10: SQL schema No secondary indices Scale for writes
- #11: Ambari-2.5.0 will support AMS HA Helix – task partitioning & service discovery Restarting daemons not needed. Auto discovered Client side
- #20: Some context. Target audience