Managing Apache HAWQ with Apache AMBARI
1 like602 views
This document summarizes a presentation about managing Apache HAWQ, an open source massively parallel processing (MPP) database, using Apache Ambari. It discusses how Ambari integrates with HAWQ for installation, configuration, topology recommendations, high availability, alerts and more. Challenges in the integration are addressed as HAWQ is not part of the Hortonworks Data Platform stack. The presentation recommends future work for Ambari like supporting automated HAWQ upgrades and enabling dynamic configuration reloads without requiring a service restart.
Managing Apache HAWQ with Apache AMBARI
- 1. Managing Apache HAWQ with Apache AMBARI Apache Ambari Meetup - June 27, 2016 Alexander Denissov Bhuvnesh Chaudhary Mithun Mathew Apache HAWQ (incubating) Apache Ambari
- 2. Hadoop-native SQL query engine and advanced analytics MPP database that offers: 1 2 3 4 5 interactive query execution high performance machine learning algorithms tools for Data Analysts and Data Scientists processing for large and complex data sets APACHE HAWQ (incubating)
- 3. APACHE HAWQ (incubating) ARCHITECTURE
- 4. HAWQ - AMBARI INTEGRATION SCOPE Installation and configuration Topology and configuration recommendations and validations Kerberos and High Availability support HAWQ Master - HAWQ Standby failover Service and Component Alerts Visual Widgets
- 5. HAWQ - AMBARI INTEGRATION EFFORT Praises Ambari’s pluggable architecture makes integrations like this possible and easy Kerberos setup is fully metadata driven — major kudos! Challenges HAWQ is not part of the HDP stack and is not available in Ambari out-of-the box Advanced features and wizards require JavaScript code modifications Driven by the team of engineers at Pivotal Developed integrations from basic to more advanced Invaluable support from Ambari Community THANK YOU!
- 8. RECOMMEND AND VALIDATE CONFIGS
- 9. HAWQ SERVICE SUMMARY PAGE
- 11. ACTIVATE HAWQ STANDBY WIZARD Activate HAWQ Standby Wizard (Manual Operation)
- 12. ACTIVATE HAWQ STANDBY WIZARD HAWQ Standby Master promoted to HAWQ Master Add HAWQ Standby Master action becomes visible
- 13. HAWQ ALERTS Status of HAWQ Components Communication issues between HAWQ Components
- 14. HAWQ AMBARI FUTURE INTEGRATION Support automated upgrade independent of stack Ongoing related work: AMBARI-14854, AMBARI-12885 Ambari requires service restart for pushing configuration changes. What if, the service can reload configurations without restart? Ongoing related work: AMBARI-17241 HAWQ Upgrade Dynamic Configuration Reload Display query history Manage resource queues HAWQ View
- 15. Currently Ambari does not support configuration changes without restarting service Some parameters do NOT require restart! HDFS dfs.heartbeat.interval, dfs.namenode.heartbeat.recheck-interval HAWQ default_hash_table_bucket_number, hawq_rm_memory_limit_perseg DYNAMIC CONFIGURATION RELOAD
- 16. Currently Ambari does not support configuration changes without restarting service Some parameters do NOT require restart! HDFS dfs.heartbeat.interval, dfs.namenode.heartbeat.recheck-interval HAWQ default_hash_table_bucket_number, hawq_rm_memory_limit_perseg DOWNTIME!!! Consequence of Restarting the Service: DYNAMIC CONFIGURATION RELOAD
- 17. No more DOWNTIME!!! DYNAMIC CONFIGURATION RELOAD
- 19. Ambari UI Show Reload Button Desired Configs Updater HeartBeat Processor ServiceComponentHost (updates requires_reload) Request Handling and Execution Ambari Agent Reload Method Ambari Server Ambari Web Ambari Agent Reload Method Ambari Agent Reload Method Ambari Agent Reload Method REST API POST PUT GET Heartbeats invalidates requires_reload user specifies COLLABORATION DIAGRAM
- 20. Feedback? AMBARI-17241 Reload vs Restart - Are they mutually exclusive? THINGS TO DECIDE Seriously, purple?