Jumbo the Hadoop cluster bootstrapper
0 likes144 views
The document outlines 'Jumbo,' a Hadoop cluster bootstrapper that simplifies cluster provisioning and management through a Python CLI tool and trusted DevOps technologies. It highlights features such as fast environment setup, reusable code, and support for multiple development environments, including automated component placement and high availability. Future enhancements aim to support more services and improve cluster management with smart topology features.
Jumbo the Hadoop cluster bootstrapper
- 2. Jumbo Presentation and future of the Hadoop cluster bootstrapper Marrakech October 2018 2
- 3. What is Jumbo? ● Hadoop cluster bootstrapper 1. Describe your cluster in minutes 2. Let Jumbo provision everything 3. Start using your cluster Marrakech October 2018 3
- 4. What is Jumbo? ● Python 3 CLI tool ● Script generator ● Leverages trusted DevOps tools: ○ Vagrant ○ Ansible ○ Ambari Marrakech October 2018 4
- 5. Why Jumbo? Need for multiple development environments: ● Fast and easy definition ● Fast provisioning ● Reusable code Marrakech October 2018 5
- 6. Jumbo User Interface Script generation and tools control Marrakech October 2018 6 Hadoop installation Node configuration VM provisioning VBox | KVM
- 7. Jumbo internals Marrakech October 2018 7 Script generation: Vagrantfile Jinja templating Ansible playbooks Python dict → YAML (Ansible group_vars) Ambari blueprint Python dict → JSON
- 8. ● service ● component ● cluster ● node Abstraction layer Marrakech October 2018 8 mycluster HDFS node01 node02 node03 NN SNN DN DN
- 9. Abstraction layer Marrakech October 2018 9 node types master master components of master-slave services sidemaster key components of non master-slave services worker slave components of master-slave services edge server components ldap security components
- 10. Available features ● Nodes (VM) definition ● Services topology ○ Automatic components placement ○ Automatic installation of service dependencies ○ Manual tuning with compatibility control ○ High Availability support Marrakech October 2018 10
- 11. Hadoop ecosystem Database SecurityProvisioning and monitoring Available features Supported services: Marrakech October 2018 11
- 12. ● Cluster templating ● Cluster security (Kerberos) ● Centralized versions and repositories management ● VM control within Jumbo Available features Marrakech October 2018 12
- 13. Jumbo in action Marrakech October 2018 13
- 14. What’s next for Jumbo? Marrakech October 2018 14 ● HDP - Hortonworks Data Platform ○ Support for all the services of the stack ○ HA support for all the services ● Smart cluster topology based on available resources
- 15. What’s next for Jumbo? Marrakech October 2018 15 ● Create a bundle model for services to ease contribution ○ Service definition ○ Ansible playbooks ● Support other stacks ○ HDF - Hortonworks DataFlow ○ CDH - Cloudera Distribution for Hadoop
- 16. Get involved! Facing problems using Jumbo? Missing a service for your development environment? github.com/adaltas/jumbo jumbo.adaltas.com Marrakech October 2018 16