GraceHopper 2015, Cluster Management for Big Data in the Cloud
Download as PPTX, PDF0 likes520 views
This document discusses Cloudman, a cluster management software from Qubole that launches and manages big data clusters on major public clouds. Cloudman automates cluster provisioning, configuration of big data software stacks, and cluster lifecycle management. It aims to provide highly optimized compute costs. The document outlines some of Cloudman's key features and challenges in areas like autoscaling clusters based on workload and addressing differences between cloud providers.
GraceHopper 2015, Cluster Management for Big Data in the Cloud
- 1. 2015 Cloudman: Cluster Management for Big Data in the Cloud Swati Singhi December 3, 2015 #GHCI15 2015
- 2. 2015 ▪Fixed pre-provisioned capacity ▪Variable and Unpredictable workloads ▪Do not scale well ▪Expensive ▪On-site IT team Challenges of On-premise Big Data Infra
- 3. 2015 Cloud offers salvation... ▪Stretches with the workload ▪Pay-as-you-go ...but brings its own challenges ▪Moving data to the cloud ▪Security/Privacy Big Data in the Cloud
- 4. 2015 Qubole as Big Data Service ▪Enables Big Data on the cloud ▪Enterprise ready deployments ▪On major public clouds ▪Simple and Fast
- 5. 2015 Cloudman ▪Qubole’s Cluster management software ▪Launches half a million nodes per month ▪Works across AWS, GCE and Azure ▪Provides higher level APIs
- 6. 2015 Cloudman Goals ▪Automated cluster provisioning ▪Configure Big Data Stack ▪Manage cluster lifecycle ▪Highly optimized cost of compute
- 7. 2015 UI SDK API Cloudman Layers of Big Data as a Service
- 9. 2015 Challenges ▪ Autoscale based on workload ▪ Abstractions to address differences in behaviors of each cloud provider Examples − Image creation and registration − Configuring clusters
- 10. 2015 ▪ Launched automatically when needed ▪ Expands automatically if the load is high ▪ Terminate the cluster with no running jobs ▪ Remove nodes at billing boundary Autoscaling Clusters
- 11. 2015 insert overwrite table dest select … from ads join campaigns on …group by …; Map Tasks ReduceTasks Demand Supply Progress Master Slaves Job Tracker Cloudman Cloudman: AutoScaling
- 12. 2015 Image registration in AWS vs. Azure Image creation and registration
- 13. 2015 ▪ Image creation ▪ Public images in AWS ▪ Not well supported in Azure ▪ Images copied to user’s account in Azure Image creation and registration
- 14. 2015 ▪ Configure credentials −Storage and Compute keys ▪Configure the big data stack −Start appropriate s/w, example JobTracker and NameNode on Master and TaskTracker and DataNode on Slaves Cluster Configuration
- 15. 2015 Optimizing cost of compute in Cloud ▪ Utilize ephemeral compute instances to lower cost − AWS Spot Instances − GCE Preemptible VMs ▪ Challenges − Data loss − Big data job failures
- 16. 2015 Demo
- 17. 2015
- 18. 2015 Key Takeaways ▪ Highly efficient cluster management system ▪ Proven at scale in production ▪ Works on multiple clouds
- 19. 2015 Got Feedback? Rate and review the session on our mobile app – Convene For all details visit: http://ghcindia.anitaborg.org
- 20. 2015 Appendix
- 21. 2015 Architecture ▪ QDS has a user interface, Python and Java SDKs and APIs that allows users to talk to QDS and analyze data sets without knowing cluster management. ▪ A QDS user can submit primitive commands to logical clusters. ▪ The middleware layer communicates to the cloud orchestration layer called Cloudman ▪ Cloudman is responsible for spinning up clusters in the concerned cloud
- 22. 2015 ▪ One such example is Image creation and registration ▪ Procedure ▪ Precreate a machine image with all the the softwares to be deployed baked into it ▪ We start the cluster machines using this as the underlying image ▪ Saves us the time in deploying the softwares on the nodes after they are up ▪ This process is very different in all the cloud providers Image creation and registration
- 23. 2015 Cluster Configuration ▪Another operation that had to be implemented differently for each cloud ▪Startup scripts are used for to programmatically customize virtual machine instances ▪AWS and Google cloud had support for this ▪Azure did not support automatic execution of this script at the VM boot up time in the Centos VMs
- 24. 2015 ▪Hadoop clusters in QDS come up automatically when applications that require them are launched ▪If the load on the cluster is high, then the cluster automatically expands. ▪Cloudman automatically launches additional nodes which eventually join the running cluster and are able to pick up part of the workload Autoscaling Clusters