OpenNebulaConf2017EU: Elastic Clusters for Data Analysis by Carlos de Alfonso, EUBra-BIGSEA
0 likes156 views
The document discusses the EUBra-BIGSEA project which aims to provide elastic data analysis clusters in OpenNebula. The project will deploy big data analytics services on cloud infrastructure using TOSCA specifications. It will define a security framework and use the Infrastructure Manager to deploy virtual machine images and configure instances. The Elastic Clusters for Cloud Computing (EC3) will leverage CLUES to dynamically scale the clusters based on resource usage, deploying new VMs when needed and undeploying idle ones.
OpenNebulaConf2017EU: Elastic Clusters for Data Analysis by Carlos de Alfonso, EUBra-BIGSEA
- 1. Elastic Clusters for Data Analysis in OpenNebula in EUBra-BIGSEA Carlos de Alfonso, on Behalf of the EUBRA-BIGSEA Consortium www.eubra-bigsea.eu | @bigsea_eubr 1
- 2. BIGSEA in a NutShell www.eubra-bigsea.eu | @bigsea_eubr • A European-Brazilian Consortium aiming at • European-Brazilian User Scenario will make use of the Data Analytics API and the Programming model interface. • The Programming model interface will use the services deployed using TOSCA standard ― Service specifications will be instantiated on a cloud IaaS, automatically managing Quality of Service. ― The BigData Analytics services will run on the cloud infrastructure. • A security framework will be defined to provide security and privacy. QoS IaaS Big Data Analytics Program. model interface Security User Scenario Access tokens Assurance BigData Privacy Assurance Access tokens TOSCA specs Requests Development IDE 2
- 3. BIGSEA Architecture www.eubra-bigsea.eu | @bigsea_eubr 3
- 4. Infrastructure Manager (IM) www.grycap.upv.es/im www.eubra-bigsea.eu | @bigsea_eubr •Two approaches to VMI management •Deploy existing vanilla images (plain OS) and configure instances using tools to recreate the desired conf. •Example: Instantiate a plain Ubuntu 14.04 and use Ansible to automatically install a full LAMP (Apache, MySQL, PHP) stack. •Create specific VMIs for different Clouds from templates. •Deployments based on recipes, configuration and contextualization services. •Enables platform-agnostic or hybrid deployments. •Supports TOSCA. VMI catalogue *
- 5. Elastic Clusters on the Cloud (EC3), http://www.grycap.upv.es/ec3 • EC3 (Elastic Cloud Computing Cluster) leverages CLUES (http://www.grycap.upv.es/clues) to create elastic virtual clusters in the Cloud. - No upfront investment. - Customizable. - Usage-dependent cost. - Automatic elasticity. 5 www.eubra-bigsea.eu | @bigsea_eubr
- 6. Resource-level elasticity • When a new job in a Mesos framework is submitted, EC3 will check if there are enough resources to fulfil the request - EC3 will request the Cloud Management Framework to deploy new VMs to allocate the necessary resources for the new framework • EC3 uses Infrastructure Manager to install and re-configure the software. • A snapshot from the first WN is stored to speed-up the deployment of further nodes. - When resources become idle for a while, they are undeployed. - Not-elastic resources are not undeployed. 6 www.eubra-bigsea.eu | @bigsea_eubr
- 7. Platform-agonstic Deployment www.eubra-bigsea.eu | @bigsea_eubr 7
- 8. Vertical elasticity at the resource level • VMs are allocated from a CMF that manages a general- purpose on-premise cloud • Physical limitations are normally at the memory size rather than at the CPU share. • A physical machine can be shared among different deployments • Both BIGSEA stack or other one. • Or due to increased isolation. • Oversubscribing memory can enable reaching higher number of VMs per physical node, not bounding to a specific partitioning. • A VM from one deployment can ”release” the free memory to other VMs in the node. 8www.eubra-bigsea.eu | @bigsea_eubr
- 9. The ONE Backend www.eubra-bigsea.eu | @bigsea_eubr 9
- 10. Conclusions •EUBra-BIGSEA aims at providing a set of cloud services to • Facilitate the deployment of complex multi-framework Big Data infrastructures. • Provide elasticity at the level of the physical resources and frameworks. • Include QoS Constraints. • Facilitate the use of customized environments. •Those cloud services are directly used by higher- level programming models hiding the platform particularities. 10www.eubra-bigsea.eu | @bigsea_eubr
- 11. Thanks •The authors would also like to thank the Spanish "Ministerio de Economía, Industria y Competitividad" for the project "BigCLOE'' with reference number TIN2016-79951-R. 11