20150425 experimenting with openstack sahara on docker
Download as PPTX, PDF3 likes1,127 views
The document discusses the integration of Docker with OpenStack Sahara for big data technologies, highlighting benefits like improved performance, resource optimization, and fast deployment. Key features of Sahara include easy provisioning of Hadoop clusters, analytics as a service, and support for various job types. The text also covers architectural aspects, performance testing, and future contributions needed for enhancing big data solutions on cloud platforms.
20150425 experimenting with openstack sahara on docker
- 1. Big Data Technologies Experimenting with Openstack Sahara on Docker Weiting Chen weiting.chen@intel.com
- 2. BIG DATA TECHNOLOGY Legal Disclaimers No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. The products and services described may contain defects or errors known as errata which may cause deviations from published specifications. Current characterized errata are available on request. © 2015 Intel Corporation.
- 3. BIG DATA TECHNOLOGY Agenda Background How to use Docker with Sahara Performance Testing Conclusion
- 4. BIG DATA TECHNOLOGY Who We Are We are from Intel Big Data Technology Group. We push big data technology forward into OpenStack We contribute Sahara source code in OpenStack, bring Cloudera CDH 5.3 plugin in Kilo.
- 5. BIG DATA TECHNOLOGY Sahara Background • Sahara becomes a core project in Juno • Bring Hadoop into OpenStack • Add more features to Kilo release • Two Key Features 1. To provide users easily provisioning Hadoop clusters by specifying several parameters 2. Analytics as a Service for data scientist or analyst
- 6. BIG DATA TECHNOLOGY Sahara Key Features - Provision Cluster Create/Terminate Cluster • Heat API/Nova Direct API • Integrate with Neutron/Nova Network • Use Guide as a template • Anti-affinity Cluster Scaling • Add Node/Remove Node Support More Plugins in Kilo • Vanilla/Hortonworks Data Platform/Cloudera/Spark/MapR/Storm
- 7. BIG DATA TECHNOLOGY Sahara Key Features - Elastic Data Processing Support Job Type • Hive/Pig/MapReduce/MapReduce Streaming/Java/Spark/Shell/HBase Support Data Locality • Rack/Hypervisor/Swift Data Source • Internal: Internal HDFS(Ephemeral Disk/Cinder) • External: Swift/HDFS Run Job in Transient Cluster *Different Plugin provide different capabilities
- 8. BIG DATA TECHNOLOGY Sahara Working Flow Fast Cluster Provisioning Select Hadoop Version Select Base Image w/ Hadoop Define Cluster Configuration Provision Cluster Operate Cluster Terminate Cluster Analytic as a Service using Elastic Data Processing Select Hadoop Version Configure Jobs Set Limit for Cluster Execute Jobs Get The Result • Choose type of the job: pig, hive, jar-file, etc. • Select input and output data location (Swift support) • Cluster will be removed automatically after the job completion • Provide the details Hadoop configuration, like size, topology, and others • Sahara will provision VMs, install and configure Hadoop • Support Scale out Cluster to add/remove nodes
- 9. BIG DATA TECHNOLOGY Sahara Data Processing Swift OpenStack Virtual Clusters OpenStack Virtual Clusters HDFS Collector Agent Data Stream Pattern 2: External - SwiftPattern 1: Internal - HDFS Only Collector Agent Collecting Data Collecting Data OpenStack use Swift as a data source to store input and output data. The benefit is to process the data directly and persist the data via Swift. OpenStack support to create HDFS on Cinder or Ephemeral Disk. This method can provide a better data processing performance via Ephemeral Disk or to persist the data via Cinder with lower performance. Cinder Ephemeral Disk MapReduce MapReduce
- 10. BIG DATA TECHNOLOGY Docker Background • An open source project • The latest version is v1.6 • Automates the deployment of applications inside software containers • Provide fast and application portability • Use libcontainer library to use virtualization facilities from Linux kernel • Resource isolation using cgroups, kernel namespaces, …etc
- 11. BIG DATA TECHNOLOGY Sahara + Docker • Deliver Better Performance (compare with hypervisors) • Optimize Resource Utilization • Reduce Cost • Fast Deployment
- 12. BIG DATA TECHNOLOGY Sahara Architecture Sahara RESTAPI Horizon Python Sahara Client Sahara Pages Keystone Auth DAL Image Registry Provisioning Engine Vendor Plugins EDP Hadoop VM Hadoop VM Hadoop VM Hadoop VM Nova|Heat|Cinder Glance Neutron
- 13. BIG DATA TECHNOLOGY Sahara + Docker Architecture Sahara RESTAPI Horizon Python Sahara Client Sahara Pages Keystone Auth DAL Image Registry Provisioning Engine Vendor Plugins EDP Hadoop VM Hadoop VM Hadoop VM Hadoop VM Nova|Heat|Cinder Glance nova docker driver Docker Registry Docker Image Docker Neutron
- 14. BIG DATA TECHNOLOGY Sahara CDH Plugin Controller Computing Node1 Cloudera Manager API Python Client (Migrate from CM-API Client) Sahara Service Horizon(OpenStack Dashboard) CDH Plugin Step1: Create VM via Heat by using Cluster Template. CM must be included in one master machine. Step2: Use CM API Client to connect to CM and provision the other services in the cluster. STEP1 STEP2 End Customer VM1 - Master VM2 - Slave Cloudera Manager (Cloudera Express v5.1.3, CDH v5.0.0 & CM API v7) Job History Resource Manager Oozie Server Name Node Secondary Name Node Data Node Node Manager STEP3 CDH Cluster
- 15. BIG DATA TECHNOLOGY Nova Docker Driver Introduced with Havana, move out Icehouse and Juno For Juno, • Must install an older version novadocker # git checkout -b pre-i18n 9045ca43b645e72751099491bf5f4f9e4bddbb91 • Implement a RESTFul client via httplib to communicate with Docker For Kilo(Upstream), • Need to install docker-py • Use Docker API Client to communicate with Docker
- 16. BIG DATA TECHNOLOGY Authenticate & Hostname Issue Use username & password instead of inject authorized key into instance • No cloud-init in docker image, use username & password instead of inject key Upgrade Docker version to support change hostname • Docker v1.2 or later can support to change hostname Change “sudo mv etc-host /etc/hosts” to “sudo cp etc-host /etc/hosts” • Docker v1.3 response the device is busy when using “mv”. By using “cp” to replace “mv” can be success to run the change
- 17. BIG DATA TECHNOLOGY Network Port Issue Open Privilege Mode to expose all the ports in the container • Modify nova docker driver source code to add “privileged=True” and publish all ports
- 18. BIG DATA TECHNOLOGY Docker Image Build a docker image by using Dockerfile • Refer sahara-image-elements to build a CDH5 docker image • Build a docker image may take a lot of time(try-and-error) • Better use Dockerfile cache to reduce the time building the image Copy docker image to every compute node manually • Must copy docker image to all the compute nodes, currently glance cannot support to copy the image to compute node • If the image cannot be found in docker images, nova will raise an error during starting an instance
- 19. BIG DATA TECHNOLOGY Build Docker Image - using Dockerfile Using docker build to build image by DockerFile # docker build -t $image_name:$tag From centos:centos6 MAINTAINER Weiting Chen weiting.chen@intel.com ENV http_proxy http://xxx:1080 … RUN echo 'proxy=http://xxx:1080' >> /etc/yum.conf RUN yum install -y cloudera-manager-agent … … EXPOSE 21 … Add ENV variables at beginning 1. Add proxy setting in individual software configuration 2. Install required software Expose Required Service Port Dockerfile Example
- 20. BIG DATA TECHNOLOGY Register & Copy Docker Image to Compute Nodes Register docker image to glance # docker save cdh5:20150425 | glance image-create --is-public=True -- container-format=docker --disk-format=raw --name cdh5:20150425 Copy image to all compute nodes # scp cdh5:20150425.tar $compute_node:./ Load image to docker registry # docker load -i cdh5:20150425.tar If no image can be used in computing node, it will raise an error from nova.
- 21. BIG DATA TECHNOLOGY Nova Docker Driver Network • Set network to “none” • Nova docker driver would leverage existing network configuration from Neutron • Support Linux Bridge or OVS • NOT use docker0 • Use VXLAN in our experiment • Create a bridge to OVS automatically • Set Privilege Mode to True for convenience • Need to set port mapping during docker run if not use privilege mode
- 22. BIG DATA TECHNOLOGY Docker Network Host1 Docker Container1 Container2 Container3 eth0 172.17.42.10 eth0 172.17.42.11 eth0 172.17.42.12 docker0 172.17.42.1 Host1 Docker Container1 Container2 Container3 eth0 192.168.0.1 eth1 10.10.10.1 docker0 172.17.42.1 Host1 Docker Container1 Container2 Container3 docker0 172.17.42.1 Bridge Mode Host Mode None Mode • Default Mode • Support multiple namespaces • Only one namespace • Nova Docker Driver use this • Configure network and connect to bridge via driver
- 23. BIG DATA TECHNOLOGY Docker Network Performance Host1 Host2 Host1 C1 941 Mb Host2 941 Mb Host1 C1 Host2900 Mb C1 Host1 C1 941 Mb Host1 C1 C2 14 Gb • Container to the same Host • Container to Container in the same Host• Container to Container in different Host • Container to different Host • Host to Host phy. network br-ex(floating ip) br-tun BACKGROUND • OpenStack Juno using VXLAN • Use Docker v1.3 • 1Gb Ethernet br-ex(floating ip) qbr~ 14Gb w/ DVR
- 24. BIG DATA TECHNOLOGY Neutron VXLAN without DVR Controller/Network Node Compute Node br-tun patch-tun br-int br-ex eth1 br-tun patch-tun br-int qvo~ 172.16.0.0/16 192.168.0.0/16 VM vm0 eth2 eth2 VM vm0 qbr~qvb~ patch-int tap~ 10.0.0.0/16 tap patch-int tap qdhcp ns~ snat- sg~ qg~ qrouter~ qr~ tap tap
- 25. BIG DATA TECHNOLOGY Neutron VXLAN with DVR Controller/Network Node Compute Node br-tun patch-tun br-int br-ex eth1 br-tun patch-tun br-int tap qvo~ 172.16.0.0/16 192.168.0.0/16 br-ex eth1 VM vm0 eth2 eth2 VM vm0 tap qbr~qvb~ patch-int tap~ fip- qrouter~ fpr~ rfp~ 10.0.0.0/16 fg~ qr~ tap patch-int tap qdhcp ns~ snat- sg~ qg~ qrouter~ qr~ tap tap
- 26. BIG DATA TECHNOLOGY Change MTU Size • Change MTU Size if you are using VXLAN • Impact: MTU size could impact the network performance. If the MTU size is not change, create instances still can work, but network performance is going down to 1MB. • Solution: Change MTU Size in VM #sudo ifconfig eth1 mtu 1400 up
- 27. BIG DATA TECHNOLOGY Container Disk Space • Default image disk space only use 10 GB • Impact: Default reserve 10GB space for HDFS configuration, there is no space to put data in HDFS • Solution: Assign parameters when starting Docker service # sudo ./docker -d --storage-opt dm.basesize=20G --storage-opt dm.loopdatasize=200G & *To enable the parameters must clean up /var/lib/docker/ and restart docker
- 28. BIG DATA TECHNOLOGY vCPU Numbers The number of vCPU is always 1. • Impact: vCPUs calculation may be fail. • Solution: In Juno, change the number in nova docker driver source code and set it equal to the number of physical cores.
- 29. BIG DATA TECHNOLOGY Docker in OpenStack Performance Network Performance Instance Boot/Cluster Provision Disk Performance using DD HiBench Testing
- 30. BIG DATA TECHNOLOGY Our Testing Environment CLUSTER CONFIGURATION Role Details Controller w/ Compute x 1 Controller, Network, Compute Compute x 5 Compute HARDWARE CONFIGURATION Items Details CPU Intel Xeon X5670 2.93Ghz Memory 64GB(1333Mhz 8GB x 8) Storage 1TB SATA HDD SOFTWARE CONFIGURATION Software Name Versions CentOS 7.0 Docker v1.6 OpenStack Juno
- 31. BIG DATA TECHNOLOGY Create an instance/Provision a cluster Assume image has been copied to all the computing node. Create an instance and check the log to capture the response time. Use Docker in 1sec Use KVM in 10sec Provision a cdh cluster still take a long time, this issue comes from Sahara CDH plugin.
- 32. BIG DATA TECHNOLOGY DD Test Docker Container use CentOS6.6 in Host with CentOS7. File System is XFS Use DD Command: dd if=/dev/zero of=test1 bs=1M count=8192 conv=fdatasync Host: 140~160MB/s Host w/ OpenStack: 100~130MB/s(Controller), 140~160MB/s(Compute) Container Result: 100~140MB/s Docker can provide almost closer disk IO performance with Bare Metal
- 33. BIG DATA TECHNOLOGY Conclusion • Docker can bring benefit to boot mass instances • Docker can provide good performance in Disk and Network with a little overhead • How to optimize resource utilization will be the focus
- 34. BIG DATA TECHNOLOGY Call-For-Action • Contribute more for Docker and OpenStack • Find the critical components for Big Data on Cloud and let it become better • Need more customer use cases for Sahara Contact: weiting.chen@intel.com
Editor's Notes
- #10: Support External HDFS, but needs to have some configurations manually
- #24: Container to the same Host can be better
- #26: DVR can enhance the performance in “Container to the same Host”, from 941Mb to 14Gb
- #33: Other machine testing result using SSD(Write Through): Host: 180MB/s 1.3GB/s VM(Ubuntu): 111MB/s 107MB/s