Containerization: The DevOps Revolution
3 likes651 views
The document discusses the significance of containers in the context of DevOps and cloud environments, comparing them to virtual machines and highlighting their advantages such as rapid innovation, resource efficiency, and isolation. It explains how technologies like Mesos and Docker can be used to optimize resource utilization, facilitate service deployment, and manage multiple clusters effectively. Additionally, the document addresses potential challenges related to container networking and performance.
Containerization: The DevOps Revolution
- 1. AWS AND SOFTSERVE CONTAINERIZATION: THE DEVOPS REVOLUTION Tuesday, April 12, 2016 8 AM PST/4 PM BST/5 PM CEST webinar webinar@softserveinc.com
- 2. SPEAKERS Valentyn Kropov Solutions Architect, SoftServe Craig Williams Solutions Architect, Amazon Web Services webinar
- 3. WHY CONTAINERS ARE SUCH A BIG DEAL webinar webinar@softserveinc.com
- 4. • Own Process Space • Can run commands as root • Can install ad-hoc packages • Can run Services • You can get a shell on it (e.g. through SSH) CONTAINER IS LIKE A LIGHTWEIGHT VM: webinar
- 5. • It uses hosts kernel • Can’t boot different OS • Can’t have it’s own kernel modules • From host OS perspective Containers are the bunch of processes DIFFERENCES TO VM: webinar
- 7. • Resource Metering and Limiting • CPU and CPUSet • Memory • Network • Block I/O • /dev/* CGROUPS (CONTROL GROUPS): webinar
- 8. • Provides containers with their own view of the system • Limits what you can see (and use) • Multiple namespaces: pid, net, mnt, uts, ipc, user NAMESPACE: webinar
- 9. • Create new container instantly instead of copying whole system • Storage keeps tracking of what has change (AUFS, ZFS, etc) • Reduces footprint and overhead • Decreases boot time COPY-ON-WRITE STORAGE: webinar
- 10. • LXC • systemd-nspawn • Docker Engine • rkt/runC • OpenVZ • Jails (FreeBSD), Zones (Solaris) CONTAINER RUNTIMES: webinar
- 11. • Uses the same kernel features => Performance will be the same • What matters is: Design Features Ecosystem (e.g. 100.000+ apps in Docker Hub) Support WHAT’S THE DIFFERENCE BETWEEN THEM? webinar
- 12. THE STORY OF SUCCESS webinar webinar@softserveinc.com
- 13. PROBLEM & OPPORTUNITY • Rapid innovation in computing and application development services • No single service is optimal for all solutions • Customers want to run multiple services in a single cluster and run multiple clusters in Intercloud environment ...to maximize utilization ...to share data between services webinar
- 14. DATACENTER AND SOLUTION TODAY VM7 VM8 VM4 VM5 VM6 VM1 VM2 VM3 VM1 VM2 VM2 Visualization Service Data Ingestion Service Analytics Service • Configuration and management of 3 separate clusters • Resources stay idle if service is not active • Need to move data between clusters for each service webinar
- 15. WHAT DO WE WANT TO DO? Data Ingestion Service Analytics Service Visualization Service ….to maximize utilization ...to share data between services Shared cluster Multiple clusters webinar
- 16. SHARED CLUSTER AWS VM1 VM2 VM3 VM4 VM5 webinar
- 17. WHAT IS IN IT FOR CUSTOMERS? Maximize utilization Deliver more services with smaller footprint Shared clusters for all services Easier deployment and management with unified service platform Shared data between services Faster and more competitive services and solutions webinar
- 18. HOW DOES THIS WORK? Mesos Master Mesos Slave Spark Task Executor Mesos Executor Task #1 Task #2 ./python XYZ Mesos Slave Docker Executor Docker Executor java -jar XYZ.jar ./xyz Mesos Master Mesos Master Spark Service Scheduler Marathon Service Scheduler Zookeeper quorum webinar
- 19. HOW DOES THIS WORK? Mesos provides fine grained resource isolation webinar Mesos Slave Process Spark Task Executor Mesos Executor Task #1 Task #2 ./python XYZ COMPUTE NODE Executor Container (cgroups)
- 20. HOW DOES THIS WORK? Mesos provides scalability Mesos Slave Process ./python XYZ COMPUTE NODE Python executor finished, more available resources more Spark Container (cgroups) webinar Spark Task Executor Task #1 Task #2 Task #3 Task #4
- 21. Mesos has no single point of failure Services keep running if VM fails! webinar HOW DOES THIS WORK? Mesos MasterMesos Master Mesos Master VM1 VM2 VM3 VM4 VM5 !
- 22. webinar HOW DOES THIS WORK? Mesos MasterMesos Master Mesos Master VM1 VM2 VM3 VM4 VM5 ! Master node can failover Services keep running if Mesos Master fails!
- 23. HOW DOES THIS WORK? Mesos Slave Process ./python XYZ COMPUTE NODE Python executor finished, more available resources more Spark Container (cgroups) webinar Spark Task Executor Task #1 Task #2 Task #3 Task #4 ! Slave process can failover Tasks keep running if Mesos Slave Process fails!
- 24. Can deploy in many environments Get orchestrated by Openstack, Ansible (scripts), Cloudbreak True Hybrid Cloud deployment: AWS, CIS, UCS, vSphere, other AWS VM5VM1 VM2 VM3 VM4 Terraform REST API (policy, auto-scaling) REST API (direct provisioning) Scripted provisioning webinar HOW DOES THIS WORK?
- 25. CONTAINERS: Service Product Cloud/Virtualization AWS/CIS/vSphere/Metacloud/UCS… Provisioning Terraform Automation Ansible Clustering & Resource Management Mesos, Marathon, Docker Load Balancing Avi Networks ETL & Data Shaping StreamSets Log Data Gathering Logstash Metrics Gathering CollectD, Avi Networks Messaging Kafka, Solace Data Storing (Batch) HDFS Data Storing (OLTP/Real-time) Cassandra Data Storing (Indexing) Elastic search Data Processing Apache Spark Visualization Zoomdata *Subset example webinar
- 26. ISSUES • Service Discovering • Networking for Containers • Persistent Storage • Docker Performance webinar