Sergey Dzyuban "To Build My Own Cloud with Blackjack…"
Download as PPTX, PDF1 like384 views
The document outlines the implementation and management of a cloud infrastructure using DC/OS, focusing on deploying microservices and orchestration. It discusses the challenges encountered in maintaining the infrastructure manually and the transition towards a more automated and scalable solution that includes CI/CD practices via Jenkins. Additionally, it highlights resource management, health checks, and flexible service deployment to enhance operational efficiency.
Sergey Dzyuban "To Build My Own Cloud with Blackjack…"
- 1. To Build My Own Cloud with B Sergey Dzyuban SBTech
- 2. LAMP Windowsadmin 2003 2007 2010 2014 2016 Web Developer .NET Developer Technical Account Manager Team Lead DevOps Tech Lead Sergey Dzyuban DevOps Tech Lead at SBTech
- 3. R&D offices Datacenters 1000+ 2500 700 200 (1K+ instances) 300 CPU/3Tb RAM 28 nodes 2 Tb 100K+ 3
- 4. Windows and Data Centers
- 5. Platform as a Service Unit – place in SBTech geography
- 6. Infrastructure Team efforts Platform as a Service Team
- 7. How resource request works DEV Dev Team asked to create separated Kafka instance 7
- 8. How resource request works DEV IT Dev Team asked to create separated Kafka instance • request IT to provide Linux VM • configure access 8
- 9. How resource request works DEV IT Infra Team • request Infrastructure Team to setup Kafka • add monitoring • add healthcheck 9
- 10. How resource request works DEV IT Infra Team • Configure monitoring • Configure access to the requester 10
- 11. Maintain infrastructure manually - issues • Number of requests are increasing • Services and machines after some time go offline, die, appear in broken state. • Some services required continuous maintenance DEV Infra Team DEV DEV DEV DEV DEV DEV 11
- 12. PaaS for Microservices Building hosting platform for microservices orchestration
- 13. Who are You – Mr. Microservice? Hashi Corp Vault is good enough for securing configuration settings, but also requires some initial infrastructure setup. Mr. Microservice Consul Fabio-lb logs metrics trace cache state eventsdiscovery config 21
- 14. Who are You – Mr. Microservice? But sometimes it’s something specific even for experienced DevOps engineers. Mr. Microservice Consul Fabio-lb logs metrics trace cache state eventsdiscovery config 22
- 15. Who are You – Mr. Microservice? But sometimes it’s something specific even for experienced DevOps engineers. Mr. Microservice Consul Fabio-lb logs metrics trace cache state eventsdiscovery config 23
- 16. So what is Dev ENV ? EXPECTATION REALITY
- 17. Maintain infrastructure manually - IaaC • Deployment of each component requires unique deployment activities and scripts. • Do not resolve problem with health and status checks Infra Team 25
- 18. Maintain infrastructure manually - IaaC • Deployment of each component requires unique deployment activities and scripts. • Do not resolve problem with health and status checks Infra Team 26
- 19. Maintain infrastructure manually - Docker • Using Docker allows to standardize deployment. • Docker Image for each of components needs to be prepared and preconfigured for our needs. Infra Team 27
- 20. Maintain infrastructure manually - Docker … lots of SSH connections Infra Team 29
- 21. Maintain infrastructure manually Manual resource assignment has a lot of complexity and require some operation flow to be proceeded. 30
- 22. Maintain infrastructure manually - Cloud • Cloud Engineers have a lot of good examples how component deployment automation should looks like. • Next step was to provide simple user experience to maintain this process just simply stupid. 31
- 23. Cloud Management Experience • Browser based • Clear configuration and deployment • Simple scaling • Build-in monitoring • Services catalog • Self documented 32
- 24. It’s all about Jenkins The short story how we started to use DC/OS
- 25. Jenkins for CI/CD • ~ 700 repositories • ~ 500 builds per day • ~ 1 build per minute • ~ 40 windows slaves 35
- 26. Jenkins for CI/CD It requires some additional interfaces to provide and process information for developers and build engineers. 36
- 27. Jenkins for CI/CD • Elastic + Kibana • Go APIs • Angular SPAs • MySQL and Redis • Zabbix for Jenkins slaves monitoring • Sonar 37
- 28. Jenkins for CI/CD And here is a moment, You realize that having many pets in your datacenter may be not a good idea. 38
- 29. Jenkins for CI/CD The good news – Simpsons already did it. 39
- 30. Jenkins for CI/CD Requirements to infrastructure management: • Works on-premise • Needs to be distributed • Supports health-checks • Self-Healing • Easy deploy and maintain • Can scale • Has persistence storage support • User friendly 40
- 31. DC/OS overview Short story about evolution from Mesos to DC/OS
- 32. DC/OS Overview • Physical master nodes cluster • Any number of worker nodes Masters Private Node Private Node Public Node PrivateZone DMZ 42
- 33. PrivateZone DMZ Master-Slave Architecture Master Node: • Zookeeper • Marathon • Mesos-master Worker Node • Mesos-slave Zookeeper Mesos Marathon 43
- 35. DCOS Components
- 36. DMZ Master-Slave Architecture To provide Software as a Service, better way to share full access to the services. Zookeeper Mesos Marathon 47
- 37. Master-Slave Architecture Mesos resource management allows to share resources across different components automatically. Zookeeper Mesos Marathon DMZ 48
- 38. Resource Management in Mesos When new deployment starts, task will be started on a random Agent Node with free resource available. In case of lack of resources, deployment will be put on Pending state. workerA 8 Gb4 10 Gb workerB CPU MEM 4 Gb HDD 2 10 Gb 2 CPU 6 GB RAM 5 Gb HDD 49
- 39. Resource Management in Mesos The killing feature of DC/OS is a better resources utilization. Health check, deployment, distribution and resources management will be proceed on DC/OS (Mesos) side. workerA 8 Gb4 10 Gb workerB CPU MEM 4 Gb HDD 2 10 Gb 1 CPU 1 GB RAM 0 Gb HDD 2 x 50
- 40. Resource Management in Mesos DC/OS takes care about Agent Nodes health check and each running service instance health check. In case of unhealthy state, service or all services in broken node will be redeployed to other nodes. workerA 8 Gb4 10 Gb workerB CPU MEM 4 Gb HDD 2 10 Gb x 2 51
- 41. DEMO – DC/OS cluster in AWS
- 42. Hiding Pets behind the Cattles Managing hardware for Docker orchestration private cloud
- 43. Install Deploy Configure Observe 55
- 44. How we started • Vagrant + Virtual Box • Mini PC • 8 TVs with Mini PC • Ubuntu 16.04 • Daily usage – for Scrum and Monitoring Dashboards • 8 * 2 CPU = 16 CPU • 8 * 4 Gb RAM = 32 GB RAM • PC • VMWare • Google Cloud 63
- 45. Install Deploy Configure Observe 64
- 46. DC/OS Initial Setup We started from the simple one-master node configuration and one slave node, dedicated for services deployments. Elastic was the first try – it aggregates a lot of logs, and goes broken from time to time. Master cluster DevOps 4 8 Gb 65
- 47. DC/OS Initial Setup – first customer We need some service to be running – we request machine for this service in VMWare and add it as DC/OS Agent. At backstage – all VMs became a DC/OS nodes. Master cluster DevOps 4 8 Gb 66
- 48. DC/OS Initial Setup – TV boxes The best start was to use TV boxes for scrum meetings in all office rooms. It gives lot of free resources just for fun. 10 * 2 CPU = 20 CPU 10 * 4Gb = 40 Gb Master cluster 24 48 Gb 67
- 49. DC/OS Initial Setup – internal services Such setup allow us to run all services required for internal needs of infrastructure teams. .. and a little more like bots for Slack, Sonar, etc. 24 48 Gb Slack Bot 68
- 50. DC/OS Initial Setup – issues 24 48 Gb The main issues on this stage were: • Master Node performance Master nodes had lack of resources, which causes often DC/OS UI failures or Marathon failures. Temporary solution – master node restart. • Agent Nodes failures Out of free disk space, machine shutdown, CPU high load, out of memory – the most common reason of failures. 69
- 51. DC/OS Initial Setup – Cluster With number of Agents greater than 1 single master became the gap: • In case of failure – system goes down • Lack of performance Master cluster Master Master 24 48 Gb 70
- 52. DC/OS Initial Setup – Cluster With number of Agents greater than 1 single master became the gap: • In case of failure – system goes down • Lack of performance Master cluster Master Master 24 48 Gb 71
- 53. DC/OS Initial Setup – Cluster Master cluster Master Master 24 48 Gb With number of Agents greater than 1 single master became the gap: • In case of failure – system goes down • Lack of performance
- 54. DC/OS Initial Setup – more VMs With current setup, DC/OS became ready for consuming external requests. Master cluster Master Master Dev 40 60 Gb 73
- 55. DC/OS Initial Setup – Hardware PCs Few dedicated PCs became a cluster member in worker Agent role. Master cluster Master Master DevTeam 60 90 Gb 74
- 56. DC/OS Initial Setup – Google Cloud Creating scaling group in the cloud makes a DC/OS cluster unlimited by resources. Master cluster Master Master Unit 100 160 Gb 75
- 57. Summary • Number of nodes was increased eventually • Infra Team used DCOS for own needs only for the first time • To monitor and bootstrap the cluster some additional resources were required: Zabbix, Grafana • Different type of nodes allow to increase flexibility and give positive grow speed. • Adding Google Cloud instance eliminated cluster size limit. With hybrid cloud DC/OS can grow much more quickly. 76
- 58. Install Deploy Configure Observe 77
- 59. Add services quickly Service catalog allows to chose service from a predefined list and deploy in one click. If needed – own repository can be added. 78
- 60. Add services flexible For all the other cases – Service manual deployment are available 1. Single Container (Docker) 2. Bash runtime 3. Multi-container 79
- 61. Be structural Services can be organized as folder structure. This feature allows to isolate environments for different dev teams. 80
- 62. Be discoverable Mesos DNS, integrated with company DNS server, allows to access each service directly by Agent IP/port.
- 63. Summary • DC/OS allows to build complex DEV/UAT environments bases on Docker infrastructure • The simplest way of deployment – Universe Catalog with well known services deployed in one click. • Each service can be placed to a separated folder. • Mesos DNS includes full folder structure in service DNS name. • Marathon-LB allows to proxy any external call thought HA Proxy to target service instance (transforming IP and Port) 92
- 64. Is there life after delivery? Sure, service life only begins here …
- 65. Install Deploy Configure Observe 94
- 67. Cluster breath Nodes health uncertainty Service mobility 96
- 73. What is DC/OS? Distributed System Cluster Manager Container Platform Operating System Service Catalog Network 102
- 75. References • DC/OS official page: https://dcos.io/ • DC/OS Documentation: https://docs.mesosphere.com/1.11/overview/ • Marathon GitHub: https://github.com/mesosphere/marathon • Mesos: http://mesos.apache.org/documentation/latest/ • Zookeeper: https://zookeeper.apache.org/ • Exhibitor: https://github.com/soabase/exhibitor/wiki
- 76. Thank You Q&A