Data Agility for Devops - OSI 2018
Download as PPTX, PDF2 likes248 views
Mayadata specializes in cloud native storage and data agility for DevOps, particularly through their open-source projects OpenEBS and Litmus. The document discusses the challenges and solutions related to data agility in cloud-native environments, such as managing stateful applications, ensuring data resilience, and implementing effective backup and restore strategies. Mayadata emphasizes the benefits of container-attached storage and offers insights into DevOps workflows that leverage these technologies.
Data Agility for Devops - OSI 2018
- 1. Cloud Native Storage & Data Agility for DevOps @uma_mukkara 12 October 2018
- 2. ● MayaData delivers Data Agility ● Sponsor of OpenEBS and Litmus projects ● Large Bangalore based OSS contributor ● https://mayaonline.io is SaaS software used to manage stateful workloads on Kubernetes Uma Mukkara Co-founder & COO
- 3. ● Popular open source storage project for stateful applications on Kubernetes ● Team of 50+ hackers with storage and DevOps experience ● Leading project in CAS category ● 250+ contributors ● 5K+ github stars ● 1M+ docker pulls slack.openebs.io
- 4. DevOps Micro- services Agility Cloud Native technologies and practices Docker Kubernetes Istio OpenEBS HelmPrometheus Adopt Enable Achieve
- 6. Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach. These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil. Definition from What is “Cloud Native” ?
- 7. Data Agility Challenges in Cloud Native Space
- 8. 1. HA within a cluster • Microservices architecture separates every component. A legacy application now becomes • A stateless application and/or • A stateful application with a database in it • Stateless applications are mobile across nodes in a cluster but Stateful applications need persistent data for easy movement Synchronous replication of cloud native data
- 9. 2. HA across clusters • multi-cloud is now a reality • Multi Cloud example • Code is in Github • Dev uses On-Prem • CI/CD in Google • Pre-Prod in Azure • Production in AWS Move data across clouds
- 10. 3. CI Pipelines • State of the data moves to CI pipelines • Developer needs access to the exact state of the data when a pipeline fails • Pipelines have to snapshot the data • Pipelines have to clone the data and automate the access to the developer Data snapshots in CI pipelines
- 11. 4. E2E Tests • For gaining credibility to push the stateful application into production, building an E2E test suite is a must • The stateful application that is containerized need to be tested for all state transitions and changes in the infrastructure • E2E matrix for stateful applications is large and complex App specific data resilience and availability
- 12. 5. Data Backup and Restore • The traditional Backup and Restore systems do NOT work in Cloud Native eco-systems • Irrespective of the reliable infrastructure designs that Cloud Native systems provide, DevOps still need to have Backup and Restore strategy • Converting the application into microservices architecture is the one challenge but finding and integrating/automating the Backup/Restore process is another Backup and DR for Cloud Native Data
- 13. Choose Cloud Native technologies to resolve Data Agility challenges
- 14. Cloud Native Tech for Data Agility * These are applicable when you are using Kubernetes as Orchestration platform Synchronous replication of cloud native data Move data across clusters, clouds Data snapshots in CI pipelines App specific data resilience and availability Backup and DR for Cloud Native Data Data Agility Challenge Choose any CAS like OpenEBS Choose any CAS like OpenEBS OpenEBS & Litmus Litmus Heptio ARK with OpenEBS Plugin Cloud Native Technology
- 15. Cloud Native Tech for Data Agility
- 16. What if storage for container native applications was itself container native? https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/ Blog White Paper https://openebs.io/assets/docs/WP-OpenEBS-0_7.pdf Container Attached Storage or CAS - OpenEBS
- 17. Container Attached Storage or CAS - OpenEBS Architecture
- 18. Container Attached Storage or CAS - OpenEBS
- 19. Chaos Engineering Framework - Litmus Litmus- Chaos Engineering Framework for Stateful applications on Kubernetes It is a K8S JOB K8S deployers Provider specific deployers Test cases (ansible playbooks)Framework contains Standard & customized Workload containers Tools to help execute tests (Chaos, logging etc)
- 20. Litmus - For CI Pipelines / E2E Chaos Engineering Framework - Litmus
- 21. Backup and Restore framework - Heptio ARK+OpenEBS Heptio ARK OpenEBS Plugin + Cloud Native incremental data backup
- 22. Backup and Restore framework - Heptio ARK+OpenEBS
- 24. DevOps Application agility Data agility Cloud Native tools and best practices (sufficient expertise is already in the public domain) Use CAS for data storage and data resiliency Build Snapshot/aware CI pipelines Build data aware E2E suites Choose Cloud Native Backup and DR tools
- 25. Data Agility Example DevOps Developer DevOps Architect Dev Cluster CI/CD Cluster Production Cluster 1.Develop Application YAML Volume: Name : Volume 1 Mount : /mnt/volume PVC : OpenEBS-PVC Persistent volumes are provisioned automatically for developer. Code is merged2. Code is merged 3. CI/CD Pipeline Build Stage Setup Stage Litmus CD 4. Deploy
- 26. Data Agility Example DevOps Developer DevOps Architect Dev Cluster CI/CD Cluster Production Cluster 1.Develop 2. Code is merged 3. CI/CD Pipeline Build Stage Setup Stage Litmus One test fails Snapshot Clone Access clone and debug
- 27. Data Agility Example DevOps Developer DevOps Architect Dev Cluster CI/CD Cluster Production Cluster 1.Develop 2. Code is merged 3. CI/CD Pipeline Build Stage Setup Stage Litmus CD 4. Deploy 5. Chaos Engineering using Litmus
- 28. Data Agility Example DevOps Developer DevOps Architect Dev Cluster CI/CD Cluster Production Cluster 1.Develop 2. Code is merged 3. CI/CD Pipeline Build Stage Setup Stage Litmus CD 4. Deploy 6. Backup Heptio Ark + OpenEBS
- 29. Data Agility . Delivered
- 30. QUESTIONS?
- 31. We are Hiring DevOps Engineers jobs@mayadata.io