Shift-left SRE: Self-healing on OpenShift with Ansible
- 1. Shift-Left SRE - Self-healing on OpenShift with Ansible #RedHatOSD Jürgen Etzlstorfer, Technology Strategist @jetzlstorfer
- 2. A relentless pursuit of software perfection 1600 Employees 5000 Enterprise Customers 79 of the Global 100
- 3. confidential If you write applications, they will break eventually ~ Murphy‘s law
- 4. On average, a single transaction uses 82 different types of technology Browser Multi-geo Mobile Network Code Hosts Logs IoT 3rd parties Services Cloud SDN Containers Applications are getting more complex!
- 5. confidential What if you had something similar to a self-healing robot?
- 7. Building auto-remediation into your pipeline #RedHatOSD ✓ ✓ ✓ ✓ commit build test stage prod ✓
- 8. Building auto-remediation into your pipeline #RedHatOSD ✓ ✓ ✓ ✓ ✗ commit build test stage prod
- 9. Building auto-remediation into your pipeline #RedHatOSD ✓ ✓ ✗ commit build test stage prod Break the pipeline early
- 10. Building auto-remediation into your pipeline #RedHatOSD ✓ ✓ commit build test stage prod Include remediation-as-code …
- 11. Building auto-remediation into your pipeline #RedHatOSD ✓ ✓ commit build test stage prod Include remediation-as-code ✓ … … … … … ✓ ✓
- 12. OpenShift Container Platform Dynatrace Software Intelligence Ansible Automation Self-healing applications
- 13. Automation via APIs ▪ Automation/Execution: perform mitigation/remediation actions ▪ Access to all systems ▪ Monitoring: know what’s going on in your applications ▪ End-to-end ▪ Full-stack – fully integrated in production Self-healing building blocks #RedHatOSD
- 15. Self-healing with Ansible Tower and Dynatrace #RedHatOSD ● APIs are key to enable automation ● Ansible Tower provides rich API for managing Ansible jobs ● Playbooks can be orchestrated in workflows and job templates
- 16. How to enable auto-remediation Full-stack environment is monitored Anomalies are detected automatically Root cause analysis is performed Problem notification is sent Event is received Job is triggered Playbook is executed Problem is remediated #RedHatOSD
- 17. How to enable auto-remediation Full-stack environment is monitored Anomalies are detected automatically Root cause analysis is performed Problem notification is sent Event is received Job is triggered Playbook is executed Problem is remediated #RedHatOSD
- 18. Ansible Tower integration in Dynatrace #RedHatOSD
- 19. Ansible Tower integration in Dynatrace #RedHatOSD
- 20. Ansible Tower integration in Dynatrace #RedHatOSD
- 21. confidential What we will see in the demo ▪ TicketMonster application running on OpenShift ▪ Full-stack, end-to-end monitoring by Dynatrace ▪ Feature release via Ansible Tower ▪ Auto-remediation as code (Ansible playbooks) #RedHatOSD
- 22. DEMO TIME :)
- 23. confidential What we have seen in the demo – short recap Release of a new feature Dynatrace detects increase of failure rate Dynatrace fires a problem notification to Ansible Tower Ansible Tower kicks off a playbook Check for latest deployment with remediation scripts Remediation script is executed Problem is remediated #RedHatOSD
- 24. Self-healing in an enterprise environment Auto Mitigate! 1 CPU Exhausted? Add a new service instance! Escalate at 2AM? 5 1 2 3 4
- 25. Self-healing in an enterprise environment Auto Mitigate! 1 CPU Exhausted? Add a new service instance! Escalate at 2AM? 2 High Garbage Collection? Adjust/Revert Memory Settings! 5 1 2 3 4
- 26. Self-healing in an enterprise environment Auto Mitigate! 1 CPU Exhausted? Add a new service instance! 3 Issue with BLUE only? Switch back to GREEN! Escalate at 2AM? 2 High Garbage Collection? Adjust/Revert Memory Settings! 5 1 2 3 4
- 27. Self-healing in an enterprise environment Auto Mitigate! 1 CPU Exhausted? Add a new service instance! 3 Issue with BLUE only? Switch back to GREEN! Escalate at 2AM? 2 High Garbage Collection? Adjust/Revert Memory Settings! 4 Hung threads? Restart Service! 5 1 2 3 4
- 28. Self-healing in an enterprise environment Auto Mitigate! 1 CPU Exhausted? Add a new service instance! 3 Issue with BLUE only? Switch back to GREEN! Escalate at 2AM? 2 High Garbage Collection? Adjust/Revert Memory Settings! 4 Hung threads? Restart Service! 5 1 2 3 4 Update Dev TicketsImpact Mitigated??
- 29. Self-healing in an enterprise environment Auto Mitigate! 1 CPU Exhausted? Add a new service instance! 3 Issue with BLUE only? Switch back to GREEN! Escalate at 2AM? 2 High Garbage Collection? Adjust/Revert Memory Settings! 4 Hung threads? Restart Service! 5 Still ongoing? Initiate Rollback! 5 1 2 3 4 Mark Bad Commits Update Dev TicketsImpact Mitigated??
- 30. Self-healing in an enterprise environment Auto Mitigate! 1 CPU Exhausted? Add a new service instance! 3 Issue with BLUE only? Switch back to GREEN! Escalate at 2AM? 2 High Garbage Collection? Adjust/Revert Memory Settings! 4 Hung threads? Restart Service! 5 Still ongoing? Initiate Rollback! Escalate? Still ongoing? 5 1 2 3 4 Mark Bad Commits Update Dev TicketsImpact Mitigated??
- 31. confidential Embed auto-remediation in your CI/CD pipeline
- 32. confidential Embed auto-remediation in your CI/CD pipeline Path to NoOps: Self-Healing, … Shift-Right: Tags, Deploys, Events
- 33. confidential Embed auto-remediation in your CI/CD pipeline Shift-Left: Break Pipeline Earlier Path to NoOps: Self-Healing, … Shift-Right: Tags, Deploys, Events Actionable Feedback Loops
- 35. OpenShift Dynatrace♥ “Beyond years of industry knowledge in the APM space, Dynatrace offers one of the best solutions I’ve seen for monitoring applications running on OpenShift. What really distinguishes them from others is the use of artificial intelligence based root-cause analysis. OpenShift is a platform to allow you to run decoupled services and applications, which can be a monitoring nightmare, but Dynatrace’s insights makes it less scary.” Chris Morgan, Technical Director – Red Hat OpenShift Ecosystem
- 36. GRAZIE PER L’ATTENZIONE Jürgen Etzlstorfer, Technology Strategist #RedHatOSD