OSMC 2017 | Monitoring Challenges in a World of Automation by Anthony Goddard
0 likes123 views
The document discusses the complexities of monitoring within automated cloud environments, emphasizing the need for cloud-native monitoring solutions. It outlines three key requirements: automated discovery, automated monitoring, and automated decommissioning, while highlighting common anti-patterns that hinder effective monitoring. The session aims to address these challenges and demonstrate the importance of adapting monitoring practices to suit increasingly ephemeral infrastructures.
OSMC 2017 | Monitoring Challenges in a World of Automation by Anthony Goddard
- 1. Monitoring Challenges in a World of Automation Monitoring is hard enough on its own. Automation makes it harder.
- 2. Anthony Goddard VP Operations Sensu, Inc. @anthonygoddard // @sensu
- 3. ● Open core monitoring framework, released in 2011 ● Enterprise offering launched in 2015 ● Sensu Inc formed in January 2017 ● 20 employees & growing! About Sensu
- 4. What is Sensu? ● An open source, cloud native monitoring framework ● The monitoring router ● Infrastructure, service, and application monitoring ● Designed for automation ● Cross platform (linux, Windows, BSD, AIX, Solaris, MacOS, etc) ● Learn more: https://sensuapp.org
- 5. Mission Statement Obviate the need to (re)build custom monitoring solutions.
- 6. This isn't a talk about Sensu.
- 7. Purpose of this talk ● Discuss challenges of monitoring ephemeral systems ● Review basic cloud native monitoring requirements ○ Automated discovery ○ Automated monitoring ○ Automated decommissioning ● Talk about cloud native monitoring anti-patterns ● Live demo! (what could possibly go wrong?)
- 9. Cloud computing has changed the world.
- 10. Which came first? Cloud computing or DevOps?
- 11. Problem Statement ● Cloud platforms and automation systems cause changes in infrastructure that increase the complexity of monitoring ● New systems/endpoints must be discovered and monitored automatically ● Monitoring must now distinguish the subtle differences between "down" and "decommissioned"
- 12. Expectations Our infrastructure is becoming increasingly more automated and ephemeral. Shouldn't we expect similar capabilities from our monitoring?
- 13. Cloud Native Monitoring Requirements Overview 1. Automated discovery 2. Automated monitoring 3. Automated decommissioning
- 15. New systems should be automatically discovered. Cloud Native Monitoring Requirements
- 16. Cloud concepts ● Provisioning events create and replace instances ● Cloud providers automate replication of instances (e.g. auto-scaling groups, etc) ● APIs allow external systems to invoke provisioning events Automated Discovery
- 17. Automated Discovery Cloud monitoring anti-patterns ● Polling-based discovery (regardless of protocol) ● Discovery that precludes complex network topologies ● Punching holes in firewalls (ingress traffic)
- 18. Polling is not a reliable discovery solution.
- 19. Automated Discovery Cloud-native monitoring requirements ● New systems must be discovered in realtime ● Provide push-based or event-based discovery + discovery APIs
- 21. New systems should be monitored automatically. Cloud Native Monitoring Requirements
- 22. Automated Monitoring Cloud concepts ● Almost all infrastructures are distributed systems ● Disparate systems fulfill unique roles (e.g. db, web service) ● Simple architectures = one or more roles per system ● Complex architectures = one role per system
- 23. Automated Monitoring Cloud monitoring anti-patterns ● Monitoring configuration mapped to individual systems ● Monitoring via remote access (e.g. SSH, WinRM, NRPE)
- 24. Nope.
- 25. Automated Monitoring Cloud-native monitoring requirements ● Monitoring configuration should be mapped to roles ● Monitoring should begin the moment systems come online
- 26. Automated monitoring should "just work"
- 28. Terminated systems should be automatically removed from monitoring. Cloud Native Monitoring Requirements
- 29. Automated Decommissioning Cloud Concepts ● Utility computing incentivizes cost savings ● Decommission systems when not in use, or during reduced load ● Intentional actions look very similar to failure scenarios
- 30. Automated Decommissioning Cloud monitoring anti-patterns ● Making assumptions about the lack of monitoring data ● Making assumptions about the loss of network connectivity ● Using a monitoring system as a source of absolute truth
- 31. Cloud-native monitoring requirements ● Should be invoked by the terminated system (i.e. stop signal) ● May be triggered by the provisioning system (i.e. via APIs) ● Optionally verified via external source(s) of truth (as needed) ● Must be the most reliable function of the monitoring system Automated Decommissioning
- 32. When you can no longer trust your monitoring alerts.
- 33. Demo! But first, some questions…
- 34. Public/Private Cloud (IaaS) Who knows what "the cloud" is? Who understands basic cloud computing concepts like ASGs and ELBs? Who is currently using a IaaS provider like AWS, GCP, Azure, or OpenStack? Kubernetes Who knows what Kubernetes is? Who has Kubernetes on their roadmap? Who is currently using Kubernetes? Audience participation time!
- 35. (DEMO)
- 38. QUESTIONS?
- 39. Conclusion ● Cloud computing introduces challenges that demand cloud-native monitoring solutions. ● Monitoring solutions must automatically discover new systems. ● Monitoring configuration should be applied automatically. ● Monitoring should comprehend "down" vs "decommissioned".
- 40. Thank You