The IBM dashboard for operational metrics
Download as PPTX, PDF1 like6,663 views
The document summarizes IBM's use of a metrics dashboard to monitor their Cloud Foundry deployments. It discusses: 1. The key metrics tracked by the dashboard like DEA and app health, resource usage, and growth trends. 2. How the dashboard provides views of component health, resource usage, growth trends, access to logs, and email notifications to help find and fix problems and see patterns. 3. Ideas to find better data in their next generation Cloud Foundry deployment like restoring user and app linkages, detecting errors traceable to users, and automating responses based on metrics to make systems more self-healing and self-scaling.
The IBM dashboard for operational metrics
- 1. 1
- 2. 2 Daniel Krook Senior Certified IT Specialist, IBM The IBM dashboard for operational metrics
- 3. 3 We run Cloud Foundry on dozens of OpenStack VMs Two intranet clusters In the past year, we’ve learned how to Classic: 38 huge VMs deployed with Chef: 1,302 users, 1,710 apps NG: 41 medium VMs deployed with BOSH: 123 users, 247 apps Not counting Dev deployments All on 50+ Nova Compute nodes • Keep Cloud Foundry running smoothly • Discover and prevent impending problems • Resolve unexpected issues quickly
- 4. 4 1. Show the key data points we track 2. Show how our metrics dashboard helps us monitor that data 3. Share ideas on how to find better data in NG and beyond 4. Spark discussion on improved visibility for CF admins and customers. Goals of this lightning talk We are looking to get better at this, and help the community get better as well.
- 5. 5 1. The key data
- 6. 6 What are the important metrics? Data that can be tracked over time to see trends and behaviors Data that can help us predict problems before they happen DEAs and apps health  Memory reserved as a proportion of the memory available General health of all components  Health of the virtual machines  Status of the processes running on them Database nodes and services  Number of provisioned services against capacity available At the PaaS layer, that means:
- 7. 7  Deliver continuous availability in the cloud  Proactively solve problems rather than react to them  Understand the behavior of the system to automate it Why do we need metrics?
- 8. 8  NATS message bus • Discover the components to interrogate • Best for dynamically changing data Where can we find them?  Cloud Controller database (CCDB) • Longer lived data that isn’t in the varz endpoints
- 9. 9 2. Monitoring that data
- 10. 10 1. Views of component health 2. Resource usage details 3. Ongoing growth trends 4. Access to logs and raw varz 5. Email notifications Our metrics dashboard provides…
- 11. 11  Components nearing capacity or failure  Already failed components  Out of control apps and noisy users  Active/inactive users and apps  Growth trends and runtime/service adoption It helps us find (and fix) problems It helps us see patterns
- 12. 12 User and app trends There is also one unauthenticated page for high level stats
- 13. 13 DEA list
- 14. 14 DEA details
- 17. 17 User list
- 18. 18 User details
- 19. 19 App list
- 20. 20 App details
- 21. 21 Log list
- 22. 22 Log details
- 24. 24 3. Finding and acting on better data
- 25. 25  NG provides granular user/org/space views… • This enables better BSS potential in terms of QoS and departmental billing  …But we lost user and app data linkages from the health manager • Can’t see what DEA my app resides on (not currently enabled in our NG version) • Can’t see how many apps a user has (replaced by orgs and spaces, but still valuable to trace) • See https://github.com/cloudfoundry/cloud_controller_ng/issues/81  We’d like to restore that data, either surface it • in varz endpoints (dynamic data, preferred) or • CC_DB (static data, could be a security concern) Let’s resolve gaps in data captured from NG
- 26. 26  Detect errors in applications that are traceable to users/orgs • Preemptively reach out to them to see if they need help • Think customer service and proactive support! • Can we hook into to BOSH or Jenkins for automation?  Automate (and expand links to the IaaS and SaaS stacks) • Self healing systems (out of disk, move apps) • Self scaling systems (detect when nearing thresholds) • Evolving topologies (replace unused service nodes with popular ones) Let’s begin to link metrics to automation
- 27. 27  Admins are the primary beneficiary right now • But data is almost completely read only • Should we provide UAA based tiers of access to admins?  Others can and should benefit • Customers • End users • Developers • Management • Executives, line of business owners • Finance Let’s expand the broadcast of metrics to more users
- 28. 28 Thanks!
- 29. 29 The metrics dashboard innovators Chris Peters Russell Boykin Doug Davis Wei Feng
- 30. 30 We’re hiring! Search Jobs at IBM by: SmartCloud Application Services
- 31. 31