Intro to open source telemetry linux con 2016
1 like2,170 views
The document presents an introduction to datacenter telemetry using open source tools, focusing on the infrastructure for capturing and analyzing performance metrics. Key topics include the definition of telemetry, its differentiation from metrics and analytics, and the capabilities of the Snap framework for data collection and processing. It also discusses various telemetry tools and practices, alongside practical examples and setup instructions for users in a lab environment.
Intro to open source telemetry linux con 2016
- 1. LinuxCon 2016 An introduction to datacenter telemetry using open source tools Matt Brender (@mjbrender)
- 2. Briefly, About Me Am: @mjbrender (everywhere) Developer Advocate, Orchestration Engineering Pretty good at Open Source practices Was: Storage array performance VMware NoSQL
- 3. Loose Agenda 1. Wishful thinking of the lab config 2. What is telemetry 3. One opinion on the state of open source tooling
- 4. Let’s Test the Network 4 linuxcon.snap-telemetry.io then git clone I encourage you to keep downloading stuff until you’re ready to go.
- 5. Lab Hopes 5
- 6. High Level View 6 Grafana + InfluxDB Snap Snap “Admin” ”Production”
- 7. Less High Level View 7 Your Laptop Ubuntu 16.04 Vagrant Ubuntu 16.04Ubuntu 16.04
- 8. Less High Level View 8 Your Laptop Ubuntu 16.04 Vagrant Ansible Ubuntu 16.04Ubuntu 16.04 SnapDocker Snap
- 9. Less High Level View 9 Your Laptop Ubuntu 16.04 Vagrant Ansible Ubuntu 16.04Ubuntu 16.04 SnapDocker Snap Compose InfluxDB Grafana
- 10. Why??? 10
- 11. 11 Telemetry
- 13. 13 what my friends think telemetry is what my parents think telemetry is what society thinks telemetry is what my boss thinks telemetry is what I think telemetry is what telemetry actually is
- 14. What Is Telemetry? Telemetry is the stuff you can measure and the process of capturing it: from the heat generated on a CPU core to the throughput of Nginx* running in a Docker* container on a Kubernetes cluster. It’s all measurable and it’s all summarized in that one word. • Telemetry - the process of using equipment to take measurements of something and send them to another place • Metrics - measurements of facts throughout the data center • Analytics - the method of logical analysis that determines the consequences of information
- 15. What Is Telemetry? What How Application Availability ping Operating System Performance psutil Hardware Utilization Intel Performance Counter Metrics (PCM)
- 16. What Is Telemetry? What How Why Application Availability ping SLA compliance Operating System Performance psutil System performance Hardware Utilization Intel Performance Counter Metrics (PCM) Scaling capacity
- 17. What snap is and what it isn’t 17 Telemetry Analytics
- 18. What snap is and what it isn’t 18 Telemetry Analytics snap snap is a framework for metrics. snap is NOT an analytics alternative.
- 19. What snap is and what it isn’t 19 Telemetry Analytics Automation Scheduling IRO
- 20. collect process publish The Watcher Workflow 20
- 24. 24 Collectors in snap Collect telemetry data once via plugins for: § Bare metal, including Intel specific platform metrics (CPU, NIC, BMC, SMARTS) § Operating Environments and existing telemetry (Docker, libvirt, psutil) § Application services and adjacencies (Ceph, HAProxy, Etcd, Facter, MySQL, Apache) Populate a dynamically generated single-namespace telemetry catalog
- 25. 25 Filter, alter or append metadata via plugins for: § Filtering (Moving Averages) § Normalization § Encryption for all or part of the data set § Injection of metadata § Tokens § Tenant IDs Forking to one or more endpoints Processors in snap
- 26. 26 Publish data via plugins for: § Dashboard Tools (Graphite, Grafana, Riemann) § Queues and Logs (RabbitMQ, Kafka, File) § Databases (PostgreSQL, InfluxDB, OpenTSDB, SAP HANA) To one or more endpoints Publishers in snap
- 27. Visibility at all layers 27 App OS HW ? ? ? ? Analytics Pipeline Dashboards
- 28. Visibility at all layers 28 ? App OS HW Analytics Pipeline Dashboards
- 29. Visibility at all layers 29 Snap App OS HW Analytics Pipeline Dashboards
- 30. Visibility at all layers 30 OS HW Analytics Pipeline Dashboards App OS Virtualization HW App Snap
- 31. Visibility at all layers 31 OS HW Analytics Pipeline Dashboards App OS Virtualization HW App Snap
- 32. Visibility at all layers 32 OS HW Analytics Pipeline Dashboards App OS HW App Snap Kubernetes
- 33. Visibility at all layers OS HW App Snap Kubernetes OS HW App OS HW App OS HW App OS HW App OS HW App OS HW App
- 34. 34 REST & CLI Flexible Scheduling Caching Security Plugin Lifecycle Management Worker Queues Metric Catalog Tribe
- 43. Better Thought Leadership 43 by @obscurify by @caskey https://github.com/mjbrender/what-we-talk-about-when-we-talk-about-telemetry
- 44. Q&A 44
- 45. FAQ 45 Do I need telemetry?
- 46. FAQ 46 I don’t need telemetry, I have ____________.
- 47. FAQ 47 I don’t need telemetry, I have ____________.Graphite
- 49. FAQ 49 Do I need monitoring?
- 50. FAQ 50 We run ________ for monitoring.Nagios
- 52. What Is Telemetry? (revisited) What How Application Availability ping Operating System Performance psutil Hardware Utilization Intel Performance Counter Metrics (PCM)
- 53. What Is Telemetry? (revisited) What Query Collect Process Publish Visualize Application Availability ping ? ? ? ? Operating System Performance psutil ? ? ? ? Hardware Utilization PCM ? ? ? ? How Expanded
- 54. What Is Telemetry? (revisited) What Query Collect Process Publish Visualize Application Availability ping ? ? ? ? Operating System Performance psutil ? ? ? ? Hardware Utilization PCM ? ? ? ? How Expanded Snap Grafana
- 55. 55
- 56. Next Up 56 Start using Snap! • snap-telemetry.io • github.com/intelsdi-x Find me: • on The Geek Whisperers • and @mjbrender
- 58. Everything is Challenging At Scale 58
- 59. Add new task 59
- 60. Add new task 60
- 61. define as a tribe Scaling with Tribe 61
- 62. Scaling with Tribe Add new task 62
- 63. snap | What’s next? Physical/Virtual Host Scheduler Processing Publishing Collection 63
- 64. snap | What’s next? 64 Physical/VM Host Physical/VM Host Physical/VM Host Physical/VM Host Physical/VM Host Physical/VM Host Collection Collection Collection Scheduler Processing Publishing
- 65. § Plugin load § Dynamic, does not require restart § Automatically is informed by plugin on the features, metrics, and configuration detail. § Dynamically extends the metric catalog when loaded. § Plugin unload § Removes metrics from catalog automatically § Loading a new plugin automatically upgrades running workflows in tasks § Optionally the collection can be pinned to a version (ex: get /intel/server/cpu/load/v1) § Each scheduled workflow automatically uses the most mature plugin for that step § Coupled with dynamic plugin loading results in instantaneous updates to existing workflows § Helpful for bug fixes, security patching, improving accuracy snap | Plugin Lifecycle 65
- 66. Customizable definition of task and related workflow: Collect Publish Publish Collect Publish ProcessCollect Publish Collect Process Publish Process Publish snap | Overview – Example Workflows 66
- 67. The Catalog 67 Intel PCM psutil HAProxy /intel/psutil/load/load1 /intel/psutil/load/load5 /intel/psutil/vm/available /intel/pcm/EXEC /intel/pcm/FREQ /intel/linux/docker/cpu_stats/throttling_data/periods snapctl metric list /intel/server/health/score Docker Intel Health /intel/haproxy/info/MaxConnRate snap