SlideShare a Scribd company logo
Monitoring in a
Scalable World
Jacob Tomlinson
Senior Solutions Engineer
Agenda
● The Informatics Lab
(10 mins)
● What makes a monitoring system ? (10 mins)
● Overview of technologies (10 mins)
● Making it scale
(10 mins)
● Questions
(10 mins)
Monitoring in a scalable world
Demystify
Buzzwords
Prototypes
www.informaticslab.co.uk
Monitoring in a scalable world
Monitoring in a scalable world
Monitoring in a scalable world
What is a
monitoring
system?
Monitoring in a scalable world
Monitoring in a scalable world
Monitoring in a scalable world
Logs vs Telemetry
Twelve-Factor App
XI. Logs
Treat logs as event streams
12factor.net
Monitoring in a scalable world
Components
Collection
Storage
Aggregation
&
Dashboards
Alerting
Monitoring
technologies
Managed services
Monitoring in a scalable world
Monitoring in a scalable world
Telegraf cAdvisor FilebeatTelegraf cAdvisor Filebeat
NGINX
Grafana Kibana
InfluxDB ElasticSearch
Logstash
Telegraf cAdvisor Filebeat
Lab Monitoring Stack v1
Kapacitor
Scaling
Scalability vs Availability
NGINX
Grafana Kibana
InfluxDB ElasticSearch
Logstash
Telegraf cAdvisor Filebeat
Kapacitor
NGINX
Grafana Kibana
InfluxDB ElasticSearch
Logstash
Telegraf cAdvisor Filebeat
Kapacitor
NGINX
Grafana Kibana
InfluxDB ElasticSearch
Logstash
Telegraf cAdvisor Filebeat
Kapacitor
Telegraf cAdvisor Filebeat
InfluxDB ElasticSearch
Kapacitor
Logstash
NGINX
Grafana Kibana
InfluxDB ElasticSearch
Logstash
Telegraf cAdvisor Filebeat
Kapacitor
Telegraf cAdvisor Filebeat
InfluxDB ElasticSearch
Logstash
NGINX
Grafana Kibana
NGINXGrafana Kibana
InfluxDB ElasticSearch
Logstash
cAdvisor Filebeat
Lab Monitoring Stack v2
Kapacitor
Telegraf
ELB
PostgreSQL
Questions
Jacob Tomlinson
www.informaticslab.co.uk
@_jacobtomlinson
inlb.co/tech-exeter-2016

More Related Content

PDF
InfluxDB and Grafana: An Introduction to Time-Based Data Storage and Visualiz...
PDF
Introduction to InfluxDB and TICK Stack
PDF
Developing Ansible Dynamic Inventory Script - Nov 2017
PDF
Measure your app internals with InfluxDB and Symfony2
PDF
Virtual training Intro to InfluxDB & Telegraf
PDF
InfluxDB & Grafana
PDF
Grafana introduction
InfluxDB and Grafana: An Introduction to Time-Based Data Storage and Visualiz...
Introduction to InfluxDB and TICK Stack
Developing Ansible Dynamic Inventory Script - Nov 2017
Measure your app internals with InfluxDB and Symfony2
Virtual training Intro to InfluxDB & Telegraf
InfluxDB & Grafana
Grafana introduction

What's hot (20)

PDF
Time Series Database and Tick Stack
PPTX
InfluxDb
PDF
Beautiful Monitoring With Grafana and InfluxDB
PDF
Introduction to influx db
PDF
Timeseries - data visualization in Grafana
PDF
Introduction to InfluxDB, an Open Source Distributed Time Series Database by ...
PDF
Kafka practical experience
PDF
Monitoring and Scaling Redis at DataDog - Ilan Rabinovitch, DataDog
PDF
Apache Flink Training Workshop @ HadoopCon2016 - #1 System Overview
PDF
Monitoring NGINX (plus): key metrics and how-to
PPTX
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
PPTX
Scaling an ELK stack at bol.com
PDF
Apache airflow
PDF
Flink Forward SF 2017: Kenneth Knowles - Back to Sessions overview
PPTX
Need for Time series Database
ODP
Nginx monitoring with graphite
PDF
Airflow introduction
PDF
Best Practices for Scaling an InfluxEnterprise Cluster
PPTX
How bol.com makes sense of its logs, using the Elastic technology stack.
PDF
Kafka on Kubernetes—From Evaluation to Production at Intuit
Time Series Database and Tick Stack
InfluxDb
Beautiful Monitoring With Grafana and InfluxDB
Introduction to influx db
Timeseries - data visualization in Grafana
Introduction to InfluxDB, an Open Source Distributed Time Series Database by ...
Kafka practical experience
Monitoring and Scaling Redis at DataDog - Ilan Rabinovitch, DataDog
Apache Flink Training Workshop @ HadoopCon2016 - #1 System Overview
Monitoring NGINX (plus): key metrics and how-to
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
Scaling an ELK stack at bol.com
Apache airflow
Flink Forward SF 2017: Kenneth Knowles - Back to Sessions overview
Need for Time series Database
Nginx monitoring with graphite
Airflow introduction
Best Practices for Scaling an InfluxEnterprise Cluster
How bol.com makes sense of its logs, using the Elastic technology stack.
Kafka on Kubernetes—From Evaluation to Production at Intuit
Ad

Viewers also liked (14)

PDF
Custom DevOps Monitoring System in MelOn (with InfluxDB + Telegraf + Grafana)
PDF
MPLS-TE Fault Tolerance
PDF
Scalable Monitoring & Alerting
PDF
Time Series Data with InfluxDB
PDF
Native container monitoring
PDF
Prometheus and Docker (Docker Galway, November 2015)
PPTX
How to Monitor Application Performance in a Container-Based World
PPTX
Sysdig Monitorama Slides
KEY
Introduction to Cassandra: Replication and Consistency
PDF
Analyzing Time Series Data with Apache Spark and Cassandra
PDF
SREcon 2016 Performance Checklists for SREs
PDF
Container orchestration
PPTX
DockerCon EU 2015: Docker Monitoring
PDF
TEDx Manchester: AI & The Future of Work
Custom DevOps Monitoring System in MelOn (with InfluxDB + Telegraf + Grafana)
MPLS-TE Fault Tolerance
Scalable Monitoring & Alerting
Time Series Data with InfluxDB
Native container monitoring
Prometheus and Docker (Docker Galway, November 2015)
How to Monitor Application Performance in a Container-Based World
Sysdig Monitorama Slides
Introduction to Cassandra: Replication and Consistency
Analyzing Time Series Data with Apache Spark and Cassandra
SREcon 2016 Performance Checklists for SREs
Container orchestration
DockerCon EU 2015: Docker Monitoring
TEDx Manchester: AI & The Future of Work
Ad

Similar to Monitoring in a scalable world (20)

PDF
YOW2018 Cloud Performance Root Cause Analysis at Netflix
PDF
Benmarking Orange Forge with CLIF, OW2con'15, November 17, Paris
 
PDF
Netflix Keystone—Cloud scale event processing pipeline
PDF
Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
PDF
Observability from the Home
PDF
nextcomputing-cyberpro
PDF
Yannis Zarkadas. Stefano Fioravanzo. Enterprise data science workflows on kub...
PDF
Yannis Zarkadas. Enterprise data science workflows on kubeflow
PDF
Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec
PDF
K. Tzoumas & S. Ewen – Flink Forward Keynote
PDF
Flink forward-2017-netflix keystones-paas
PDF
C++ for Marine Streamer Positioning and Navigation - ACCU 2011
PDF
OSMC 2022 | The Power of Metrics, Logs & Traces with Open Source by Emil-Andr...
PDF
Using Elastic to Monitor Anything
PDF
Circonus: Design failures - A Case Study
PDF
Monal Daxini - Beaming Flink to the Cloud @ Netflix
PDF
Beaming flink to the cloud @ netflix ff 2016-monal-daxini
PDF
Мониторинг облачной CI-системы на примере Jenkins / Александр Акбашев (HERE T...
PDF
Cloud CI Monitoring
PPTX
Jenkins world 2017 - Data-Driven CI Pipeline with Gerrit Code Review
YOW2018 Cloud Performance Root Cause Analysis at Netflix
Benmarking Orange Forge with CLIF, OW2con'15, November 17, Paris
 
Netflix Keystone—Cloud scale event processing pipeline
Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Observability from the Home
nextcomputing-cyberpro
Yannis Zarkadas. Stefano Fioravanzo. Enterprise data science workflows on kub...
Yannis Zarkadas. Enterprise data science workflows on kubeflow
Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec
K. Tzoumas & S. Ewen – Flink Forward Keynote
Flink forward-2017-netflix keystones-paas
C++ for Marine Streamer Positioning and Navigation - ACCU 2011
OSMC 2022 | The Power of Metrics, Logs & Traces with Open Source by Emil-Andr...
Using Elastic to Monitor Anything
Circonus: Design failures - A Case Study
Monal Daxini - Beaming Flink to the Cloud @ Netflix
Beaming flink to the cloud @ netflix ff 2016-monal-daxini
Мониторинг облачной CI-системы на примере Jenkins / Александр Акбашев (HERE T...
Cloud CI Monitoring
Jenkins world 2017 - Data-Driven CI Pipeline with Gerrit Code Review

More from TechExeter (20)

PPTX
Exeter Science Centre, by Natalie Whitehead
PPTX
South West InternetOfThings Network by Wo King
PPTX
Generative Adversarial Networks by Tariq Rashid
PDF
Conf 2019 - Workshop: Liam Glanfield - know your threat actor
PDF
Conf 2018 Track 1 - Unicorns aren't real
PDF
Conf 2018 Track 1 - Aerospace Innovation
PDF
Conf 2018 Track 2 - Try Elm
PPTX
Conf 2018 Track 3 - Creating marine geospatial services
PPTX
Conf 2018 Track 2 - Machine Learning with TensorFlow
PPTX
Conf 2018 Track 2 - Custom Web Elements with Stencil
PDF
Conf 2018 Track 1 - Tessl / revolutionising the house moving process
PPTX
Conf 2018 Keynote - Andy Stanford-Clark, CTO IBM UK
PPTX
Conf 2018 Track 3 - Microservices - What I've learned after a year building s...
PPTX
Gps behaving badly - Guy Busenel
PDF
Why Isn't My Query Using an Index?: An Introduction to SQL Performance
PPTX
Turning Developers into Testers
PDF
Remote working
PPTX
Developing an Agile Mindset
PDF
Think like a gardener
PDF
The trials and tribulations of providing engineering infrastructure
Exeter Science Centre, by Natalie Whitehead
South West InternetOfThings Network by Wo King
Generative Adversarial Networks by Tariq Rashid
Conf 2019 - Workshop: Liam Glanfield - know your threat actor
Conf 2018 Track 1 - Unicorns aren't real
Conf 2018 Track 1 - Aerospace Innovation
Conf 2018 Track 2 - Try Elm
Conf 2018 Track 3 - Creating marine geospatial services
Conf 2018 Track 2 - Machine Learning with TensorFlow
Conf 2018 Track 2 - Custom Web Elements with Stencil
Conf 2018 Track 1 - Tessl / revolutionising the house moving process
Conf 2018 Keynote - Andy Stanford-Clark, CTO IBM UK
Conf 2018 Track 3 - Microservices - What I've learned after a year building s...
Gps behaving badly - Guy Busenel
Why Isn't My Query Using an Index?: An Introduction to SQL Performance
Turning Developers into Testers
Remote working
Developing an Agile Mindset
Think like a gardener
The trials and tribulations of providing engineering infrastructure

Recently uploaded (20)

PDF
NewMind AI Weekly Chronicles - August'25-Week II
PDF
Univ-Connecticut-ChatGPT-Presentaion.pdf
PDF
DP Operators-handbook-extract for the Mautical Institute
PPTX
Tartificialntelligence_presentation.pptx
PDF
Web App vs Mobile App What Should You Build First.pdf
PPTX
Chapter 5: Probability Theory and Statistics
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
PDF
A comparative analysis of optical character recognition models for extracting...
PDF
A novel scalable deep ensemble learning framework for big data classification...
PDF
Unlocking AI with Model Context Protocol (MCP)
PDF
Mushroom cultivation and it's methods.pdf
PPTX
Group 1 Presentation -Planning and Decision Making .pptx
PPTX
SOPHOS-XG Firewall Administrator PPT.pptx
PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
PDF
A comparative study of natural language inference in Swahili using monolingua...
PDF
gpt5_lecture_notes_comprehensive_20250812015547.pdf
PDF
Hybrid model detection and classification of lung cancer
PDF
WOOl fibre morphology and structure.pdf for textiles
PDF
Zenith AI: Advanced Artificial Intelligence
PDF
Assigned Numbers - 2025 - Bluetooth® Document
NewMind AI Weekly Chronicles - August'25-Week II
Univ-Connecticut-ChatGPT-Presentaion.pdf
DP Operators-handbook-extract for the Mautical Institute
Tartificialntelligence_presentation.pptx
Web App vs Mobile App What Should You Build First.pdf
Chapter 5: Probability Theory and Statistics
Building Integrated photovoltaic BIPV_UPV.pdf
A comparative analysis of optical character recognition models for extracting...
A novel scalable deep ensemble learning framework for big data classification...
Unlocking AI with Model Context Protocol (MCP)
Mushroom cultivation and it's methods.pdf
Group 1 Presentation -Planning and Decision Making .pptx
SOPHOS-XG Firewall Administrator PPT.pptx
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
A comparative study of natural language inference in Swahili using monolingua...
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Hybrid model detection and classification of lung cancer
WOOl fibre morphology and structure.pdf for textiles
Zenith AI: Advanced Artificial Intelligence
Assigned Numbers - 2025 - Bluetooth® Document

Monitoring in a scalable world

Editor's Notes

  • #2: Introduce myself
  • #4: Informatics Lab
  • #5: Demystify Buzzwords
  • #7: Organisation using AWS & Azure Lab using AWS, but not for any particular reason
  • #15: Heroku
  • #16: CPU/Mem HTTP response codes
  • #34: Introduce myself