SlideShare a Scribd company logo
Scaling out
Metrics Flow and Graphite
Vladimir Mevzos
PRESENTED BY:
Senior DevOps , Appsflyer
Measure Anything,Everything
Graphite Original Stack
Feeding Graphite with Data
· $ echo "foo.bar 1 `date +%s`" | nc localhost 2003
· $ echo [(foo.bar, (`date +%s`, 1)), …] | nc localhost 2003
Feeding Graphite with Data
· Plaintext protocol
· Pickle protocol
· StatsD
· Collectd
· Sensu
· Telegraf
Use your Data
· Grafana - UI for graphite
· Sensu – monitoring framework
· Autoscale/Healer – Homebrew ruby scripts
Aggregate Everything First
Scaling Collectd
Scaling StatsD
· AWS shared ENA (master/slave)
· Keepalived for HA and LB
· LVS – Linux Virtual Server UDP (DR ,TUN)
· StatsD Proxy vs statsite
Scaling Statsd
StatsD Kernel Trick
· net.ipv4.udp_mem > 10 MB
· net.core.wmem_max > 10 MB
Looks Good ...
But,if metrics > 30k/s
· Scaling
· Redundancy
· Performance
· Backup
· Management
What is other people do?
OpenSource projects
•Carbonapi
•Carbonzipper
•Gocarbon
•Carbon-c-relay
•Carbon-relay-ng
•Bucky
•Buckytools
Routing and Filtering
Tribute to carbon-c-relay
· Flexibility
· IOPS: Ephemeral / RAM / EBS
· Filtering by Regexp
· Buffer
· Clone
New Graphite Stack
Tribute to carbonapi
Tribute to go-carbon
· Performance
· Flexibility
· Easy to manage
Go-carbon Key Points
· IOPS: max-updates-per-second
· IOPS: scan-frequency
· CPU: max-cpu ,workers
· Cache: max-size,write-strategy
Measure Anything
Measure Everything
Questions?
vladimir@appsflyer.com

More Related Content

PPTX
Graphite
PDF
Graphite, an introduction
PDF
Graphite
PDF
Collecting metrics with Graphite and StatsD
PPTX
InfluxDb and Grafana fighting with data
PDF
Anatomy of an action
PPTX
The next generation of the Montage image mosaic engine
PDF
MapReduce with Hadoop
Graphite
Graphite, an introduction
Graphite
Collecting metrics with Graphite and StatsD
InfluxDb and Grafana fighting with data
Anatomy of an action
The next generation of the Montage image mosaic engine
MapReduce with Hadoop

What's hot (20)

PDF
Statsd introduction
PDF
"Metrics: Where and How", Vsevolod Polyakov
PPTX
What's new with serverless on google cloud
PDF
Climate data in r with the raster package
PDF
The Directions Pipeline at Mapbox - AWS Meetup Berlin June 2015
PDF
Cypher for Gremlin
PDF
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
PDF
Python and trending_data_ops
PDF
ACDKOCHI19 - Become Thanos of the Lambda Land: Wield all the Infinity Stones
PPTX
Stabilising the jenga tower
PPT
Apache Cassandra at Videoplaza — Stockholm Cassandra Users — September 2013
PDF
Cassandra&map reduce
PDF
Apache Storm
PDF
Parallel computing with GPars
PPTX
Всеволод Поляков (DevOps Team Lead в Grammarly)
PDF
QCon SF-2015 Stream Processing in uber
PDF
Kafka short
PDF
C* Summit 2013: Time-Series Metrics with Cassandra by Mike Heffner
ODP
PDF
Kinetic swift quick tour
Statsd introduction
"Metrics: Where and How", Vsevolod Polyakov
What's new with serverless on google cloud
Climate data in r with the raster package
The Directions Pipeline at Mapbox - AWS Meetup Berlin June 2015
Cypher for Gremlin
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Python and trending_data_ops
ACDKOCHI19 - Become Thanos of the Lambda Land: Wield all the Infinity Stones
Stabilising the jenga tower
Apache Cassandra at Videoplaza — Stockholm Cassandra Users — September 2013
Cassandra&map reduce
Apache Storm
Parallel computing with GPars
Всеволод Поляков (DevOps Team Lead в Grammarly)
QCon SF-2015 Stream Processing in uber
Kafka short
C* Summit 2013: Time-Series Metrics with Cassandra by Mike Heffner
Kinetic swift quick tour
Ad

Similar to Scaling metrics (20)

PDF
GPU-Accelerating A Deep Learning Anomaly Detection Platform
PDF
饿了么 TensorFlow 深度学习平台:elearn
PDF
Flink Forward Berlin 2017: Andreas Kunft - Efficiently executing R Dataframes...
PDF
S51281 - Accelerate Data Science in Python with RAPIDS_1679330128290001YmT7.pdf
ODP
Glusterfs for sysadmins-justin_clift
PDF
H2O Design and Infrastructure with Matt Dowle
PDF
Strata Singapore: Gearpump Real time DAG-Processing with Akka at Scale
PPTX
GPU Accelerated Data Science with RAPIDS - ODSC West 2020
PDF
Apache Flink internals
PDF
Hadoop Network Performance profile
PDF
Advanced kapacitor
PPTX
Flink internals web
PPTX
Performance Tuning Cheat Sheet for MongoDB
PDF
k8s-batch-sig_-_Dask_on_Kubernetes.pptx__1_.pdf
PDF
Infrastructure Monitoring with Postgres
PDF
Testing kubernetes and_open_shift_at_scale_20170209
PPTX
Pig on Tez: Low Latency Data Processing with Big Data
PDF
Building Google Cloud ML Engine From Scratch on AWS with PipelineAI - ODSC Lo...
PDF
Automated Application Management with SaltStack
PDF
Accelerated Machine Learning with RAPIDS and MLflow, Nvidia/RAPIDS
GPU-Accelerating A Deep Learning Anomaly Detection Platform
饿了么 TensorFlow 深度学习平台:elearn
Flink Forward Berlin 2017: Andreas Kunft - Efficiently executing R Dataframes...
S51281 - Accelerate Data Science in Python with RAPIDS_1679330128290001YmT7.pdf
Glusterfs for sysadmins-justin_clift
H2O Design and Infrastructure with Matt Dowle
Strata Singapore: Gearpump Real time DAG-Processing with Akka at Scale
GPU Accelerated Data Science with RAPIDS - ODSC West 2020
Apache Flink internals
Hadoop Network Performance profile
Advanced kapacitor
Flink internals web
Performance Tuning Cheat Sheet for MongoDB
k8s-batch-sig_-_Dask_on_Kubernetes.pptx__1_.pdf
Infrastructure Monitoring with Postgres
Testing kubernetes and_open_shift_at_scale_20170209
Pig on Tez: Low Latency Data Processing with Big Data
Building Google Cloud ML Engine From Scratch on AWS with PipelineAI - ODSC Lo...
Automated Application Management with SaltStack
Accelerated Machine Learning with RAPIDS and MLflow, Nvidia/RAPIDS
Ad

Recently uploaded (20)

PPTX
CEE 2 REPORT G7.pptxbdbshjdgsgjgsjfiuhsd
PPTX
Computer network topology notes for revision
PPTX
Supervised vs unsupervised machine learning algorithms
PDF
22.Patil - Early prediction of Alzheimer’s disease using convolutional neural...
PPTX
1_Introduction to advance data techniques.pptx
PPTX
Major-Components-ofNKJNNKNKNKNKronment.pptx
PPTX
MODULE 8 - DISASTER risk PREPAREDNESS.pptx
PPTX
Global journeys: estimating international migration
PDF
TRAFFIC-MANAGEMENT-AND-ACCIDENT-INVESTIGATION-WITH-DRIVING-PDF-FILE.pdf
PPTX
Database Infoormation System (DBIS).pptx
PDF
BF and FI - Blockchain, fintech and Financial Innovation Lesson 2.pdf
PPTX
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
PPTX
oil_refinery_comprehensive_20250804084928 (1).pptx
PPTX
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
PPTX
IBA_Chapter_11_Slides_Final_Accessible.pptx
PPTX
advance b rammar.pptxfdgdfgdfsgdfgsdgfdfgdfgsdfgdfgdfg
PPTX
Moving the Public Sector (Government) to a Digital Adoption
PPTX
Business Acumen Training GuidePresentation.pptx
PPTX
Introduction to Knowledge Engineering Part 1
CEE 2 REPORT G7.pptxbdbshjdgsgjgsjfiuhsd
Computer network topology notes for revision
Supervised vs unsupervised machine learning algorithms
22.Patil - Early prediction of Alzheimer’s disease using convolutional neural...
1_Introduction to advance data techniques.pptx
Major-Components-ofNKJNNKNKNKNKronment.pptx
MODULE 8 - DISASTER risk PREPAREDNESS.pptx
Global journeys: estimating international migration
TRAFFIC-MANAGEMENT-AND-ACCIDENT-INVESTIGATION-WITH-DRIVING-PDF-FILE.pdf
Database Infoormation System (DBIS).pptx
BF and FI - Blockchain, fintech and Financial Innovation Lesson 2.pdf
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
oil_refinery_comprehensive_20250804084928 (1).pptx
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
IBA_Chapter_11_Slides_Final_Accessible.pptx
advance b rammar.pptxfdgdfgdfsgdfgsdgfdfgdfgsdfgdfgdfg
Moving the Public Sector (Government) to a Digital Adoption
Business Acumen Training GuidePresentation.pptx
Introduction to Knowledge Engineering Part 1

Scaling metrics