Tick Stack - Listen your infrastructure and please sleep
0 likes446 views
Gianluca Arbezzano discusses the challenges and tools related to monitoring infrastructure and applications, emphasizing the importance of being familiar with system applications. The document introduces time series data and tools like InfluxDB, Telegraf, and Kapacitor for effective data management and alerting. He advocates for a well-structured monitoring system to enhance operational efficiency and application performance.
![11
2016/04/15 15:42:46 [warn] 2330#0: *167 using uninitialized
variable,
client: 10.0.1.1, server: localhost.dev, request: "POST /auth
HTTP/1.1",
host: "localhost"
2016/04/15 15:44:44 [error] 2330#0: *171 FastCGI sent in stderr: "
PHP message: PHP Fatal error: Uncaught exception
'RuntimeException' with message 'All broken)[500]'
in /var/www/my/project.php:237
Stack trace:
#0 /var/www/index.php:45 ObjectService->flush()
#1 [internal function] ->save()](https://image.slidesharecdn.com/tmstickstack-170412060737/85/Tick-Stack-Listen-your-infrastructure-and-please-sleep-11-320.jpg)
![We are here to speak about
Time Series
[
{
"name": "log_lines",
"columns": ["time", "line"],
"point": [1400425947368, "here's some useful log info"]
}
]
17](https://image.slidesharecdn.com/tmstickstack-170412060737/85/Tick-Stack-Listen-your-infrastructure-and-please-sleep-17-320.jpg)
![EASY! EASY! EASY!
{
"name": "cpu_percent_use",
"columns": ["value"],
"point": 40
}
18](https://image.slidesharecdn.com/tmstickstack-170412060737/85/Tick-Stack-Listen-your-infrastructure-and-please-sleep-18-320.jpg)
![28
[key] [fields] [timestamp]
temperature,machine=unit internal=3,external=10 1434055562000000035
Inline Protocol thinked to be smart and slim](https://image.slidesharecdn.com/tmstickstack-170412060737/85/Tick-Stack-Listen-your-infrastructure-and-please-sleep-28-320.jpg)
![29
CorleyBenchmarksInfluxDBAdapterEvent
Method Name Iterations Average Time Ops/second
------------------------ ------------ -------------- -------------
sendDataUsingHttpAdapter: [1,000 ] [0.0026700308323] [374.52751]
sendDataUsingUdpAdapter : [1,000 ] [0.0000436344147] [22,917.69026]
UDP vs TCP protocol](https://image.slidesharecdn.com/tmstickstack-170412060737/85/Tick-Stack-Listen-your-infrastructure-and-please-sleep-29-320.jpg)
Tick Stack - Listen your infrastructure and please sleep
- 1. Listen to you infrastructure and please sleep
- 2. Hello! I am Gianluca Arbezzano Software Engineer at CurrencyFair @gianarb on Twitter and GitHub 2
- 3. 3 Open Source Maintainer and Contributors penny.gianarb.it vim-php.org
- 4. 4 scaledocker.com Some resources about Docker
- 5. “ “Try again. Fail again. Fail better.” cit. Samuel Beckett 5
- 6. Trust your system To be familiar with your applications you need to know what they are doing. 6
- 9. 1 Widespread Monitoring Tools tail -f /var/log/app.log 9
- 10. 10
- 11. 11 2016/04/15 15:42:46 [warn] 2330#0: *167 using uninitialized variable, client: 10.0.1.1, server: localhost.dev, request: "POST /auth HTTP/1.1", host: "localhost" 2016/04/15 15:44:44 [error] 2330#0: *171 FastCGI sent in stderr: " PHP message: PHP Fatal error: Uncaught exception 'RuntimeException' with message 'All broken)[500]' in /var/www/my/project.php:237 Stack trace: #0 /var/www/index.php:45 ObjectService->flush() #1 [internal function] ->save()
- 14. Difficult not impossible They do an amazing work. And there are other tools!! 14
- 15. They are awesome for some use cases ▪ Extract informations ▪ They can be “human readable” ▪ … and others, and others 15
- 16. Keep your life amazing Reduce your time series to a timestamp and a value (int or float) 16 This is cheap and useful
- 17. We are here to speak about Time Series [ { "name": "log_lines", "columns": ["time", "line"], "point": [1400425947368, "here's some useful log info"] } ] 17
- 18. EASY! EASY! EASY! { "name": "cpu_percent_use", "columns": ["value"], "point": 40 } 18
- 19. “ “Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius — and a lot of courage to move in the opposite direction.” Cit. Ernest F. Shumacher 19
- 20. Time is a perfect sharding key It means that Time Series scale really well 20
- 21. We have a set of tools to use 21
- 22. 22 InfluxDB ● Optimized to store time series data ● Open Source and easy to install (go binary) ● Big community and ecosystem to manage alert and collect metrics
- 23. 23 Easy Install and start a go binary wget https://dl.influxdata.com/influxdb/releases/influxdb_1.2.2_amd64.deb sudo dpkg -i influxdb_1.2.2_amd64.deb Influxd -config /usr/local/etc/influxdb.conf
- 24. 24 Easy HTTP API on port 8086 Support for UDP connections Powerful CLI to communicate with the db
- 25. 25 Easy SELECT value FROM cpu_load_short WHERE region='us-west'
- 26. 26
- 27. 27
- 28. 28 [key] [fields] [timestamp] temperature,machine=unit internal=3,external=10 1434055562000000035 Inline Protocol thinked to be smart and slim
- 29. 29 CorleyBenchmarksInfluxDBAdapterEvent Method Name Iterations Average Time Ops/second ------------------------ ------------ -------------- ------------- sendDataUsingHttpAdapter: [1,000 ] [0.0026700308323] [374.52751] sendDataUsingUdpAdapter : [1,000 ] [0.0000436344147] [22,917.69026] UDP vs TCP protocol
- 30. 30 CREATE CONTINUOUS QUERY minnie ON world BEGIN SELECT min(mouse) INTO min_mouse FROM zoo GROUP BY time(30m) END Continuous Query
- 31. Telegraf https://github.com/influxdata/telegraf Collector to grab and send data from different sources to InfluxDB and other databases 31 Based on Input and out Plugin System
- 33. Kapacitor https://github.com/influxdata/kapacitor Trigger notifications and make action in case of specific behaviors 33 framework for processing, monitoring, and alerting on time series data
- 34. 34 Kapacitor high CPU alert stream |from() .measurement('cpu_usage_idle') .groupBy('host') |window() .period(1m) .every(1m) |mean('value') |eval(lambda: 100.0 - "mean").as('used') |alert() .message('{{ .Level}}: {{ .Name }}/{{ index .Tags "host" }} has high cpu: {{ index .Fields "used" }}') .warn(lambda: "used" > 70.0) .crit(lambda: "used" > 85.0) // Send alert to hander of choice. // Slack .slack() .channel('#alerts') // PagerDuty .pagerDuty()
- 35. 35 When you start to work with "micro"services understand the topology of your connections is really important Time series can help you
- 36. “ demo 36
- 37. 37 Use a dashboard to put together different metrics and create not obvious relations
- 38. Badoo migrated to PHP 7
- 39. @dgryski tested for 10 mins a service in Golang (previously it was in Perl)
- 40. Sometimes you just need to compare
- 41. 41 Why InfluxDB and not something else? https://www.influxdata.com/influxdb-is-27x-faster-vs-mongodb-for-time-series-workloads/ 27x greater write throughput 84x less disk space
- 42. 42 That’s it! A series of great tools to monitor your applications and your infrastructure
- 43. 43 A monitoring system isn’t for all
- 44. “Anybody” who?!? 44 ● Think about your monitoring system as “as a service” tools. ● different location (VPC or network) ● Proper team, contingency plan, deploy, documentation, everything you can! ● HA