4Developers: Time series databases
1 like104 views
The document discusses the challenges and responsibilities of developers in managing time series databases for performance monitoring in complex applications. It emphasizes the importance of measuring various metrics to gain insights and optimize systems, highlighting examples of data collection, querying, and usage of different functions for data analysis. Additionally, it provides practical examples of monitoring tools and methods for effectively gathering and processing data.
![curl -G 'http://localhost:8086/query'
--data-urlencode "db=test"
--data-urlencode "q=SELECT value FROM cpu_load_short WHERE region='us-west'"
{"results":[{"series":[{"name":"cpu_load_short","columns":["time","value"],"values":
[["2015-06-11T20:46:02Z",0.64]]}]}]}](https://image.slidesharecdn.com/timeseriesdatabases-160504122052/85/4Developers-Time-series-databases-56-320.jpg)
![// create an array of points
$points = array(
new Point(
'test_metric', // name of the measurement
0.64, // the measurement value
['host' => 'server01', 'region' => 'us-west'], // optional tags
['cpucount' => 10], // optional additional fields
1435255849 // Time precision has to be set to seconds!
),
new Point(
'test_metric', // name of the measurement
0.84, // the measurement value
['host' => 'server01', 'region' => 'us-west'], // optional tags
['cpucount' => 10], // optional additional fields
1435255849 // Time precision has to be set to seconds!
)
);
// we are writing unix timestamps, which have a second precision
$result = $database->writePoints($points, Database::PRECISION_SECONDS);](https://image.slidesharecdn.com/timeseriesdatabases-160504122052/85/4Developers-Time-series-databases-57-320.jpg)
![CREATE CONTINUOUS QUERY <cq_name>
ON <database_name>
[RESAMPLE [EVERY <interval>] [FOR <interval>]]
BEGIN
SELECT <function>(<stuff>)[,<function>(<stuff>)]
INTO <different_measurement>
FROM <current_measurement>
[WHERE <stuff>]
GROUP BY time(<interval>)[,<stuff>]
END](https://image.slidesharecdn.com/timeseriesdatabases-160504122052/85/4Developers-Time-series-databases-65-320.jpg)
4Developers: Time series databases
- 1. @SrcMinistry @MariuszGil Time series databases Data processing
- 4. Code
- 5. Write it
- 6. Test it
- 7. Release it
- 8. Where does our responsibility end?
- 10. No man’s land
- 12. Context…
- 13. The story
- 14. One app 10+ servers Millions of users
- 15. Infra monitoring
- 16. Pro blems
- 17. What’s going on?
- 18. You can't manage what you can't measure W. Edwards Deming
- 19. Gather
- 20. 2016-04-11 02:00:00 45 2016-04-11 03:00:00 42 2016-04-11 04:00:00 30 2016-04-11 05:00:00 46 2016-04-11 06:00:00 70 2016-04-11 07:00:00 120
- 21. Store
- 22. Present
- 24. React
- 26. Metrics
- 27. Metrics, values that change over time Matches everything
- 29. Everything what is important for you
- 30. Tech CPU / PV / reqs / resp_time
- 31. Business KPIs / PV / UU
- 32. Why it should be measured?
- 33. Insights
- 35. Software system that’s optimized for handling arrays of numbers indexed by time Wikipedia definition
- 36. Many solutions…
- 42. Hello world
- 45. USE test Using database test
- 46. INSERT clicks,type=cpc,source=ad-partner-1,paid=yes,widget=promo-box value=0.43 INSERT clicks,type=cpc,source=ad-partner-2,paid=yes,widget=promo-box value=0.50 INSERT clicks,type=cpc,source=ad-partner-1,paid=yes,widget=top value=0.62 INSERT clicks,type=cpm,source=ad-partner-1,paid=yes,widget=top value=0.80
- 47. SELECT * FROM clicks name: clicks ------------ time paid source type value widget 1459850636770890539 yes ad-partner-1 cpc 0.43 promo-box 1459850643866389522 yes ad-partner-2 cpc 0.50 promo-box 1459850656407067481 yes ad-partner-1 cpc 0.62 top 1459850668781668282 yes ad-partner-1 cpm 0.80 top
- 48. SELECT COUNT(value), MEAN(value), MEDIAN(value) FROM clicks WHERE time > NOW() - 2m GROUP BY time(5s) name: clicks ------------ time count mean median 1459851275000000000 0 1459851280000000000 0 1459851285000000000 3 0.330 0.33 1459851290000000000 0 1459851295000000000 2 0.420 0.420 1459851300000000000 2 0.365 0.365 1459851305000000000 3 0.446 0.43 1459851310000000000 0 1459851315000000000 0
- 49. SELECT COUNT(value), MEAN(value), MEDIAN(value) FROM clicks WHERE time > NOW() - 2m GROUP BY time(5s) FILL(0) name: clicks ------------ time count mean median 1459851275000000000 0 0 0 1459851280000000000 0 0 0 1459851285000000000 3 0.330 0.33 1459851290000000000 0 0 0 1459851295000000000 2 0.420 0.420 1459851300000000000 2 0.365 0.365 1459851305000000000 3 0.446 0.43 1459851310000000000 0 0 0 1459851315000000000 0 0 0
- 52. Real world
- 53. Writing data
- 54. App HTTP API Graphite protocol Collectd protocol JSON+UDP
- 55. curl -i -XPOST 'http://localhost:8086/write?db=test' --data-binary 'cpu_load_short,host=server01,region=us-west value=0.64 1434055562000000000’ HTTP/1.1 204 No Content Request-Id: 02e7757f-fb1e-11e5-809d-000000000000 X-Influxdb-Version: 0.11.0 Date: Tue, 05 Apr 2016 11:03:22 GMT
- 56. curl -G 'http://localhost:8086/query' --data-urlencode "db=test" --data-urlencode "q=SELECT value FROM cpu_load_short WHERE region='us-west'" {"results":[{"series":[{"name":"cpu_load_short","columns":["time","value"],"values": [["2015-06-11T20:46:02Z",0.64]]}]}]}
- 57. // create an array of points $points = array( new Point( 'test_metric', // name of the measurement 0.64, // the measurement value ['host' => 'server01', 'region' => 'us-west'], // optional tags ['cpucount' => 10], // optional additional fields 1435255849 // Time precision has to be set to seconds! ), new Point( 'test_metric', // name of the measurement 0.84, // the measurement value ['host' => 'server01', 'region' => 'us-west'], // optional tags ['cpucount' => 10], // optional additional fields 1435255849 // Time precision has to be set to seconds! ) ); // we are writing unix timestamps, which have a second precision $result = $database->writePoints($points, Database::PRECISION_SECONDS);
- 58. Data model
- 59. $statsd = new LeagueStatsDClient(); $statsd->increment('web.pageview'); $statsd->decrement('storage.remaining'); $statsd->increment(array( 'first.metric', 'second.metric' ), 2); $statsd->increment('web.clicks', 1, 0.5); Counters
- 60. $statsd = new LeagueStatsDClient(); $statsd->gauge('api.logged_in_users', 123456); Gauges
- 61. $statsd = new LeagueStatsDClient(); $statsd->timing('api.response_time', 23.1); Timers
- 62. $statsd = new LeagueStatsDClient(); $userID = 23; $statsd->set('api.unique_logins', $userID); Sets
- 63. Data volume
- 65. CREATE CONTINUOUS QUERY <cq_name> ON <database_name> [RESAMPLE [EVERY <interval>] [FOR <interval>]] BEGIN SELECT <function>(<stuff>)[,<function>(<stuff>)] INTO <different_measurement> FROM <current_measurement> [WHERE <stuff>] GROUP BY time(<interval>)[,<stuff>] END
- 67. Rsyslog
- 68. Rsyslog
- 69. Rsyslog
- 70. Possibilities!
- 71. App heart beat
- 76. Alternatives
- 79. Only rocket fuel