Making sense of your data jug
0 likes165 views
The document details the development of a custom MongoDB datasource for Grafana using Vert.x, which addresses the challenges of time series data visualization. It discusses the architecture of Grafana, the benefits of using Vert.x for high performance and scalability, and outlines the steps to implement an efficient datasource that optimizes data queries. Key takeaways emphasize Grafana's utility in data visualization and the reactive nature of Vert.x for creating responsive applications.
![/search Format
Request
{
"target" : "select metric",
"refId" : "E"
}
Response
[
"Metric Name 1",
"Metric Name2",
]
An array of strings
12](https://image.slidesharecdn.com/makingsenseofyourdata-jug-170522045234/85/Making-sense-of-your-data-jug-12-320.jpg)
![/annotations Format
Request
{ "annotation" : {
"name" : "Test",
"iconColor" : "rgba(255, 96, 96, 1)",
"datasource" : "Simple Example DS",
"enable" : true,
"query" : "{"name":"Timeseries A"}" },
"range" : {
"from" : "2016-06-13T12:23:47.387Z",
"to" : "2016-06-13T12:24:19.217Z" },
"rangeRaw" : {
"from" : "2016-06-13T12:23:47.387Z",
"to" : "2016-06-13T12:24:19.217Z"
} }
Response
[ { "annotation": {
"name": "Test",
"iconColor": "rgba(255, 96, 96, 1)",
"datasource": "Simple Example DS",
"enable": true,
"query": "{"name":"Timeseries A"}" },
"time": 1465820629774,
"title": "Marker",
"tags": [
"Tag 1",
"Tag 2" ] } ]
13](https://image.slidesharecdn.com/makingsenseofyourdata-jug-170522045234/85/Making-sense-of-your-data-jug-13-320.jpg)
![/query Format
Request
{ "panelId" : 1,
"maxDataPoints" : 1904,
"format" : "json",
"range" : {
"from" : "2016-06-13T12:23:47.387Z",
"to" : "2016-06-13T12:24:19.217Z" },
"rangeRaw" : {
"from" : "2016-06-13T12:23:47.387Z",
"to" : "2016-06-13T12:24:19.217Z" },
"interval" : "20ms",
"targets" : [ {
"target" : "Time series A",
"refId" : "A" },] }
Response
[ { "target":"Timeseries A",
"datapoints":[
[1936,1465820629774],
[2105,1465820632673],
[4187,1465820635570],
[30001,1465820645243] },
{ "target":"Timeseries B",
"datapoints":[ ] }
]
14](https://image.slidesharecdn.com/makingsenseofyourdata-jug-170522045234/85/Making-sense-of-your-data-jug-14-320.jpg)
Making sense of your data jug
- 1. Making Sense of your Data BUILDING A CUSTOM MONGODB DATASOURCE FOR GRAFANA WITH VERTX 1
- 2. About me IT Consultant & Java Specialist at DevCon5 (CH) Focal Areas Tool-assisted quality assurance Performance (-testing, -analysis, -tooling) Operational Topics (APM, Monitoring) Twitter: @gmuecke 2
- 3. The Starting Point Customer stored and keep response time measurement of test runs in a MongoDB Lots of Data Timestamp & Value No Proper Visualization 3
- 4. What are timeseries data? a set of datapoints with a timestamp and a value time value
- 5. What is MongoDB? MongoDB NoSQL database with focus on scale JSON as data representation No HTTP endpoint (TCP based Wire Protocol) Aggregation framework for complex queries Provides an Async Driver 5
- 6. What is Grafana? A Service for Visualizing Time Series Data Open Source Backend written in Go Frontend based on Angular Dashboards & Alerts
- 7. Grafana Architecture 7 Grafana Server • Implemented in GO • Persistence for Settings and Dashboards • Offers Proxy for Service Calls Browser Angular UI Data Source Data Source Plugin... Proxy DB DB
- 8. Datasources for Grafana 8 Grafana Server • Implemented in GO • Persistence for Settings and Dashboards • Offers Proxy for Service Calls Browser Datasource Angular UI Data Source Plugin • Angular • JavaScript HTTP
- 9. Connect Angular Directly to Mongo? 9
- 10. From 2 Tier to 3 Tier 10 Grafana (Angular) Mongo DB Grafana (Angular) Mongo DB Datasource Service HTTP Mongo Wire Protocol
- 11. Start Simple SimpleJsonDatasource (Plugin) 3 ServiceEndpoints /search Labels – names of available timeseries /annotations Annotations – textual markers /query Query – actual time series data 11 https://github.com/grafana/simple-json-datasource
- 12. /search Format Request { "target" : "select metric", "refId" : "E" } Response [ "Metric Name 1", "Metric Name2", ] An array of strings 12
- 13. /annotations Format Request { "annotation" : { "name" : "Test", "iconColor" : "rgba(255, 96, 96, 1)", "datasource" : "Simple Example DS", "enable" : true, "query" : "{"name":"Timeseries A"}" }, "range" : { "from" : "2016-06-13T12:23:47.387Z", "to" : "2016-06-13T12:24:19.217Z" }, "rangeRaw" : { "from" : "2016-06-13T12:23:47.387Z", "to" : "2016-06-13T12:24:19.217Z" } } Response [ { "annotation": { "name": "Test", "iconColor": "rgba(255, 96, 96, 1)", "datasource": "Simple Example DS", "enable": true, "query": "{"name":"Timeseries A"}" }, "time": 1465820629774, "title": "Marker", "tags": [ "Tag 1", "Tag 2" ] } ] 13
- 14. /query Format Request { "panelId" : 1, "maxDataPoints" : 1904, "format" : "json", "range" : { "from" : "2016-06-13T12:23:47.387Z", "to" : "2016-06-13T12:24:19.217Z" }, "rangeRaw" : { "from" : "2016-06-13T12:23:47.387Z", "to" : "2016-06-13T12:24:19.217Z" }, "interval" : "20ms", "targets" : [ { "target" : "Time series A", "refId" : "A" },] } Response [ { "target":"Timeseries A", "datapoints":[ [1936,1465820629774], [2105,1465820632673], [4187,1465820635570], [30001,1465820645243] }, { "target":"Timeseries B", "datapoints":[ ] } ] 14
- 15. Structure of the Source Data { "_id" : ObjectId("56375bc54f3c4caedfe68aca"), "t" : { "eDesc" : "some description", "eId" : "56375ae24f3c4caedfe68a07", "name" : "some name", "profile" : "I01", "rnId" : "56375b694f3c4caedfe68aa0", "rnStatus" : "PASSED", "uId" : "anonymous" }, "n" : { "begin" : NumberLong("1446468494689"), "value" : NumberLong(283) } } 15
- 16. Custom Datasource Should be Lightweight Fast / Performant Simple 16
- 17. Microservice? Options for implementation Java EE Microservice (i.e. Wildfly Swarm) Springboot Microservice Vert.x Microservice Node.js ... 17
- 18. The Alternative Options Node.js Single Threaded Child Worker Processes Javascript Only Not best-choice for heavy computation Spring / Java EE Multithreaded Clusterable Java Only Solid Workhorses, cumbersome at times 19
- 19. Why Vert.x? High Performance, Low Footprint Asynchronous, Non-Blocking Actor-like Concurrency Event & Worker Verticles Message Driven Polyglott Java, Groovy, Javascript, Scala … Scalable Distributed Eventbus Multi-threaded Event Loops 20
- 21. Asynchronous non-blocking vs Synchronous blocking 23 © Fritz Geller-Grimm © Dontworry
- 22. Event Loop 24 Photo: Andreas Praefcke
- 23. Event Loop and Verticles 25 Photo: RokerHRO 3rd Floor, Verticle A 2nd Floor, Verticle B 1st Floor, Verticle C
- 24. 26
- 25. 27
- 28. CPU Multi-Reactor 31 Core Core Core Core Eventbus Other Vert.x Instance Browser Verticle Verticle
- 29. Event & Worker Verticles Event Driven Verticles Worker Verticles 32 Verticle Verticle Verticle Thread Pool Thread Pool Verticle Verticle Verticle Verticle Verticle
- 30. Implementing the datasource Http Verticle Routing requests & sending responses Verticles querying the DB Searching timeseries labels Annotation Timeseries data points Optional Verticles for Post Processing 33
- 31. What is the challenge? Optimization Queries can be optimized Large datasets have to be searched, read and transported Source data can not be modified VS data redundancy Sizing How to size the analysis system without knowing the query-times? How to size thread pools or database pools if most of the queries will take 100ms – 30s ? 34
- 32. CPU Datasource Architecture 35 HTTP Service Eventbus Timeseries HTTP Request HTTP Response DB Labels Annotations
- 33. Step 1 – The naive approach Find all datapoints within range 36
- 34. CPU Datasource Architecture 37 HTTP Service Eventbus Query Database HTTP Request HTTP Response DB
- 35. Step 2 – Split Request Split request into chunks (#chunks = #cores) Use multiple Verticle Instance in parallel (#instances = #cores) ? 38 CPU
- 36. CPU Datasource Architecture 39 HTTP Service Split/ Merge Request Eventbus Query Database Query Database Query Database Query Database HTTP Request HTTP Response DB
- 37. Step 3 – Aggregate Datapoints Use Mongo Aggregation Pipeline Reduce Datapoints returned to service 40
- 38. CPU Datasource Architecture 41 HTTP Service Split/ Merge Request Eventbus Query Database Query Database Query Database Query Database HTTP Request HTTP Response DB
- 39. Step 4 – Percentiles (CPU) Fetch all data Calculate percentiles in service 42 CPU
- 40. Step 4 – Percentiles (DB) Build aggregation pipeline to calculate percentiles Algorithm, see http://www.dummies.com/education/math/statistics/how-to- calculate-percentiles-in-statistics/ 43 DB
- 41. CPU Datasource Architecture 44 HTTP Service Split/ Merge Request Eventbus Query Database Query Database Query Database Query Database HTTP Request HTTP Response DB
- 42. Step 5 - Postprocessing Apply additional computation on the result from the database 45
- 43. CPUCPU Datasource Architecture (final) 46 HTTP Service Split Request Eventbus Query Database Query Database Query Database Query Database Merge Result HTTP Request HTTP Response DB Post Process Post Process Post Process Post Process Eventbus
- 44. Anyone recognize this chip? CPU of the PS3 (Cell BE) Central Bus (EIB) 1 General Purpose CPU Runs the Game (Event) Loop 8 SPUs Special Processing Sound Physics AI ... 230 GFLOPS (vs 103 GFLOPS PS4) 47
- 45. Adding more stats & calculation Push Calculation to DB if possible Add more workers / node for complex (post-) processing Aggregate results before post-processing DB performance is king 48
- 46. Custom vs. timeseries DB Custom: No migration of existing data No redundant data storage More flexibility Better extensibility Custom views Custom aggregation More options scalability retention Timeseries DB: Better out-of-the-box experience / performance integration functionality Built-in retention policies Built for scalability 49
- 47. Takeaways Grafana is powerful tool for visualizing data Information becomes consumable through visualization Information is essential for decision making Vert.x is Reactive, Non-Blocking, Asynchronous, Scalable Running on JVM Polyglott Fun 52 Source code on: https://github.com/gmuecke/grafana-vertx-datasource
- 48. Still hungry? FOR GEEK STUFF 53
- 49. Let’s read a large data file Datafile is large (> 1GB) Every line of the file is a datapoint The first 10 characters are a timestamp The dataset is sorted The datapoints are not equally distributed Grafana requires reads ~1900 datapoints per chart request 54
- 50. The Challenges (pick one) How to randomly access 1900 datapoints without reading the entire file into memory? How to read a huge file efficiently into memory? 55 Index + Lazy refinement Index + Lazy load
- 51. Let’s build an index Indexes can be build using a tree- datastructure Node: Timestamp Leaf: offset position in file or the datapoint Red-Black Trees provide fast access Read/Insert O(log n) Space n 56 © Cburnett, Wikipedia
- 52. 57
- 53. java.util.TreeMap is a red-black tree based implementation* TreeMap<Long,Long> index = new TreeMap<>(); *actually all HashMaps are 58
- 54. How to build an index (fast)? Read datapoint from offset positions Build a partial index 59 Dataset
- 55. On next query Locate Block Refine Block Update Index 60
- 56. CPUCPU Datasource Architecture (again) 61 HTTP Service Split Request Eventbus Read File Read File Read File Read File Merge Result HTTP Request HTTP Response Dataset Post Process Post Process Post Process Post Process Eventbus