OSMC 2022 | Metrics Stream Processing Using Riemann by Pradeep Chhertri
0 likes27 views
The document discusses Riemann, an event processing system designed for metrics, implemented in Clojure using Netty and Protocol Buffers. It covers core concepts like Riemann events and indices, as well as configuration management, testing, and scaling methods such as throttling and rollups. The document also includes practical examples and resources for further exploration of Riemann.
OSMC 2022 | Metrics Stream Processing Using Riemann by Pradeep Chhertri
- 1. Metrics Stream Processing Using Riemann Pradeep Chhetri
- 2. Metrics Stream Processing Using Riemann Pradeep Chhetri
- 3. Introduction
- 4. Introduction Event processing system designed specifically for metrics Written in Clojure Riemann configuration is Clojure code Built on Netty and Protocol Buffers Transient system Supports mutual TLS
- 6. Riemann Core Concepts Riemann Event Riemann Index Riemann Stream
- 10. Riemann Index Lock-free implementation of Java concurrent hashmap In memory Stores the latest event for a pair of (host, service) Event stays in index upto the ttl Riemann dash reads the index and displays the data
- 11. Riemann Stream
- 12. Riemann Stream Clojure Functions Filter events from stream Split a stream into multiple streams Merge multiple streams into one stream
- 19. Sending Data to Riemann
- 20. Sending Data to Riemann Collectd Logstash Syslog-ng
- 22. Riemann Configuration Riemann config is Clojure code Divide config into smaller, testable & reusable pieces using Clojure functions Organise them into Clojure namespaces
- 23. Configuration
- 26. Riemann Configuration: Testing Clojure tests Supports adding tests with riemann.test/tap Tap is like breakpoints where you would like to observe the stream
- 29. Detect Service Going Down TTL means how many seconds the event is valid for after its occurrence Once expired: - event is removed from index - state is changed to “expired” - reinjected into the stream
- 30. Things breaking Flood of events leading to flood of alerts Need to know that problem exists but don’t need to know every single failure
- 33. Things breaking: Rollup & Throttle
- 34. Scaling Riemann
- 35. Scaling Riemann No automatic sharding
- 36. Scaling Riemann We can make use of two streams: batch async-queue
- 37. Scaling Riemann