Lambda Jam 2015: Event Processing in Clojure
0 likes208 views
The document discusses functional event processing in Clojure, specifically focusing on the Riemann event monitoring tool. It outlines the concepts of event streams, filtering, aggregation, and higher-order functions within Riemann, emphasizing Clojure's advantages for building and configuring monitoring solutions. The author provides sample use cases, configurations, and code examples to illustrate effective event processing practices.
![(ns hello.core)
(defn -main []
(prn “Hello World”))](https://image.slidesharecdn.com/rn2dzg5rloebmmzfd5u8-signature-d7192d91f1c601b9a75ee54c85f7f9229855e0242b0048b8af3bfe1e99dedf80-poli-170718014706/85/Lambda-Jam-2015-Event-Processing-in-Clojure-6-320.jpg)
![Jetty
(ns hello.core)
(defn -main []
(prn “Hello World”))
✓access requests
✓queues
✓sessions](https://image.slidesharecdn.com/rn2dzg5rloebmmzfd5u8-signature-d7192d91f1c601b9a75ee54c85f7f9229855e0242b0048b8af3bfe1e99dedf80-poli-170718014706/85/Lambda-Jam-2015-Event-Processing-in-Clojure-7-320.jpg)
![AWS EC2
Jetty
(ns hello.core)
(defn -main []
(prn “Hello World”))
✓disk
✓network interfaces
✓memory
✓cores/cpus](https://image.slidesharecdn.com/rn2dzg5rloebmmzfd5u8-signature-d7192d91f1c601b9a75ee54c85f7f9229855e0242b0048b8af3bfe1e99dedf80-poli-170718014706/85/Lambda-Jam-2015-Event-Processing-in-Clojure-8-320.jpg)
![(defrecord Event
[
service
state
description
metric
tags
time
ttl
KEY CONCEPTS: EVENTS
“reg1.foo.bar”
“reg-service http”
“ok”
“apache-log”
201
[“reg1” “http”]
104534892
60
riemannsource](https://image.slidesharecdn.com/rn2dzg5rloebmmzfd5u8-signature-d7192d91f1c601b9a75ee54c85f7f9229855e0242b0048b8af3bfe1e99dedf80-poli-170718014706/85/Lambda-Jam-2015-Event-Processing-in-Clojure-25-320.jpg)
![HOMOICONICITY
“[It] is where a program's source code is written
as a basic data structure that the
programming language knows how to access.”
Wikipedia
(defn validate-config
[file]
(try
(read-strings (slurp file))
(catch clojure.lang.LispReader$ReaderException e
(throw (logging/nice-syntax-error e file)))))
riemannsource](https://image.slidesharecdn.com/rn2dzg5rloebmmzfd5u8-signature-d7192d91f1c601b9a75ee54c85f7f9229855e0242b0048b8af3bfe1e99dedf80-poli-170718014706/85/Lambda-Jam-2015-Event-Processing-in-Clojure-29-320.jpg)
![HIGHER ORDER FUNCTIONS
“A stream is just a function that takes a variable
number of child streams and returns a function
that takes an event and responds to the event it is passed
when it is invoked.”
http://riemann.io/quickstart.html
“a higher-order function… does at least one of the
following:
[1] takes one or more functions as an input,
[2] outputs a function”
Wikipedia](https://image.slidesharecdn.com/rn2dzg5rloebmmzfd5u8-signature-d7192d91f1c601b9a75ee54c85f7f9229855e0242b0048b8af3bfe1e99dedf80-poli-170718014706/85/Lambda-Jam-2015-Event-Processing-in-Clojure-30-320.jpg)
![3
1
2
S-EXPRESSIONS
1 Split on event fields…
2 Detect state changes…
3 Collate and email
(streams
(by [:host :service]
(changed :state
(rollup 5 3600
(email “delacroix@vonbraun.com"))))
)
riemannconfig
“make the tree structure of the
streams visually apparent”](https://image.slidesharecdn.com/rn2dzg5rloebmmzfd5u8-signature-d7192d91f1c601b9a75ee54c85f7f9229855e0242b0048b8af3bfe1e99dedf80-poli-170718014706/85/Lambda-Jam-2015-Event-Processing-in-Clojure-37-320.jpg)
![MACROS
(defmacro pipe
[marker & stages]
`(let [~@(->> stages
reverse
(interpose marker)
(cons marker))]
~marker))
(streams
(pipe -
(splitp = service
"http req rate" -
"0mq req rate" (scale 2 -))
(rate 5
(graphite {:host “127.0.0.1” :port 2003})))
)
riemannconfig
riemannsource](https://image.slidesharecdn.com/rn2dzg5rloebmmzfd5u8-signature-d7192d91f1c601b9a75ee54c85f7f9229855e0242b0048b8af3bfe1e99dedf80-poli-170718014706/85/Lambda-Jam-2015-Event-Processing-in-Clojure-38-320.jpg)
![MACROS: PIPE
(streams
(let [aggregate
(rate 5
(graphite {:host “127.0.0.1” :port 2003}))]
(splitp = service
"http req rate" aggregate
"0mq req rate" (scale 2 aggregate)))
)
or
riemannconfigriemannconfig
(streams
(pipe -
(splitp = service
"http req rate" -
"0mq req rate" (scale 2 -))
(rate 5
(graphite {:host “127.0.0.1” :port 2003})))
)](https://image.slidesharecdn.com/rn2dzg5rloebmmzfd5u8-signature-d7192d91f1c601b9a75ee54c85f7f9229855e0242b0048b8af3bfe1e99dedf80-poli-170718014706/85/Lambda-Jam-2015-Event-Processing-in-Clojure-39-320.jpg)
![CLOJURE AD ABSURDUM
(defn do-you-trust-me? [event]
(prn (load-string (:description event))))
(streams
(where (service "event-gen")
do-you-trust-me?)
)
(ns event-gen.core
(:require [riemann.client :refer :all])
(:gen-class :main true))
(defn -main [& args]
(let [c (tcp-client {:host "127.0.0.1"})]
(send-event c
{:service "event-gen" :description "(+ 1 1)"})
(close-client c)))
riemannclientriemannconfig](https://image.slidesharecdn.com/rn2dzg5rloebmmzfd5u8-signature-d7192d91f1c601b9a75ee54c85f7f9229855e0242b0048b8af3bfe1e99dedf80-poli-170718014706/85/Lambda-Jam-2015-Event-Processing-in-Clojure-42-320.jpg)
![CLOJURE AD ABSURDUM
(defrecord Event
[service
description
])
“event-gen”
“(+ 1 1)”
Riemann
(Clojure)
riemann.config
(Clojure)
Client
(Clojure)
Clojure
=> “2”](https://image.slidesharecdn.com/rn2dzg5rloebmmzfd5u8-signature-d7192d91f1c601b9a75ee54c85f7f9229855e0242b0048b8af3bfe1e99dedf80-poli-170718014706/85/Lambda-Jam-2015-Event-Processing-in-Clojure-43-320.jpg)
Lambda Jam 2015: Event Processing in Clojure
- 2. FUNCTIONAL DEVOPSEVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com
- 3. Background
- 4. we all code!
- 6. (ns hello.core) (defn -main [] (prn “Hello World”))
- 7. Jetty (ns hello.core) (defn -main [] (prn “Hello World”)) ✓access requests ✓queues ✓sessions
- 8. AWS EC2 Jetty (ns hello.core) (defn -main [] (prn “Hello World”)) ✓disk ✓network interfaces ✓memory ✓cores/cpus
- 9. AWS EC2 Jetty AWS EC2 Jetty AWS EC2 Jetty DNS (Master) DNS (Slave) ✓records ✓replication ✓resolution
- 10. WS EC2 Jetty AWS EC2 Jetty AWS EC2 Jetty DNS (Master) DNS DB ✓performance ✓indexing ✓locking ✓connections
- 11. WS EC2 Jetty AWS EC2 Jetty AWS EC2 Jetty DNS (Master) DNS DB Load Balancer AWS EC2 Jetty AWS EC2 Jetty ✓connections ✓DoS ✓Clients
- 12. WS EC2 Jetty AWS EC2 Jetty AWS EC2 Jetty DNS (Master) DNS DB Load Balancer AWS EC2 Jetty AWS EC2 Jetty MessageQueue ✓queues ✓timeouts
- 13. that is a lot of information…
- 14. from a lot of different places
- 15. this is a problem
- 16. this is the problem Riemann helps solve
- 17. Riemann is written in Clojure 52 source files 7256 lines 377 functions 30 macros
- 18. Riemann is written and configured in Clojure
- 19. And so… to Riemann
- 21. OVERVIEW
- 22. Correlate events from disparate sources What is the latency of the web servers when the DB servers are generating timeouts? Filter events based on key criteria Ignore all network events with a normal status Aggregate events to perform statistical analysis (e.g., percentiles) What is the average response rate for 95% of requests? Monitor status of sources by availability of events Which services haven’t sent any events for the last 5 minutes? Route event streams to multiple consumers Send all error events from DB servers to the DBA team as well as the main operational dashboards SAMPLE RIEMANN USE CASES
- 23. i = f(e) e: thundering herd of events i: information ready for downstream processing
- 24. Hypothesis Event processing is function application at large scale… using Clojure to build/configure Riemann is a logical choice.
- 25. (defrecord Event [ service state description metric tags time ttl KEY CONCEPTS: EVENTS “reg1.foo.bar” “reg-service http” “ok” “apache-log” 201 [“reg1” “http”] 104534892 60 riemannsource
- 26. KEY CONCEPTS: STREAMS (logging/init {:file "riemann.log" :console true}) (instrumentation {:interval 5}) ;; self monitor every 5 seconds (periodically-expire 1) ;; check for expired events every second (tcp-server) (repl-server) (streams ;; insert magic here!!! prn) riemannconfig
- 27. KEY CONCEPTS: INDEX flow of events index riemann process
- 29. HOMOICONICITY “[It] is where a program's source code is written as a basic data structure that the programming language knows how to access.” Wikipedia (defn validate-config [file] (try (read-strings (slurp file)) (catch clojure.lang.LispReader$ReaderException e (throw (logging/nice-syntax-error e file))))) riemannsource
- 30. HIGHER ORDER FUNCTIONS “A stream is just a function that takes a variable number of child streams and returns a function that takes an event and responds to the event it is passed when it is invoked.” http://riemann.io/quickstart.html “a higher-order function… does at least one of the following: [1] takes one or more functions as an input, [2] outputs a function” Wikipedia
- 31. HIGHER ORDER FUNCTIONS “A stream is just a function that takes a variable number of child streams and returns a function that takes an event and responds to the event it is passed when it is invoked.” http://riemann.io/quickstart.html (streams (rate 5 prn) ) riemannconfig
- 32. IMMUTABILITY “Streams ‘change’ events by sending altered, shared-structure *copies* of events downstream.” http://riemann.io/howto.html#split-streams (streams (where (host "db04") (with :service "foo" prn) (with :service "bar" prn)) ) riemannconfig
- 34. “clojure's remarkably fast for what it is, sexprs make the tree structure of the streams visually apparent, it makes writing the concurrent algos much simpler, and macros allow us to express things like 'by and 'pipe in ways that would be awkward in other languages without building our own AST & interpreter or transpiler, etc.” “clojure's remarkably fast for what it is, sexprs make the tree structure of the streams visually apparent, it makes writing the concurrent algos much simpler, and macros allow us to express things like 'by and 'pipe in ways that would be awkward in other languages without building our own AST & interpreter or transpiler, etc.” @APHYR SAID…
- 35. S-EXPRESSIONS “a notation for nested list (tree-structured) data… popularised by the programming language Lisp” Wikipedia
- 36. S-EXPRESSIONS (streams (where (and (service #"^riak") (state “critical”)) (email “delacroix@vonbraun.com")) ) 1 Filter certain events… 2 Let people know… 1 2 riemannconfig “make the tree structure of the streams visually apparent”
- 37. 3 1 2 S-EXPRESSIONS 1 Split on event fields… 2 Detect state changes… 3 Collate and email (streams (by [:host :service] (changed :state (rollup 5 3600 (email “delacroix@vonbraun.com")))) ) riemannconfig “make the tree structure of the streams visually apparent”
- 38. MACROS (defmacro pipe [marker & stages] `(let [~@(->> stages reverse (interpose marker) (cons marker))] ~marker)) (streams (pipe - (splitp = service "http req rate" - "0mq req rate" (scale 2 -)) (rate 5 (graphite {:host “127.0.0.1” :port 2003}))) ) riemannconfig riemannsource
- 39. MACROS: PIPE (streams (let [aggregate (rate 5 (graphite {:host “127.0.0.1” :port 2003}))] (splitp = service "http req rate" aggregate "0mq req rate" (scale 2 aggregate))) ) or riemannconfigriemannconfig (streams (pipe - (splitp = service "http req rate" - "0mq req rate" (scale 2 -)) (rate 5 (graphite {:host “127.0.0.1” :port 2003}))) )
- 40. Conclusion
- 41. Hypothesis Event processing is function application at large scale: using Clojure to build/configure Riemann is a logical choice. homoiconicity higherorderfunctions s-expressions macros immutability
- 42. CLOJURE AD ABSURDUM (defn do-you-trust-me? [event] (prn (load-string (:description event)))) (streams (where (service "event-gen") do-you-trust-me?) ) (ns event-gen.core (:require [riemann.client :refer :all]) (:gen-class :main true)) (defn -main [& args] (let [c (tcp-client {:host "127.0.0.1"})] (send-event c {:service "event-gen" :description "(+ 1 1)"}) (close-client c))) riemannclientriemannconfig
- 43. CLOJURE AD ABSURDUM (defrecord Event [service description ]) “event-gen” “(+ 1 1)” Riemann (Clojure) riemann.config (Clojure) Client (Clojure) Clojure => “2”