Spec + onyx
0 likes571 views
The document presents an experience report on Onyx, a masterless, cloud-scale distributed computation system built in Clojure, highlighting its use in ETL, online and offline machine learning, and ad-hoc analysis. It discusses the architecture, including job workflows and flow conditions, as well as automation through queryable data descriptions and materialized view generation. Key takeaways emphasize Onyx's readiness for production, the importance of clean architecture, and the potential for interactive development using computation graphs.
![Job =
[[:input :processing-1]
[:input :processing-2]
[:processing-1 :output-1]
[:processing-2 :output-2]]
[{:flow/from :input-stream
:flow/to [:process-adults]
:flow/predicate :my.ns/adult?
:flow/doc "Emits segment if an adult.”}]
workflow
+ flow conditions
+ catalogue[{:onyx/name :add-5
:onyx/fn :my/adder
:onyx/type :function
:my/n 5
:onyx/params [:my/n]
:onyx/batch-size batch-size}
{:onyx/name :in
:onyx/plugin :onyx.plugin.core-async/input
:onyx/type :input
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Reads segments from a core.async channel"}
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/type :output
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Writes segments to a core.async channel"}]](https://image.slidesharecdn.com/speconyx-170217161721/85/Spec-onyx-7-320.jpg)
![Catalogue
[{:onyx/name :add-5
:onyx/fn :my/adder
:onyx/type :function
:my/n 5
:onyx/params [:my/n]
:onyx/batch-size batch-size}
{:onyx/name :in
:onyx/plugin :onyx.plugin.core-async/input
:onyx/type :input
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Reads segments from a core.async channel"}
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/type :output
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Writes segments to a core.async channel"}]
Vanilla Clojure function
(defn adder [n {:keys [x] :as segment}]
(assoc segment :x (+ n x))))
Plugins (I/O)
seq, async, Kafka,
Datomic, SQL,…
parameter
self-documenting](https://image.slidesharecdn.com/speconyx-170217161721/85/Spec-onyx-8-320.jpg)
Spec + onyx
- 1. Spec + Onyx: an experience report @sbelak simon@goopti.com
- 2. Onyxa masterless, cloud scale, fault tolerant, high performance distributed computation system … written entirely in Clojure
- 3. Onyx at • In production for almost a year • ETL • online machine learning • offline (batch) machine learning • ad-hoc analysis
- 4. Self-service infrastructure for data scientists
- 5. 1.Onyx at a glance 2.How Onyx rewired my brain 3.Building on top of spec
- 6. Onyx at a glance
- 7. Job = [[:input :processing-1] [:input :processing-2] [:processing-1 :output-1] [:processing-2 :output-2]] [{:flow/from :input-stream :flow/to [:process-adults] :flow/predicate :my.ns/adult? :flow/doc "Emits segment if an adult.”}] workflow + flow conditions + catalogue[{:onyx/name :add-5 :onyx/fn :my/adder :onyx/type :function :my/n 5 :onyx/params [:my/n] :onyx/batch-size batch-size} {:onyx/name :in :onyx/plugin :onyx.plugin.core-async/input :onyx/type :input :onyx/medium :core.async :onyx/batch-size batch-size :onyx/max-peers 1 :onyx/doc "Reads segments from a core.async channel"} {:onyx/name :out :onyx/plugin :onyx.plugin.core-async/output :onyx/type :output :onyx/medium :core.async :onyx/batch-size batch-size :onyx/max-peers 1 :onyx/doc "Writes segments to a core.async channel"}]
- 8. Catalogue [{:onyx/name :add-5 :onyx/fn :my/adder :onyx/type :function :my/n 5 :onyx/params [:my/n] :onyx/batch-size batch-size} {:onyx/name :in :onyx/plugin :onyx.plugin.core-async/input :onyx/type :input :onyx/medium :core.async :onyx/batch-size batch-size :onyx/max-peers 1 :onyx/doc "Reads segments from a core.async channel"} {:onyx/name :out :onyx/plugin :onyx.plugin.core-async/output :onyx/type :output :onyx/medium :core.async :onyx/batch-size batch-size :onyx/max-peers 1 :onyx/doc "Writes segments to a core.async channel"}] Vanilla Clojure function (defn adder [n {:keys [x] :as segment}] (assoc segment :x (+ n x)))) Plugins (I/O) seq, async, Kafka, Datomic, SQL,… parameter self-documenting
- 9. Computation entirely described with data data is code!
- 10. Everything can be run locally!
- 12. How Onyx rewired my brain
- 13. It’s not about scaling, but clean architecture
- 14. My goto architecture KafkaDB Events Onyx Onyx Onyx Persist all messages to S3 (time travel!)
- 17. Building on top of spec
- 18. Queryable data descriptions • s/registry, s/form • Build a graph (Datomic) Interact with your type system! code is data!
- 19. Case study: autogenerating materialised views Kafka Materialised views Events External data Automatic view generation • Event & attribute ontology • Manual (via spec) • Inferred • Statistical analysis (seasonality detection, outlier removal, …) Onyx Onyx Onyx
- 20. Automatic view generation 1. Walk spec registry 2. Apply rules 1. Define new view (spec) 2. Trigger Onyx job that creates the view ⤾
- 21. Code is data or data is code?
- 22. Takeouts
- 23. Onyx is production ready
- 24. Everything should be live and interactive
- 25. Computation graphs are a great way to structure data processing code
- 26. Queryable data and computation descriptions supercharge interactive development and are a great building block for automation