Doing data science with Clojure
2 likes1,179 views
The document discusses data science practices using Clojure, emphasizing design constraints, the importance of communication, and the advantages of using programmable environments and notebooks. It critiques traditional data frames and promotes a function-oriented approach through the author's library, Huri, which enhances composability and speed in data processing. The takeaways highlight the significance of sharing results effectively and leveraging Clojure's strengths in structure manipulation.
![REPL vs. notebook+[Ephemeral] [Spital grouping]](https://image.slidesharecdn.com/clj-for-datascience-curry-on-160718155848/85/Doing-data-science-with-Clojure-16-320.jpg)
Doing data science with Clojure
- 1. Doing data science with Clojure @sbelak simon@goopti.com Curry On Rome, 2016
- 3. âł Design constrains âł The environment âł notebooks vs. REPL âł programmable environments âł The tools âł design decisions behind Huri (my data science library) âł data frame considered harmful âł encoding computation into structure âł composability âł feedback loops âł Expanding the ecosystem with mini compilers (to ggplot, scipy, âŚ)
- 7. Kafka PostgreSQL ElasticSearch frontend actions orderbook changes monitoring telemetry ďŹight changes Intercom ⌠s3 Intercom
- 8. Automatic views ⢠Event & attribute ontology ⢠Manual ⢠Inferred ⢠Seasonality detection
- 9. Data science: ⨠the process (aka itâs about communication, stupid!)
- 10. The analytics chasm Ideal. Almost real-time, can be done during brainstorming without disrupting ďŹow < 2min < 20min project squeeze in somewhere in the day fail roadmap⨠ahoy!
- 11. Think in distributions, not numbers
- 12. No throwaways
- 13. Sharing results ⢠Have one canonical version that is always current. ⢠Concentrate discussion in one place and make it searchable and persistent. ⢠Include methodology (=code).
- 14. The environment
- 16. REPL vs. notebook+[Ephemeral] [Spital grouping]
- 19. Code hidden, but can be expanded Questions, comments, & annotations Shareable Periodically re-run to keep it fresh #alderaan #sales #growth discoverability
- 21. The power of sharing runtime
- 22. Wishlist/TODO ⢠Better editor (shaunlebron.github.io/parinfer/ ?) ⢠Embedded REPL ⢠Better exception reporting ⢠Browsable data structuresâ¨
- 23. The tools
- 25. Data frame considered harmful ⢠Data frame (=table) conďŹates representation and abstraction ⢠Clojure excels in structure manipulation/encoding
- 26. github.com/sbelak/huri ⢠No data structures, just functions over collections ⢠Composable (even DSLs â no macros!) ⢠Reasonably fast (transducers <3) ⢠Do-what-I-mean (auto-sort, liberal with inputs, âŚ) ⢠Minimal buy-in
- 27. composable data structure based DSLs ->> and partial friendly Support reaching into nested structures everywhere vanilla vector of maps interoperability Provide curried versions where possible
- 28. Composability is key to quick iterating ⢠Curried versions where possible ⢠->> and partial friendly ⢠Side beneďŹt: consistent API ⢠Generalised accessors (reaching into complex structures everywhere via comp) function map key âvirtualâ structure
- 29. âThis is possibly Clojureâs most important property: the syntax expresses the codeâs semantic layers. An experienced reader of Clojure can skip over most of the code and have a lossless understanding of its high- level intent.â â Z. Tellman, Elements of Clojure
- 30. On feedback
- 31. Catching errors early more context easier debugging faster iterating
- 32. clojure.spec => Should have been a keyword->fn map
- 33. <3 Bret Victor
- 34. What about machine learning? farm it out to sklearn
- 35. Mini compilers for DSLs targeting a speciďŹc library in another language
- 36. huri.plot ⢠DSL that compiles to ggplot2 ⢠Targets Gorilla REPL ⢠Follows the rest of Huriâs design philosophy ⢠bar chart, scatter plot, line chart, box & violin plot, heatmap, histogram
- 38. Takeouts ⢠Speed-of-answer matters ⢠Data science is about communication ⢠We donât have to reinvent every wheel in Clojure ⢠Clojure is fantastic at structure manipulation, play to its strengths ⢠Blurring the line between environment and work is a powerful idea