Flowex: Flow-Based Programming with Elixir GenStage - Anton Mishchuk
8 likes1,609 views
This document discusses flow-based programming (FBP) and how it can be implemented using Elixir and the Flowex library. FBP defines applications as networks of independent processes that exchange data via message passing. Railway-oriented programming (ROP) is a design pattern that structures programs as a sequence of functions that pass data along. Railway FBP combines these concepts by implementing each function as an independent process. The Flowex library allows defining Elixir modules as processing pipelines and running them concurrently using GenStage to enable parallelism. It provides an easy way to build reusable and independently running process components.
Flowex: Flow-Based Programming with Elixir GenStage - Anton Mishchuk
- 1. Flowex - Flow-Based Programming with Elixir GenStage Elixir Club 5, Kyiv, January 28, 2017
- 2. - Ruby developer at Matic Insurance - Elixir fan - Author and maintainer of ESpec BDD test framework github: antonmi Hello! I am Anton Mishchuk 2
- 3. The problem ◎ There are some problems in “conventional programing”: - code is procedural and sequential - parallelism is not native - hierarchical structure of program - visualisation is more about structure ◎ We need alternative approaches for building our programs to make software better! 3
- 4. My previous FBP talk 4 http://www.slideshare.net/AntonMishchuk /flowbased-programming-with-elixir
- 5. The goal of the talk ◎ Discuss a special case of Flow-Based Programming: “Railway FBP” (R-FBP) ◎ Present Flowex library which helps to create R-FBP abstractions ◎ Show that Elixir GenStage is not only about data-processing but about software design 5
- 6. What will be about ◎ Flow-Based Programming (FBP) ◎ Railway Oriented Programming (ROP) ◎ Railway FBP ◎ Flowex library Lots of pictures! 6
- 8. “ FBP - a programming paradigm that defines applications as networks of "black box" processes, which exchange data across predefined connections by message passing J. Paul Morrison, Flow-Based Programming, 2nd Edition 8
- 9. FBP diagram A B D C IN 1 IN 1 IN 1 IN 2 IN 1 IN 2 OUT 1 OUT 2 OUT 1 OUT 1 OUT 1 9
- 10. Express a problem in terms of transforms on streams of data http://www.jpaulmorrison.com/fbp/FBPnew.ppt 10
- 11. Design pros ◎ Independent and reusable components ◎ Clean interfaces ◎ Simple to reconfigure ◎ Minimizes side-effects ◎ Designer can sit at one “station”, or can follow an item through system http://www.jpaulmorrison.com/fbp/FBPnew.ppt 11
- 14. Express a problem in terms of sequence of functions calls request |> validate_request |> get_user |> update_db_from_request |> send_email |> return_http_message 14
- 15. Define common interface for successful and error cases ◎ For example use: {:ok, "data"} or {:error, "Failure reason"} ◎ Each function returns {:ok, "data"} or {:error, "Failure reason"} ◎ If function is called with {:error, "Failure reason"} it just bypasses it 15
- 16. Define interface to bypass errors http://fsharpforfunandprofit.com/rop/ 16
- 17. So the program is a railway from input to output http://fsharpforfunandprofit.com/rop/ 17
- 18. Elixir Plug is a good example! ◎ Each “plug” function receives and returns a “connection structure” %Plug.Conn{} ◎ %Plug.Conn{} contains: assigns, cookies, halted, host, method, params, resp_body, resp_cookies, status, and many other attributes 18
- 19. 3. Railway FBP ROP + FBP 19
- 20. Let’s transform each part of “Railway” into FBP component! 20
- 21. Place each function into separate process validate send_emailupdate_db That is the idea! 21
- 22. Why Elixir? 22
- 23. GenStage ◎ Easiest way to implement a chain of communicating processes ◎ Back-pressure mechanism producer consumer Subscribe Ask Events 23
- 24. Metaprogramming ◎ Simple way for sharing functionality ◎ Ability to create expressive DSL 24
- 27. Calculate (number + 1) * 2 - 3 27
- 28. Note, the functions are cool! ◎ They have the same interface ◎ They operate with predefined struct We can join them into pipeline: 28
- 29. Let’s add some Flowex magic! 29
- 30. … and rename the module to FunPipeline 30
- 31. “use Flowex.Pipeline” ◎ Adds ‘pipe’ macro which allows to mark functions “to be placed” into separate Genstage ◎ Defines ‘start’ and ‘stop’ functions for creating and destroying “Flowex pipelines” ◎ Defines ‘run’ function to perform calculations 31
- 33. FunPipeline instance add_one minus_threemult_by_twoproducer consumer Supervisor Elixir GenStages 33
- 34. %Flowex.Pipeline{} struct ◎ module - the name of the module ◎ in_name - unique name of 'producer' ◎ out_name - unique name of 'consumer' ◎ sup_pid - pid of the pipeline supervisor 34
- 35. Run calculations via “run” Using FunPipeline.run/2 function Note, ‘a’, ‘b’ and ‘c’ were set! 35
- 36. Run calculations using Flowex.Client Using Flowex.Client module 36
- 37. How it works in details add_one minus_three mult_by_two producer consumer Flowex.Client %FunPipeline{number: 2} %Flowex.IP{ struct: %FunPipeline{number: 2}, ...} number: 2 number: 2 number: 3 number: 6 number: 3 number: 3 self() %FunPipeline{number: 3} 37
- 38. What is sync and what is async? Client Asynchronous communicationself() Synchronous communication 38
- 39. Is there no parallelism? 39
- 40. There can be a lot of clients! Client Client Client 40
- 41. Bottlenecks. What if there is very slow process in the pipeline? 41
- 42. Just small changes in the code 42
- 43. And you get this add_one minus_three mult_by_twoproducer consumer mult_by_two minus_three mult_by_two 43
- 44. Reusable components with “module pipelines” 44
- 45. Module must implement just two functions Like Elixir Plug module 45
- 47. Conclusion 47
- 48. Why it is cool! 48
- 49. It is easy to understand ◎ Pipelines explicitly define a structure of data will be processed ◎ Pipelines explicitly define a way the data will come ◎ Pipelines explicitly define a parallel executors structure 49
- 50. It is easy to maintain and reuse ◎ There is a predefined set of working processes (components) in a program ◎ Each component is isolated ◎ Pipelines can (and should) reuse components 50
- 51. It is about controlled parallelism ◎ One can controll number of clients supplying data to a pipeline ◎ One can control the number of processes available for each component 51
- 52. Thank you! Questions? 52 You can star the project here: https://github.com/antonmi/flowex