Stephan Ewen - Running Flink Everywhere
8 likes2,704 views
The document outlines various deployment methods for Apache Flink, including standalone clusters, YARN, Mesos, and Docker/Kubernetes, emphasizing the flexibility in resource management for different job types. It discusses dynamic scaling, session management, and the introduction of composable building blocks to cater to varying user needs and job lifecycles. Additionally, it highlights the importance of resource profiles and the ongoing developments in enhancing Flink's operational flexibility.
Stephan Ewen - Running Flink Everywhere
- 1. Running Apache Flink® Everywhere Stephan Ewen (@StephanEwen)
- 2. How is Flink deployed? 2 Standalone Cluster Embedded Service (OSGI) YARN Sessions Standalone Cloud Docker on Mesos Docker/Kubernetes YARN->Myriad->Mesos YARN Jobs A two minute search on the mailing list reveals
- 3. How is Flink deployed? 3 Standalone Cluster Embedded Service (OSGI) YARN Sessions Standalone Cloud Docker on Mesos Docker/Kubernetes YARN->Myriad->Mesos YARN Jobs A two minute search on the mailing list reveals Mesos Sessions Mesos Jobs (soon!)
- 4. How is Flink deployed? 4 Standalone Cluster Embedded Service (OSGI) YARN Sessions Standalone Cloud Docker on Mesos Docker/Kubernetes YARN->Myriad->Mesos YARN Jobs Users run mostly isolated jobs or multi-job sessions Mesos Sessions Mesos Jobs
- 5. Resource Management 5 Standalone Cluster Embedded Service (OSGI) YARN Sessions Standalone Cloud Docker on Mesos Docker/Kubernetes YARN->Myriad->Mesos YARN Jobs Resources controlled by the framework or another service. Mesos Sessions Mesos Jobs
- 6. More dimensions coming up… 6 Dynamic Resources • Number of TaskManagers changes over job lifetime "Trusted" processes • Run under superuser credential and dispatch jobs No blocking on any process type • YARN job needs to continue while ApplicationMaster is down Uniform vs. Heterogeneous Resources • Run different functions in different size containers • E.g., simple mapper in small container, heavy window operator in large container Avoiding "Job Submit" step
- 7. Reworking the Flink Process Model 7
- 8. Flink Improvement Proposal 6 8 Currently driving parties: Core Idea • Creating composable building blocks • Create different compositions for different scenarios FLIP-6 design document: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077
- 9. Recap: Current status (Standalone) 9 Standalone Flink Cluster Client (2) Submit Job JobManager TaskManager TaskManager TaskManager (3) Deploy Tasks (1) Register
- 10. Recap: Current status (YARN) 10 YARN ResourceManager YARN Cluster Client (1) Submit YARN App. (FLINK) Application Master JobManager TaskManager TaskManager TaskManager (2) Spawn AppMaster (4) Start TaskManagers (8) Deploy Tasks (3) Poll status (6) All TaskManager started (5) Register (7) Submit Job
- 11. The Building Blocks 11 • ClusterManager-specific • May live across jobs • Manages available Containers/TaskManagers • Used to acquire / release resources ResourceManager TaskManagerJobManager • Registers at ResourceManager • Gets tasks from one or more JobManagers • Single job only, started per job • Thinks in terms of "task slots" • Deploys and monitors job/task execution Dispatcher • Lives across jobs • Touch-point for job submissions • Spawns JobManagers • May spawn ResourceManager
- 12. The Building Blocks 12 ResourceManager (1) Request slots TaskManager JobManager (2) Start TaskManager (3) Register (4) Deploy Tasks
- 13. Building Flink-on-YARN 13 YARN ResourceManager YARN Cluster YARN Cluster Client (1) Submit YARN App. (JobGraph / JARs) Application Master Flink-YARN ResourceManager JobManager TaskManager TaskManager TaskManager (2) Spawn AppMaster (4) Start TaskManagers (6) Deploy Tasks (5) Register (3) Request slots
- 14. Building Flink-on-YARN Main differences from current YARN mode All containers started with JARs, config files in classpath Credentials & Secrets are strictly bound to a single job Slots are allocated/released as needed/freed • Basic building block for elastic resource usage Client disconnects after submitting job, does not need to wait until TaskManagers are up 14
- 15. Building Flink-on-YARN (separate RM) 15 YARN ResourceManager YARN Cluster YARN Cluster Client (1) Submit YARN App. (JobGraph / JARs) Application Master Flink-YARN ResourceManager JobManager TaskManager TaskManager TaskManager (2) Spawn AppMaster (4) Start TaskManagers (6) Deploy Tasks (5) Register(4) Request slots (3) Start JobMngr
- 16. Building Flink-on-YARN (w/ dispatcher) 16 YARN ResourceManager YARN Cluster YARN Cluster Client (1) HTTP POST JobGraph/Jars Application Master Flink-YARN ResourceManager JobManager TaskManager TaskManager TaskManager (3) Spawn AppMaster (5) Start TaskManagers (7) Deploy Tasks (6) Register Flink YARN Dispatcher (2) Submit YARN App. (JobGraph / JARs) (4) Request slots
- 17. Building Flink-on-Mesos 17 Mesos Master Mesos Cluster Mesos Cluster Client (1) HTTP POST JobGraph/Jars Flink Master Process Flink Mesos ResourceManager JobManager TaskManager TaskManager TaskManager (3) Start Process (and supervise) (5) Start TaskManagers (7) Deploy Tasks (6) Register (4) Request slots Flink Mesos Dispatcher (2) Allocate container for Flink master
- 18. Building Standalone 18 Standalone Cluster Flink Cluster Client (1) Submit JobGraph/Jars Flink Master Process Standalone ResourceManager TaskManager TaskManager TaskManager (7) Deploy Tasks (1) Register (3) Request slots JobManager JobManager Dispatcher (2) Start JobMngr Standby Master Process Standby Master Process
- 19. Master Container Flink Master Process Building Flink-on-Docker/K8S 19 Flink-Container ResourceManager JobManager Program Runner (2) Run & Start Worker Container TaskManager Worker Container TaskManager Worker Container TaskManager (3) Register (1) Container framework starts Master & Worker Containers (4) Deploy Tasks
- 20. Building Flink-on-Docker/K8S This is a blueprint for all setups where external services control resources and start new TaskManagers • For example AWS EC2 Flink image with auto-scaling groups Can be extended to have N equal containers, out of which one becomes master, remainder workers With upcoming dynamic-scaling feature (see Till's talk), JobManager scales job to use all available resources 20
- 22. Example: YARN session ApplicationMaster Flink-YARN ResourceManager (5) Request slots JobManager (A) JobManager (B) Dispatcher (4) Start JobMngr YARN ResourceManager YARN Cluster Client (1) Submit YARN App. (FLINK – session) TaskManager TaskManager TaskManager (2) Spawn AppMaster (6) Start TaskManagers (8, 12) Deploy Tasks (7) Register (3) Submit Job A (11) Request slots (10) Start JobMngr (9) Submit Job B 22
- 23. Sessions vs. Jobs For each Job submitted, the session will spawn its own JobManager All jobs run under session-user credentials ResourceManager holds on to containers for a certain time • Jobs quickly following one another reuse containers (quicker response) Internally, sessions build on the dispatcher component 23
- 24. Wrap-up 24
- 25. More stuff Dynamically acquire/release resources • Slots are allocated/released from Resource Manager as needed • ResourceManager allocates/releases containers over time • Strong interplay with "Dynamic Scaling" (rf. talk by Till yesterday) Resource Profiles: Containers of different size • Requests can pass a "profile" (CPU / memory / disk), or simply use "default profile" • Resource Managers YARN & Mesos can allocate respective containers 25
- 26. Wrapping it up It’s a zoo of cluster managers out there • Following different paradigms Usage patterns vary because of Flink's broad use cases • Isolated long running jobs vs. many short-lived jobs • Shared clusters vs. per-user authenticated resources We are making "jobs" and "sessions" explicit constructs Flexible building blocks, composed in various ways to accommodate different scenarios 26
- 27. Appendix 27
- 28. Flink Streaming cornerstones 28 Low latency High Throughput Well-behaved flow control (back pressure) Make more sense of data Works on real-time and historic data Performant Streaming Event Time APIs Libraries Stateful Streaming Globally consistent savepoints Exactly-once semantics for fault tolerance Windows & user-defined state Flexible windows (time, count, session, roll-your own) Complex Event Processing