Eron Wright - Introducing Flink on Mesos
2 likes661 views
This document introduces Flink on Mesos, including the Flink Master Process and Dispatcher framework. The Flink Master Process acts as both an Application Master and Mesos framework, hosting components like the JobManager and ResourceManager. The Dispatcher is a highly available service for launching Flink clusters through REST. It uploads configuration and files to the Flink Master Process, which then launches TaskManagers. Future directions include dynamic scaling, improved dispatcher capabilities, and Docker/Mesos disk support.
Eron Wright - Introducing Flink on Mesos
- 1. Introducing Flink on Mesos Eron Wright – eron.wright@emc.com DELL EMC @eronwright
- 2. 2 of 15 What is Apache Mesos? • A popular cluster manager (similar to YARN) • MakesavailableCPU, memory, & diskresources • Uniquecapabilitiesforstorageservices • Emerging asa foundationfordata-centric,convergedinfrastructure • Provides a programming model for using cluster resources • A Mesosprogram is calleda “framework” • Packaged into an open-source distribution called DCOS • Prescribesbestpracticesrelatedto Mesosframeworks, relatedservices,etc.
- 3. 3 of 15 Why Flink on Mesos? • Flink works best on a cluster manager – Easy to scale each job independently – Externalize scheduling logic (fairness, quota, …) – Good job isolation • Flink can benefit from unique Mesos capabilities – Disk resources – Dynamic resource management – Unique management features (e.g. inverse offers for controlled downscaling & maintenance)
- 4. Demo
- 6. 6 of 15 Introduction Flink Master Process • The Flink Master Process is: – The “Application Master” for a single Flink cluster – A Mesos framework! • Hosts numerous components: – Job Manager – Resource Manager (acts as Mesos scheduler) – Artifact Server (HTTP server for Mesos fetcher) • Responsible for TM scaling and recovery – Handles JobManager scale change requests – Stores task state in ZooKeeper host1host2 Master JM RM HTTPD TM TM Mesos
- 7. 7 of 15 How it Works Flink Master Process • Offer handling: – Uses Netflix Fenzo as an optimizer – Gathers offers until all tasks launched • Recovery: – Stores intentional state in ZooKeeper – Master uses leader election – Mesos allows some time for recovery before killing tasks • Monitoring: – Detects task failure; launches replacement automatically. host1host2 Master TM TM 4. Launch Mesos 2. Resource Offers 1. Register 5. Fetch (HTTP) 6. Status update 3. Optimize
- 8. 8 of 15 Configuration Flink Master Process (Con’t) • Framework Info – mesos.resourcemanager.framework.secret – mesos.resourcemanager.framework.principal – mesos.resourcemanager.framework.role • Mesos Master Info – mesos.master: (IP address or ZK lookup info) – mesos.failover-timeout Note: no port configuration is necessary; Mesos automatically assigns ports.
- 9. Dispatcher
- 10. 10 of 15 Introduction Dispatcher • A highly-available service for launching Flink clusters. • A Mesos framework! • Accessed via REST by the CLI • DCOS compatibility: – HTTP-based – Accessible via the Admin Router – (future) JWT authentication • Aligned with FLIP-6 host1 1D 1C 1B 1A host2 2D 2C 2B 2A host3 3D 3C 3B 3A host4 4D 4C 4B 4A Dispatcher Master TM TM TMTM Master CLI TM Mesos
- 11. 11 of 15 Framework Hierarchy Dispatcher (Con’t) • Nesting of frameworks is a common Mesos pattern. Here, Marathon launches the dispatcher, which launches the Flink Master Process, etc. • Architecturally, it avoids a dependency on the Marathon API. For example, Aurora could be used here in place of Marathon. Dispatcher Master Maratho n TM (Task) (Task) (Task)
- 12. 12 of 15 Launching a Session Dispatcher (Con’t) • Use: mesos-session.sh • CLI uploads files to dispatcher via HTTP – Flink Configuration – Supplemental files (--ship) – Keytabs – Certificates • Dispatcher adds additional elements: – Configuration › ZooKeeper Namespace – Flink JAR – … host1 1D 1C 1B 1A host2 2D 2C 2B 2A host3 3D 3C 3B 3A host4 4D 4C 4B 4A Dispatcher Master TM TM CLI HTTP(S) TM HTTP(S) Mesos
- 13. 13 of 15 Dispatcher Deployment Modes Dispatcher (Con’t) • Dispatcher is usable in two ways • Remote Mode: – Recommended for detached execution • Local Mode: – Recommended for simple, interactive sessions (e.g. flink shell) 3C 3B 3A 4C 4B 4A Dispatcher Master Master CLI HTTP(S) 3C 3B 3A 4C 4B 4A Master CLI + Dispatcher Local Mode Remote Mode
- 14. Summary
- 15. 15 of 15 Future Directions • Dynamic Scaling – Add/remove Task Managers in response to scale changes over a job’s lifetime – Support Mesos maintenance procedures (e.g. inverse offers) • Dispatcher Evolution (FLIP-6) – Generalize to support all deployment scenarios, unified CLI – Provide a centralized Web UI (incl. job history) – Authentication Support (e.g. OAuth 2.0) • Docker Image Support – Tracking the “Mesos unified containerizer” • Mesos Disk Support – Allocate multiple disks for Task Manager temp space – Scale up the I/O
- 16. 16 of 15 Project Status • Targeted for: Flink 1.2 • Contributors: – Eron Wright (Dell EMC) – Maximilian Michels (data Artisans) • Design Doc: – Mesos Integration on Google Docs • JIRAs: – FLINK-1984 – Integrate Flink with Apache Mesos • Code: – https://github.com/EronWright/flink/tree/feature-FLINK-1984-T2