Virtual Flink Forward 2020: Implement Reliable, Isolated & Unified Job Submission - Zili Chen
Download as PPTX, PDF0 likes154 views
The document discusses implementing reliable, isolated, and unified job submission for a distributed stream processing platform. It proposes: 1) Defining job submission and execution as atomic by requiring the job graph to be persisted before a job is considered submitted, and the job status to be set to DONE before a job is considered completed. 2) Compiling jobs in isolation on the cluster side by packaging user programs and dependencies and executing them in isolated containers to avoid bottlenecks and security risks at the client. 3) Exposing a three-layer unified client interface for deployment, cluster, and job management to provide a programmatic submission approach.
Virtual Flink Forward 2020: Implement Reliable, Isolated & Unified Job Submission - Zili Chen
- 1. Implement Reliable, Isolated & Unified Job Submission Zili Chen – Software Engineer, Flink Committer
- 2. 2 • Background • Reliability • Isolation • Unification Outline
- 3. 3 • Background • Reliability • Isolation • Unification Outline
- 5. 5 Background1 Oceanus: Tencent Real-Time Compute Platform Data Distribution Monitoring Real-Time BI Online Learning 1200 Jobs 35 Trillion Messages in a day 4.5 PB Data in a day
- 6. 6 Background1 The Problem Job submission & execution is unreliable. Client becomes bottleneck & suffers security risks. Flexible programmatic interface is missing. Start with the workflow of job submission!
- 7. 7 Workflow1 Cluster Deployment(Session Mode) Deployer client side cluster side Resource Manager (YARN) 2. Launch Application Job Manager Container Dispatcher
- 8. 8 Workflow1 Job Submission(Session Mode) Client Dispatcher client side cluster side JobGraph Store JobRegistry 1. Generate JobGraph
- 9. 9 Workflow1 Job Submission(Session Mode) Client Dispatcher client side cluster side JobGraph Store JobRegistry 3. Check JobStatus 4. Persist JobGraph (otherwise)
- 10. 10 Workflow1 Job Submission(Session Mode) Client Dispatcher client side cluster side JobGraph Store JobRegistry 5. Start Job Manager Job Manager 6. Report Started
- 11. 11 Workflow1 Job Execution(Session Mode) Client Dispatcher client side cluster side JobGraph Store JobRegistry Job Manager 1. Check JobStatus 2. Finished By Others 2. Set JobStatus RUNNING
- 12. 12 Workflow1 Job Execution(Session Mode) Client Dispatcher client side cluster side JobGraph Store JobRegistry Job Manager 3. Set JobStatus DONE 4. Finished
- 13. 13 Workflow1 Job Execution(Session Mode) Client Dispatcher client side cluster side JobGraph Store JobRegistry 5. Remove JobGraph 6. Clear JobStatus
- 14. 14 Workflow1 Cluster Deployment(Job Mode) Deployer client side cluster side Resource Manager (YARN) 3. Launch Application Job Manager Container Dispatcher 1. Generate JobGraph
- 15. 15 Workflow1 Job Execution(Job Mode) cluster side Dispatcher JobGraph Store JobRegistry 2. Start Job Manager Job Manager 1. Recover JobGraph
- 16. 16 Workflow1 Job Execution(Job Mode) cluster side Dispatcher JobGraph Store JobRegistry Job Manager 3. Check JobStatus 4. Finished By Others 4. Set JobStatus RUNNING
- 17. 17 Workflow1 Job Execution(Job Mode) cluster side Dispatcher JobGraph Store JobRegistry Job Manager 5. Set JobStatus DONE 6. Finished
- 18. 18 Workflow1 Job Execution(Job Mode) cluster side Dispatcher JobGraph Store JobRegistry 7. Remove JobGraph 8. Clear JobStatus
- 19. 19 • Background • Reliability • Isolation • Unification Outline
- 20. Reliability 20 2
- 21. 21 Reliability2 What is reliability? Submited exactly once! Executed exactly once!
- 22. 22 Reliability2 Failure Case: Job Manager failed to start in time 22 Client Dispatcher client side cluster side JobGraph Store JobRegistry Job Manager Starting Job Manager
- 23. 23 Reliability2 Failure Case: Job Manager crashed after job finished 23 cluster side Dispatcher JobGraph Store JobRegistry Job Manager Set JobStatus DONE
- 24. 24 Reliability2 Failure Case: Job Manager crashed after job finished 24 cluster side Dispatcher JobGraph Store JobRegistry Job Manager Crash Finished Clear JobStatus
- 25. 25 Reliability2 Failure Case: Standby starts after JobStatus cleared 25 cluster side Dispatcher JobGraph Store JobRegistry Check JobStatus
- 26. 26 Reliability2 Failure Case: Standby starts after JobStatus cleared 26 cluster side Dispatcher JobGraph Store JobRegistry Job Manager Start Job Manager Execute twice! (FLINK-11813)
- 27. 27 Reliability2 What to define job submitted & executed 27 JobRegistry JobGraphStore (internal) JobManagerRunnerFuture Handled in Dispatcher Modified by Dispatcher Modified by Dispatcher & Job Manager Many factors to define job status! Goal: Achieve atomic job submission and execution
- 28. 28 Reliability2 How to achieve atomic job submission 28 • What is the sign of a successful submission? JobGraph persisted in JobGraphStore
- 29. 29 Reliability2 Successful job submission defined by JobGraph 29 Client Dispatcher client side cluster side JobGraph Store JobRegistry 4. Persist JobGraph
- 30. 30 Reliability2 How to achieve atomic job submission 30 • What is the sign of a successful submission? JobGraph persisted in JobGraphStore • What is the sign of a successful execution? DONE in JobRegistry(only Dispatcher modifies it; not cleared after written)
- 31. 31 Reliability2 Successful job execution defined by DONE 31 Dispatcher JobGraph Store JobRegistry Job Manager 6. Set JobStatus RUNNING cluster side 7.Start Job Manager
- 32. 32 Reliability2 Successful job execution defined by DONE 32 Dispatcher JobGraph Store JobRegistry Job Manager 8. Finished 9. Set JobStatus DONE cluster side
- 33. 33 • Background • Reliability • Isolation • Unification Outline
- 34. Isolation 34 3
- 35. 35 Isolation3 Where is JobGraph generated? 35
- 36. 36 Isolation3 Job Submission(Session Mode) Client Dispatcher client side cluster side JobGraph Store JobRegistry Generate JobGraph
- 37. 37 Isolation3 Cluster Deployment(Job Mode) Deployer Resource Manager (YARN) client side cluster side Generate JobGraph
- 38. 38 Isolation3 Where is JobGraph generated? 38 Compilation always happens at client side!
- 39. 39 Isolation3 Drawbacks 39 Require user code dependencies Execute arbitrary user code Bottleneck at client Goal: Achieve isolated job compilation
- 40. 40 Isolation3 Compilation on cluster side Application Mode
- 41. 41 Isolation3 Application Mode: Package User Program 41 ShipFiles MainClass Parallelism SavepointSettings Arguments ... Deployer client side CommandLine Interface Pass Program Metadata
- 42. 42 Isolation3 Application Mode: Package User Program 42 Deployer client side CommandLine Interface MainClass Parallelism SavepointSettings Arguments ... Package User Program
- 43. 43 Isolation3 Application Mode: Prepare Dependencies 43 Deployer Distributed Storage client side cluster side Upload Dependencies
- 44. 44 Isolation3 Application Mode: Prepare Dependencies 44 Deployer Distributed Storage client side cluster side Return URIs ShipFiles
- 45. Distributed Storage 45 Isolation3 Application Mode: Prepare Dependencies 45 Deployer client side cluster side Configure Dependencies URIs SharedLibs ShipFiles
- 46. 46 Isolation client side cluster side Resource Manager (YARN) 3 Application Mode: Cluster Deployment Deployer
- 47. 47 Isolation client side cluster side Resource Manager (YARN) 3 Application Mode: Cluster Deployment Launch Application Job Manager Container Deployer
- 48. 48 Isolation3 Application Mode: Download Dependencies 48 cluster side Job Manager ContainerDistributed Storage SharedLibs ShipFiles Localize(YARN) InitContainer(K8s)
- 49. Job Manager Container 49 Isolation3 Application Mode: Execute in Isolation 49 cluster side DispatcherClient Submit JobGraph Generate JobGraph
- 50. 50 Isolation3 Deployment: Recap 50 Session Mode Job Mode User Program execute as is abort on execute Client Perspective cluster deployed & job submitted cluster deployed with bundled job High Availability configured JobGraphStore special JobGraphStore Application Mode execute as is cluster deployed & (local) job submitted configured JobGraphStore
- 51. 51 • Background • Reliability • Isolation • Unification Outline
- 52. Unification 52 4
- 53. 53 Unification4 Client Interface: The Problem Flink does not provide public & stable client interface Various customized submission requires programmatic interface Goal: Expose unified layered client interface!
- 54. 54 Unification4 Client Interface: Three-Layers Architecture Deployment: Deployer Cluster: Cluster Client Job: Job Client Target: Kubernetes, YARN, etc. Target: Flink Cluster Target: Job Manager
- 55. 55 Unification4 Client Interface: Deployer Target: YARN, Kubernetes Deployer Resource Manager Submit Application: ApplicationDescriptor → ApplicationID Shutdown Application: ApplicationID → Acknowledge Get Application Status: ApplicationID → ApplicationStatus
- 56. 56 Unification4 Client Interface: Cluster Client Submit Job: JobGraph → JobID Shutdown Cluster: () → Acknowledge Get Cluster Overview: () → ClusterOverview Target: Flink Cluster Cluster Client Dispatcher
- 57. 57 Unification4 Client Interface: Job Client Get Job Status: () → JobStatus Trigger Savepoint Savepoint Path → Savepoint Path Cancel Job: () → Acknowledge Target: (Flink) Job Manager Job Client Job Manager
- 58. 58 Unification4 User Case: Platform Deployer Resource Manager Submit Session Application Session Mode Cluster Client Submit Multiple JobGraphs Dispatcher
- 59. 59 Unification4 User Case: Platform Deployer Resource Manager Submit Multiple Applications Application Mode
- 60. 60 Unification4 User Case: End-User Deployer Resource Manager Submit Application env.executeAsync(): JobClient Cluster Client Submit JobGraph Dispatcher (locally in Application Mode) (remotely in Session Mode)
- 61. 61 Unification4 User Case: End-User env.executeAsync(): JobClient Job Client Job Manager 1. Monitor Job Status 2. Trigger Savepoint 3. Manipulate Job
- 62. Thanks!