Apache Apex as YARN Application
- 1. Apache Apex as YARN Application Chinmay Kolhatkar (chinmay@apache.org) Jan 6, 2016 Apache Apex Meetup
- 2. Agenda • Directed Acyclic Graph • Apex as a YARN Application • Application Components of Apex • Lifecycle of Apex as a YARN Application Apache Apex Meetup
- 3. Directed Acyclic Graph (DAG) • Defines compute stages of streaming application • Defines tuple flow across Operators via Stream Compute 1 Compute 4 Compute 3 Compute 2 Apache Apex Meetup
- 4. DAG Components • Tuple ● Atomic data that flows over a stream • Operator ● Basic compute unit per tuple • Stream ● Connector abstraction between operators ● Tuples flow over this Operator1 Operator2tuple3 tuple2 tuple1 Stream Apache Apex Meetup
- 5. DAG Types • Logical Plan ● Logical representation of computation ● Defines operators, streams and dataflow • Physical Plan ● Deployable plan on cluster ● Contains partition information of operators ● Has ready-to-deploy serialized operator instances O1 O2 O3 O4 O5 O2P2 O3 O4 O5 O2P3 O2P1 U O1P1 O1P3 O1P2 Logical Plan Physical Plan Apache Apex Meetup
- 6. Apex as YARN application Node ResourceManager (AsM + Scheduler) NM Node NM Node NM YarnClientClientRM Protocol AppMaster AMRM Protocol YarnContainer YarnContainer YarnContainer ContainerManager Protocol ContainerManager Protocol StrAM (AppMaster) AMRM Protocol ClientRM Protocol YarnContainer StrAMChild O1 O2 YarnContainer StrAMChild O3 ContainerManager Protocol DTCLI StrAMClient YarnClient Apache Apex Meetup
- 7. Application Components of Apex - StrAMClient • Part of dtcli client interface • Invoked by “launch” command of dtcli • Tasks: ● Copy required the application package files into HDFS ● Validate Logical Plan ● Serialize Logical plan to HDFS ● Launch Application Master i.e. StrAM Apache Apex Meetup
- 8. Application Components of Apex - StrAM • Streaming Application Master • Started by StrAMClient on a YarnContainer • Tasks: ● Convert logical plan to physical plan ● Serialize operators to HDFS ● Request for resources to ResourceManager ● Start StrAMChild in YarnContainer(s) ● Monitor StrAMChild using ContainerManager protocol ● Generate Application statistics ● Host results on WebService (dtManage) ● Fault Tolerance ● Checkpointing/Committing Application States ● Support Security ● Shutdown Application Apache Apex Meetup
- 9. Application Components of Apex - StrAMChild • Deployed on YarnContainer • Started by NodeManager as instructed by StrAM • Instance of StreamingContainer • Contains Operators (compute-related) • Contains BufferServer (stream-related) • Tasks: ● Regularly send heartbeat to StrAM ● Execute commands from StrAM ● Shutdown or Kill self if instructed ● Manage lifecycle of an Operator ● Network communication using BufferServer Apache Apex Meetup
- 10. Lifecycle of Apex/YARN Application - Start Node ResourceManager (AsM + Scheduler) NM Node NM Node NM DTCLI/ StrAMClient (YarnClient)ClientRMProtocol 1) Access cluster information HDFS 2) Copies file to HDFS3) Submit Application to RM StrAM (AppMaster) AMRMProtocol 4) StrAM Registers with RM 5) StrAM sends heartbeats regularly 6) StrAM request containers with specifications 7) StrAMChild reads serialized operator from HDFS 8) StrAMChild starts operator lifecycle ContainerManager Protocol YarnContainer StrAMChild O2 O1 YarnContainer StrAMChild O3 YarnContainer StrAMChild O4 ContainerManager Protocol Apache Apex Meetup
- 11. Lifecycle of Apex/YARN Application - Running Node ResourceManager (AsM + Scheduler) NM Node NM Node NM DTCLI/ StrAMClient (YarnClient)ClientRMProtocol 4) Query Status of application HDFS StrAM (AppMaster) AMRMProtocol 3) StrAM send regular heartbeats to RM 1) StrAMChild send heartbeats 2) StrAMChild sends operator data ContainerManager Protocol YarnContainer StrAMChild O2 O1 YarnContainer StrAMChild O3 YarnContainer StrAMChild O4 ContainerManager Protocol Apache Apex Meetup
- 12. Lifecycle of Apex/YARN Application - Shutdown Node ResourceManager (AsM + Scheduler) NM Node NM Node NM DTCLI/ StrAMClient (YarnClient)ClientRMProtocol HDFS StrAM (AppMaster) AMRMProtocol ContainerManager Protocol YarnContainer StrAMChild O2 O1 YarnContainer StrAMChild O3 YarnContainer StrAMChild O4 ContainerManager Protocol 1) Connect on WebService 2) Send Shutdown command to StrAM REST API 3) Send shutdown signal to StrAMChild 4) StrAMChild finishes Operator lifecycle 5) Check if all containers are freed 6) StrAM unregisters itself 7) StrAM exits 8) Check if application has shutdown Apache Apex Meetup
- 13. Lifecycle of Apex/YARN Application - Kill Node ResourceManager (AsM + Scheduler) NM Node NM Node NM DTCLI/ StrAMClient (YarnClient) ClientRMProtocol HDFS StrAM (AppMaster) AMRMProtocol ContainerManager Protocol YarnContainer StrAMChild O2 O1 YarnContainer StrAMChild O3 YarnContainer StrAMChild O4 ContainerManager Protocol 1) Send kill-app command to Yarn 2) RM kills all containers Apache Apex Meetup
- 14. Summary • Apex enables YARN to be used for Streaming Applications • Apex takes care of YARN specific work • User can focus on business logic defined in Operators Apache Apex Meetup
- 16. Resources • Apache Apex page ● http://apex.incubator.apache.org ● http://apex.incubator.apache.org/community.html • Mailing list ● dev@apex.incubator.apache.org ● users@apex.incubator.apache.org • Repository ● https://github.com/apache/incubator-apex-core ● https://github.com/apache/incubator-apex-malhar • Issue Tracking ● https://issues.apache.org/jira/browse/APEXCORE/ ● https://issues.apache.org/jira/browse/APEXMALHAR/ Apache Apex Meetup