Building YARN Applications
Download as PPTX, PDF0 likes376 views
This document serves as a hands-on guide for creating and executing Apache Apex applications, specifically focusing on word filtering and counting operations. It outlines prerequisites, application structure, and provides step-by-step assignments to build a functional word count application. Additional resources and links for further learning and community engagement are also included.
Building YARN Applications
- 1. © 2015 DataTorrent Akshay Gore, Bhupesh Chawda DataTorrent Apex Hands-on Lab - Into the code! Getting started with your first Apex Application!
- 2. © 2015 DataTorrent Operators • Input Adaptor Vs Generic Operators ? • What are streams? • What are ports?
- 3. © 2015 DataTorrent Apex Operator Lifecycle
- 4. © 2015 DataTorrent Apex Streaming Application public class Application implements StreamingApplication { populateDAG(DAG dag, Configuration conf) { // Add Operators to dag - dag.addOperator(args) // Add Streams between operators - dag.addStream(args) // Additional config + Hints to YARN - Optional } }
- 5. © 2015 DataTorrent Apex Application - FilterWords Apex Application DAG • Problem statement - Filter words in the file ᵒ Read a file located on HDFS ᵒ Split each line into words, check if it is not one of the forbidden words and write it down to HDFS HDFS Lines Filtered Words HDFS
- 6. © 2015 DataTorrent FilterWords Application DAG Reader Tokenize Processor Writter Input Operator (Adapter) Output Operator (Adapter) Generic Operators HDFS HDFS Lines Words Filtered Words
- 7. © 2015 DataTorrent Prerequisites • JAVA 1.7 or above • Maven 3.0 or above • Apache Apex projects: ᵒ Apache Apex Core: core platform, engine ᵒ Apache Apex Malhar: operators library • Hadoop cluster in running state • Your favourite IDE - Eclipse / vi
- 8. © 2015 DataTorrent Demo time! • Apex application structure • Application code walk through • How to execute the application • Assignment
- 9. © 2015 DataTorrent Assignment - WordCount Apex Application DAG • Problem statement - Count occurrences of words in a file ᵒ Read a file located on HDFS ᵒ Emit count at the end of the every window and writes into HDFS HDFS Lines <Word, Count> HDFS
- 10. © 2015 DataTorrent Assignment - Word Count Application DAG Reader Tokenize Counter Output HDFS HDFS Lines Words <Word, count>
- 11. © 2015 DataTorrent Assignment - What you need to do Reader Tokenizer Processor Writer String String String Line Words Words’ Counter Writer Map {Word: Count} Assignment
- 12. © 2015 DataTorrent Assignment - Hints • Create copy of Processor.java. Name it Counter.java • Modify Counter.java as follows: ᵒ Define a data structure which can hold counts for words ᵒ Process method of input port must count the occurrences ᵒ Clear the counts in beginWindow() call ᵒ Emit the counts in endWindow() call
- 13. © 2015 DataTorrent Solution - Changes to Counter.java • Need to define a data structure which can hold counts for words private HashMap<String, Integer> counts = new HashMap<>(); • Process method of input port must count the occurrences if(counts.containsKey(refinedWord)) { counts.put(refinedWord, counts.get(refinedWord) + 1); } else { counts.put(refinedWord, 1); } ● Clear the counts in beginWindow call counts.clear(); ● Emit the counts in endWindow call output.emit(counts.toString()); ● Run Application Test
- 14. © 2015 DataTorrent Assignment - Are we done yet? • Change the DAG ᵒ Replace Processor operator with the newly created operator - Counter
- 15. © 2015 DataTorrent Assignment - Slight change • We are emitting a Map. However it is still a string. ᵒ Change type of output port of Counter to type Map ᵒ Change type of input port of Writer to Map ᵒ Make appropriate changes to Writer to read a Map and write in a format such that each line belongs to a single word.
- 16. © 2015 DataTorrent Assignment - Final change • Change the code such that each count is the overall count, not just for each window?
- 17. © 2015 DataTorrent Summary - Recap • Writing Apache Apex operators • Chaining the operators into an Apache Apex application • Executing the application on the Apache Apex platform
- 18. © 2015 DataTorrent Where to go from here? Apache Apex Documentation - http://apex.incubator.apache.org/docs.html Apache Apex Core Git - https://github.com/apache/incubator-apex-core Apache Apex Malhar Git - https://github.com/apache/incubator-apex-malhar Join Users Mailing List - users-subscribe@apex.incubator.apache.org Join Dev Mailing List - dev-subscribe@apex.incubator.apache.org Send queries to Users Mailing List - users@apex.incubator.apache.org Send queries to Dev Mailing List - dev@apex.incubator.apache.org
- 19. © 2015 DataTorrent Thank You
Editor's Notes
- #3: Operators are basic compute units. Operators process each incoming tuple and emit zero or more tuples on output ports as per the business logic. Input Adapter - This is one of the starting points in the application DAG and is responsible for getting tuples from an external system. At the same time, such data may also be generated by the operator itself, without interacting with the outside world Generic Operator - This type of operator accepts input tuples from the previous operators and passes them on to the following operators in the DAG Output Adapter - This is one of the ending points in the application DAG and is responsible for writing the data out to some external system.
- #4: An operator passes through various stages during its lifetime. Each stage is an API call that the Streaming Application Master makes for an operator. setup() call initializes the operator and prepares itself to start processing tuples. beginWindow() call marks the beginning of an application window and allows for any processing to be done before a window starts process() call belongs to the InputPort and gets triggered when any tuple arrives at the Input port of the operator emitTuples() call is used by Input adapters to emit any tuples that are fetched from the external systems endWindow() call marks the end of the window and allows for any processing to be done after the window ends teardown() call is used for gracefully shutting down the operator and releasing any resources held by the operator
- #5: Skeleton for Apex application
- #8: For application development or for functional testing, hadoop cluster or services as it can run in the local file system as single process with multiple threads. A hadoop cluster (distributed cluster) is recommended for benchmarking and production testing. For single node cluster, throughput will not be high as multi node cluster, memory constraints