Building Your First Apache Apex (Next Gen Big Data/Hadoop) Application
Download as PPTX, PDF1 like621 views
This document provides an overview of building a first Apache Apex application. It describes the main concepts of an Apex application including operators that implement interfaces to process streaming data within windows. The document outlines a "Sorted Word Count" application that uses various operators like LineReader, WordReader, WindowWordCount, and FileWordCount. It also demonstrates wiring these operators together in a directed acyclic graph and running the application to process streaming data.
Building Your First Apache Apex (Next Gen Big Data/Hadoop) Application
- 1. © 2015 DataTorrent Munagala V. Ramanath (“Ram”) <ram@datatorrent.com> Dec 15th, 2015 Building your First Apache Apex Application
- 2. © 2015 DataTorrent Outline Main concepts of an Apex Application. Brief description of the "Sorted Word Count" application. Hands on demonstration of cloning the source repository and building Apex source code. Hands on demonstration of creating a new application. Running the application. Code walk-through. Questions
- 3. © 2015 DataTorrent Main Concepts Applications are built from Operators which implement the Operator interface; each operator has input/output ports which are connected by streams to form a directed acyclic graph (DAG). A BaseOperator class is provided which provides empty implementations of all the required methods. Within an operator, define necessary input and output ports typically using the DefaultInputPort and DefaultOutputPort classes. The Application class implements the StreamingApplication interface; need only implement populateDAG() method which wires the operators together. Applications process data within time-based windows, typically 0.5s.
- 4. © 2015 DataTorrent The Sorted Word Count Application The following operators are involved: LineReader: reads file dropped into input directory and outputs lines (on its output port). WordReader: splits each line into words using a regex. WindowWordCount: compute and emit word frequencies for all words in lines processed in current window. FileWordCount: accumulates all word counts for current file and emits final sorted list when EOF is reached. WordCountWriter: writes list to output file in output directory.
- 5. © 2015 DataTorrent The DAG
- 6. © 2015 DataTorrent Resources 6 Apache Apex Community Page Apache Apex LinkedIn Group