3. Apache Tez Introducation - Apache Kylin Meetup @Shanghai
1 like983 views
Jeff Zhang is a committer for Apache Tez and Hortonworks employee. He gave a presentation on Tez introducing it as a next generation execution engine for Hadoop. Tez uses a directed acyclic graph (DAG) to express computations and optimize jobs by executing multiple tasks simultaneously. It allows data sharing between jobs and dynamic resource allocation.
3. Apache Tez Introducation - Apache Kylin Meetup @Shanghai
- 1. © Hortonworks Inc. 2015 Page 1 Apache Tez -‐ Next Generation of execution engine upon hadoop Jeff Zhang (@zjffdu)
- 2. © Hortonworks Inc. 2015 Who’s this guy • Start use pig from 2009. Become Pig committer from Nov 2009 • Join Hortonworks in 2014. • Tez Committer from Oct 2014
- 3. © Hortonworks Inc. 2015 Agenda •Tez Introduction •Tez Feature Deep Dive •Tez Status & Roadmap
- 4. © Hortonworks Inc. 2015 I/O Synchronization Barrier I/O Synchronization Barrier Job 1 ( Join a & b ) Job 3 ( Group by of c ) Job 2 (Group by of a Join b) Job 4 (Join of S & R ) Hive -‐ MR Example of MR versus Tez Page 4 Single Job Hive -‐ Tez Join a & b Group by of a Join b Group by of c Job 4 (Join of S & R )
- 5. © Hortonworks Inc. 2015 Tez – Introduction Page 5 • Distributed execution framework targeted towards data-‐processing applications. • Based on expressing a computation as a dataflow graph (DAG). • Highly customizable to meet a broad spectrum of use cases. • Built on top of YARN – the resource management framework for Hadoop. • Open source Apache project and Apache licensed.
- 6. © Hortonworks Inc. 2015 What is DAG & Why DAG Projection Filter GroupBy … Join Union Intersect … Split … • Directed Acyclic Graph • Any complicated DAG can been composed of the following 3 basic paradigm – Sequential – Merge – Divide
- 7. © Hortonworks Inc. 2015 Expressing DAG in Tez API • DAG API (Logic View) – Allowuser to build DAG – Topological structure of the data computation flow • Runtime API (Runtime View) – Application logic of each computation unit (vertex) – How to move/read/write data between vertices
- 8. © Hortonworks Inc. 2015 DAG API (Logic View) Page 8 • Vertex (Processor, Parallelism, Resource, etc…) • Edge (EdgeProperty) – DataMovement – Scatter Gather (Join, GroupBy … ) – Broadcast ( Pig Replicated Join / Hive Broadcast Join ) – One-‐to-‐One ( Pig Order by ) – Custom
- 9. © Hortonworks Inc. 2015 Runtime API (Runtime View) Page 9 ProcessorInput Output • Input – Through which processor receives data on an edge – Vertex can have multiple inputs • Processor – Application Logic (One vertex one processor) – Consume the inputs and produce the outputs • Output – Through which processor writes data to an edge – One vertex can have multiple outputs • Example of Input/Output/Processor – MRInput & MROutput (InputFormat/OutputFormat) – OrderedGroupedKVInput & OrderedPartitionedKVOutput (Scatter Gather) – UnorderedKVInput & UnorderedKVOutput (Broadcast & One-‐to-‐One) – PigProcessor/HiveProcessor
- 10. © Hortonworks Inc. 2015 Benefit of DAG • Easier to express computation in DAG • No intermediate data written to HDFS • Less pressure on NameNode • No resource queuing effort & less resource contention • More optimization opportunity with more global context
- 11. © Hortonworks Inc. 2015 Agenda •Tez Introduction •Tez Feature Deep Dive •Tez Improvement & Debuggability •Tez Status & Roadmap
- 12. © Hortonworks Inc. 2015 Container-‐Reuse • Reuse the same container across DAG/Vertices/Tasks • Benefit of Container-‐Reuse – Less resources consumed – Reduce overhead of launching JVM – Reduce overhead of negotiate with Resource Manager – Reduce overhead of resource localization – Reduce network IO – Object Caching (Object Sharing)
- 13. © Hortonworks Inc. 2015 Tez Session • Multiple Jobs/DAGs in one AM • Container-‐reuse across Jobs/DAGs • Data sharing between Jobs/DAGs
- 14. © Hortonworks Inc. 2015 Dynamic Parallelism Estimation • VertexManager – Listen to the other vertices status – Coordinate and schedule its tasks – Communication between vertices
- 15. © Hortonworks Inc. 2015 ATS Integration • Tez is fully integrated with YARN ATS (Application Timeline Service) – DAG Status, DAG Metrics, Task Status, Task Metrics are captured • Diagnostics & Performance analysis – Data Source for monitoring & diagnostics – Data Source for performance analysis
- 16. © Hortonworks Inc. 2015 Recovery • AM can crash in corner cases – OOM – Node failure – … • Continue from the last checkpoint • Transparent to end users AM Crash
- 17. © Hortonworks Inc. 2015 Order By of Pig f = Load ‘foo’ as (x, y); o = Order f by x;Load Sample （Calculate Histogram) HDFS Partition Sort Broadcast Load Sample （Calculate Histogram) Partition Sort One-‐to-‐One Scatter Gather Scatter Gather
- 18. © Hortonworks Inc. 2015 Tez UI
- 19. © Hortonworks Inc. 2015 Tez UI
- 20. Tez UI 20 Download data from ATS
- 21. © Hortonworks Inc. 2015 RoadMap • Shared output edges – Same output to multiple vertices • Local mode stabilization • Optimizing (include/exclude) vertex at runtime • Partial completion VertexManager • Co-‐Scheduling • Framework stats for better runtime decisions
- 22. © Hortonworks Inc. 2015 Tez – Adoption • Apache Hive • Start from Hive 0.13 • set hive.exec.engine = tez • Apache Pig • Start from Pig 0.14 • pig -‐x tez • Cascading • Flink Page 22
- 23. © Hortonworks Inc. 2015 Tez Community • Useful Links – http://tez.apache.org/ – JIRA : https://issues.apache.org/jira/browse/TEZ – Code Repository: https://git-‐wip-‐us.apache.org/repos/asf/tez.git – Mailing Lists – Dev List: dev@tez.apache.org – User List: user@tez.apache.org – Issues List: issues@tez.apache.org • Tez Meetup – http://www.meetup.com/Apache-‐Tez-‐User-‐Group
- 24. © Hortonworks Inc. 2015 Thank You! Questions & Answers Page 24