Tuning up with Apache Tez
Download as PPTX, PDF0 likes870 views
This document discusses tuning a pipeline with Apache Tez. It provides an overview of Tez, including that it aims to enhance scenarios not well served by MapReduce by supporting Hive and Pig. It details lessons learned from using Tez, such as some Pig tasks not compiling and occasional freezing, lack of DistributedCache support for S3, and poor Amazon support. The document concludes by stating that Tez is good for early adopters of Pig and Hive and those with bounded workloads.
Tuning up with Apache Tez
- 1. Tuning Up With Apache Tez Gal Vinograd @ Crosswise - 2016/03/09
- 3. Agenda The Pipeline The Problem Why we chose Tez Lessons Learned Demo
- 6. 250 c3.2xlarge X 30 hours 10TB per Batch
- 8. “Tez aims to be a general purpose execution runtime that enhances various scenarios that are not well served by classic Map-Reduce. In the short term the major focus is to support Hive and Pig ...” Tez Design v1.1
- 9. “Tez aims to be a general purpose execution runtime that enhances various scenarios that are not well served by classic Map-Reduce. In the short term the major focus is to support Hive and Pig ...” Tez Design v1.1
- 10. Hortonworks
- 13. Logical and Physical Graphs PhysicalLogical Hortonworks
- 15. Optimizations No Barrier Between Jobs Project GroupBy Project Project Distinct Project Distinct GroupBy Tez MR
- 16. Optimizations No Redundant Resource Allocation Project Project Distinct GroupBy Project Project Distinct GroupBy Pig Process Pig Process Tez MR
- 18. Lessons Learned Some Pig Tasks Did Not Compile Occasionaly Froze No DistributedCache Support For S3 Poor Amazon Support No Pre-Built Releases Additional Deployment for Tez UI
- 19. What is it good for? Earily Adopters Pig Hive Bounded
Editor's Notes
- #19: -Dpig.tez.opt.union=false