![Page22 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49
TFile parser
• Tez logs can be parsed via PIG
• Allows us to treat our logs exactly like we treat our big-data
• Processing using “pig –x tez” + UDFs [1]
rawLogs = load ‘/app-logs/root/logs/application_1409012059361_0539/*' using
org.apache.tez.tools.TFileLoader() as (machine:chararray, key:chararray, line:chararray);
[1] - https://github.com/rajeshbalamohan/tez_log_parser/blob/master/src/main/resources/pig/udf.groovy](https://image.slidesharecdn.com/tezui-141204163521-conversion-gate02/85/TEZ-8-UI-Walkthrough-21-320.jpg)
TEZ-8 UI Walkthrough
- 1. Page1 © Hortonworks Inc. 2014 Tez: UI & Debugging Fall 2014 Version 1.0 gopalv@apache.org
- 2. Page2 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 TEZ (nomenclature) • DAG • Vertex • Task • Attempt • Container • Edge
- 3. Page3 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Directed Acyclic Graphs
- 4. Page4 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 How to view raw DAGs from logs • Tez Application logs contain .dot files in Graphviz format • To generate images: dot –Tpng –o dag.png dag.dot • OR javascript version: http://people.apache.org/~gopalv/dagviz/
- 5. Page5 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 TEZ-8 JIRA & branch • TEZ UI for progress tracking and history • https://issues.apache.org/jira/browse/TEZ-8 • https://github.com/apache/tez/tree/TEZ-8 • UI-centric branch
- 6. Page6 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Tez-UI: Landing page
- 7. Page7 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Tez UI: DAG view
- 8. Page8 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Tez UI: Vertex view
- 9. Page9 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Tez UI: Vertex -> Tasks view
- 10. Page10 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Tez UI: Task logs Task logs
- 11. Page11 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Tez UI: Task counters Task counters
- 12. Page12 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Tez UI: Task counters Search for counters
- 13. Page13 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Tez UI: Per-edge shuffle counters Map 3 to Map 1 only
- 14. Page14 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Tez UI: Payload view
- 15. Page15 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Tez UI: Failed DAGs (diagnostic)
- 16. Page16 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Tez UI: Failed tasks indication Failed tasks
- 17. Page17 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Tez UI: Failed tasks
- 18. Page18 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Tez UI: Failed attempts
- 19. Page20 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Post-hoc/Ad-hoc analysis helpers • tez/tez-tools ships with two helper tools • swimlanes • tez-tfile-parser
- 20. Page21 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Swimlanes • ./yarn-swimlanes.sh application_1415860665053_0098
- 21. Page22 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 TFile parser • Tez logs can be parsed via PIG • Allows us to treat our logs exactly like we treat our big-data • Processing using “pig –x tez” + UDFs [1] rawLogs = load ‘/app-logs/root/logs/application_1409012059361_0539/*' using org.apache.tez.tools.TFileLoader() as (machine:chararray, key:chararray, line:chararray); [1] - https://github.com/rajeshbalamohan/tez_log_parser/blob/master/src/main/resources/pig/udf.groovy
- 22. Page23 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 TFile parser (contd) • Parsing INFO logs for shuffle for instance (for time taken + machine) Problematic machine
- 23. Page24 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 TFile parser (node/rack traffic at 350 nodes) Problematic machine Fetcher in node-100 is always slow (irrespective of where its pulling data from) Other faulty nodes Mapout served from node-100 to node-120 To any node is always slow
- 24. Page25 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49 Questions? • Thanks all tez contributors for their efforts! • FYI, Hadoop Summit 2015 (Europe) Call for papers is out