Debugging of hadoop production errors using open source flow analyzer – semiconductor industry case study
1 like545 views
This document describes how a semiconductor manufacturer used the open source Jumbune flow analyzer to debug errors in their Hadoop MapReduce data partitioning process. The QA team found inconsistencies between wafer test data and packaging test data that were traced back to issues in how pass/fail thresholds were interpreted during MapReduce partitioning. Using Jumbune's debugger to validate data flows, custom classes were created to correlate test results with actual outputs, revealing that a critical test threshold was misinterpreted during partitioning, incorrectly flagging valid wafers as faulty. This helped resolve the production issues.
Debugging of hadoop production errors using open source flow analyzer – semiconductor industry case study
- 1. Debugging of Hadoop Production Errors Using Open Source Flow Analyzer – Semiconductor Industry Case Study Reach the community at users@collaborate.jumbune.org Download Jumbune from http://www.jumbune.org An Open Source initiative LGPLv3 licensed
- 2. Table of Contents I. Overview ................................................................ 2 II. Business Challenge and Future Proofing ............. 2 III. Use Case ................................................................ 2 IV. MapReduce Debugging using Jumbune’s Debugger ............................................................... 3 V. Impact: Zero issues in Production........................ 3
- 3. DEBUGGING OF HADOOP PRODUCTION ERRORS USING OPEN SOURCE FLOW ANALYZER – SEMICONDUCTOR INDUSTRY CASE STUDY 2 Overview A renowned manufacturer of microprocessors and GPUs, depends on data generated during manufacturing to improve upon its processors, manufacturing engineering processes, detect flaws during fabrication and also determine when a specific step in one of its manufacturing processes starts to deviate from normal tolerances. This data can include logs from the fabrication which contains the temperature, stage and wafer details and also logs from the multi-tier tests that are run during each phase of the manufacturing process. Business Challenge and Future Proofing The company traditionally used RDBMS to store all the current and historical data. As the data grew, as the costs associated with running analytics and maintenance of the database started multiplying the company decided to migrate to Apache Hadoop system to create an enterprise data lake on HDFS (Hadoop Distributed File System) for faster analytics. All the data from the Wafer processing, Die preparation to IC packaging and IC testing data would be pushed on to the HDFS enterprise data lake. One of the initial challenge faced by the data engineering team was partitioning data into their respective departments and/ sections for faster querying and accessibility. The team decided to go ahead with MapReduce for implementing the partitioning system. Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi- terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner. A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system. The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks. Use Case The various environments in the organization have heterogeneous Hadoop setups, Development and QA use Hadoop 2.4.1 Yarn, whereas Production environment has a commercial distribution of Hadoop. A system was put in place to partition data on the HDFS according to the source, type and ingestion period and was running smoothly till the QA team found some inconsistencies in the partitioned data. During the semiconductor device fabrication once the front- end process has been completed, the semiconductor devices are subjected to a variety of electrical tests to determine if they function properly. The fab tests the chips on the wafer with an electronic tester that presses tiny probes against the chip. The machine marks each bad chip with a drop of dye and is logged into a central repository and chips are sorted into virtual bins according to predetermined test limits. The resulting data can be graphed, or logged, on a wafer map to trace manufacturing defects and mark bad chips. This map can also be used during wafer assembly and packaging. Chips are also tested again after packaging and logged into the repository again. All the respective log and result data were dumped onto the HDFS data lake for further analysis. The data from the test phases were categorized according to their source, result and environment for pattern recognition. The QA team found that there were a lot of inconsistencies in the test data from the wafer phase and the packaging phase. Large number of the wafers that had passed the failure threshold during fabrication was found to be unresponsive during the later phases, this lead to a lot of effort being spent on finding out the source of the issue. Multiple failure points such as the test probe, test bed, the MapReduce system that sorts the data and the final quality checks were
- 4. DEBUGGING OF HADOOP PRODUCTION ERRORS USING OPEN SOURCE FLOW ANALYZER – SEMICONDUCTOR INDUSTRY CASE STUDY 3 analyzed to zero in on the source of the discrepancy. After spending lot of man hours testing the test equipment and the different phases of manufacturing it was concluded that the discrepancies were instigated during the MapReduce phase. MapReduce Debugging using Jumbune’s Debugger MapReduce debugging is a tiresome but unavoidable routine for any Hadoop developer. An effective debugger can make programmers more productive by allowing them to dedicate more time in implementing use cases. Debugger should allow the programmer to inspect the code at the site of an anomaly and trace back in the program stack to derive more clues about the cause of the problem. A successful debugger should collect data about billions of key value pairs that go through each component, phase and control block of the MapReduce application. Such an extensive system should be frugal, scalable and yet detailed. o It must be frugal that so that the instrumentation does not escalate the execution time of the code. o It must also collect information that is detailed enough to let the developer understand the bottlenecks and faults in their program. o The debugging system must also scale to large data sets which would accurately mimic all the use cases that the developers have in mind. Jumbune’s MapReduce debugger is a unique solution that gives the cluster level correlation of data with execution flow inside the code. It can bring down hours of execution logic debugging trials by identification of root causes within minutes. User may apply regex validations or user defined validation classes. As per the applied validation, Flow Debugger checks the flow of input data tuples essentially <key, value> pair data for each mapper and reducer in the submitted job. Jumbune provides a comprehensive table/chart view depicting the flow of input records through the job. The flow is displayed at job level, MapReduce level, and instance level. Unmatched keys/values represent the number of unexpected flow of key/value data through the job. Debugger drills down into the code to examine the flow of data for various counters like loops and if- conditions, else-if, etc. The team with the help of Jumbune’s flow debugger team created custom validation classes in java for each test and with this they were able to correlate the number of tests that passed and to the actual result, was able to detect that during the partitioning of data in the MapReduce phase, the pass threshold for a certain critical test was interpreted in a wrong manner which lead to a lot of valid wafer data being diagnosed as faulty. Impact: Zero issues in Production Production errors and failures results in significant loss in revenue and time, enterprise soliutions have zero tolerence to them. Jumbune’s Debugger has helped this organisation to get rid of production issues all tougether.