Imaginea cs hadoop
- 1. CASE STUDY Enhancing, Monitoring and Managing a Hadoop Based Analytics Solution In this engagement, Imaginea team contributed over 14 patches to the Hadoop community and all of them were verified and accepted. COPYRIGHT © 2012, IMAGINEA TECHNOLOGIES, INC. COPYRIGHT © 2012, IMAGINEA TECHNOLOGIES, INC. THIS DOCUMENT IS CONFIDENTIAL AND NOT FOR DISTRIBUTION WITHOUT WRITTEN PERMISSION FROM IMAGINEA TECHNOLOGIES, INC.
- 2. CASE STUDY 1. Executive Summary One of Imaginea’s clients is a video marketing company that deals with branding, real time media buying, ad serving, targeting, optimization and brand measurement. Imaginea enhanced and managed a platform for video playtime statistical analysis for our client. The solution used Hadoop (Cloudera distribution) and Hive. The cluster was 500 nodes with 300 TB of existing data and over 200 GBs data being streamed in and processed every day. 2. Hadoop Migration and New Features We helped in migration of the entire platform from 0.19 to 0.20.2, porting all the MR jobs. Migration also included back-porting some feature from .21 to 20. Features that were back-ported included: Map-Side join CompositeInputFormat 3. Cluster Monitoring, Management & Resolution We helped in monitoring and managing the cluster during IST Business hours. We were able to uncover workflow instability issues and lack of resume feature during these phases, which we resolved later. The solution had used a custom workflow manager; it had stability issues especially as the load increased by orders of magnitude. Zookeeper was introduced as the central workflow status manager and changes were made to the workflow manager to use the same. This helped the system stability improve by about 90%. We also discovered problems in publishing configuration and code changes to all the nodes in the cluster during this phase. We used Ganglia and Nagios for COPYRIGHT © 2012, IMAGINEA TECHNOLOGIES, INC. THIS DOCUMENT IS CONFIDENTIAL AND NOT FOR DISTRIBUTION WITHOUT WRITTEN PERMISSION FROM IMAGINEA TECHNOLOGIES, INC.
- 3. CASE STUDY monitoring. We also solved some of the memory overflow issues in the Hadoop nodes. 4. Configuration Management using Puppet As part of the engagement, Imaginea worked to introduce Puppet in to the system removing a custom configuration management tool. We had developed some recipes and were able to solve many issues that were raised with replication of configuration changes and deployment of new codebase 5. Performance Improvements Imaginea contributed to enhance performance in a variety of ways. Below are a couple of highlight scenarios Job Starvation Problem: Many cases of data overflow at a collector level The solution had business analytics Hive queries, which used to starve the normal MR jobs. Imaginea helped in development of a fair scheduling algorithm which balances the production tasks and hive query jobs. Before this solution there were many cases of data overflow at the collector level. Job Optimization Problem: Job to identify if the user was unique took over 8 hours Imaginea helped in optimizing the job from 8-10 hours to 4 hours using better distribution of keys and better hashing algorithm. COPYRIGHT © 2012, IMAGINEA TECHNOLOGIES, INC. THIS DOCUMENT IS CONFIDENTIAL AND NOT FOR DISTRIBUTION WITHOUT WRITTEN PERMISSION FROM IMAGINEA TECHNOLOGIES, INC.
- 4. CASE STUDY 6. Apache Hadoop involvement and other contributions We have worked on Apache Hadoop and other components. Following are the list of patches that were contributed to the community by Imaginea. Jira Id Severity/ Priority Component Brief Description MAPREDUCE-3360 Critical-Improvement mrv2 Provide information about lost nodes in the UI MAPREDUCE-3686 Critical-Bug mrv2 history server web ui - job counter values for map/reduce not shown properly MAPREDUCE-3532 Critical-Bug mrv2,nodema When 0 is provided as port number in nager yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0 port to RM MAPREDUCE-3952 Major-Bug mrv2 In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split. MAPREDUCE-3686 Critical-Bug mrv2 history server web ui - job counter values for map/reduce not shown properly MAPREDUCE-3532 Critical-Bug mrv2,nodema When 0(zero) is provided as port number in nager yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0(zero) port to RM MAPREDUCE-3316 Major-Bug Resource Rebooted Link not working Manager MAPREDUCE-3708 Major-Bug mrv2 Metrics: Incorrect Apps submitted count MAPREDUCE-3723 Major-Bug mrv2, test, TestAMWebServicesJobs & TestHSWebServicesJobs webapp incorrectly asserting tests MAPREDUCE-4050 Major-Bug mrv2 Invalid Node link MAPREDUCE-3870 Major – Bug mrv2 Invalid App Metrics MAPREDUCE-4102 Major – Bug Webapps Job counter not available in Job History Web UI for killed jobs MAPREDUCE-4002 Major – Bug Examples MultiFileWordCount job fails if the input path is not from default file system MAPREDUCE-4040 Minor-Bug mrv2, History links should use hostname rather than IP address. jobhistoryserv er MAPREDUCE-3212 Minor-Bug mrv2 Message displays while executing yarn command should be proper COPYRIGHT © 2012, IMAGINEA TECHNOLOGIES, INC. THIS DOCUMENT IS CONFIDENTIAL AND NOT FOR DISTRIBUTION WITHOUT WRITTEN PERMISSION FROM IMAGINEA TECHNOLOGIES, INC.