Rfhoc a random forest approach to auto-tuning hadoop's configuration
0 likes73 views
This paper presents rfhoc, an automatic tuning approach for Hadoop configuration parameters aimed at optimizing performance for applications. The method utilizes a random-forest approach to build performance models and a genetic algorithm to explore configuration settings, achieving an average performance speedup of 2.11 times, and up to 7.4 times compared to a recent cost-based optimization method. Additionally, the performance benefits of rfhoc enhance with larger input data sets.
Related topics:1 of 1
Download to read offline
More Related Content
What's hot (20)
Ad
Viewers also liked (7)
Ad
Similar to Rfhoc a random forest approach to auto-tuning hadoop's configuration (6)
More from ieeepondy (20)
Rfhoc a random forest approach to auto-tuning hadoop's configuration
- 1. RFHO C: A Random-Forest Approach to Auto-Tuning Hadoops Configuration Abstract: Hadoop is a widely-used implementation framework of the MapReduce programming model for large-scale data processing. Hadoop performance however is significantly affected by the settings of the Hadoop configuration parameters. Unfortunately, manually tuning these parameters is very time- consuming, if at all practical. This paper proposes an approach, called RFHOC, to automatically tune the Hadoop configuration parameters for optimized performance for a given application running on a given cluster. RFHOC constructs two ensembles of performance models using a random-forest approach for the map and reduce stage respectively. Leveraging these models, RFHOC employs a genetic algorithm to automatically search the Hadoop configuration space. The evaluation of RFHOC using five typical Hadoop programs, each with five different input data sets, shows that it achieves a performance speedup by a factor of 2.11 on average and up to 7.4 over the recently proposed cost-based optimization (CBO) approach. In addition, RFHOC's performance benefit increases with input data set size.