Hadoop & Spark Performance tuning using Dr. Elephant
6 likes3,025 views
Dr. Elephant is an automated performance monitoring and tuning tool designed to optimize Hadoop infrastructure by improving user productivity and providing performance insights. It highlights common mistakes, offers tuning tips, and helps manage resource usage effectively. Future plans include real-time job analysis and enhanced visualization tools to further support users.
Hadoop & Spark Performance tuning using Dr. Elephant
- 2. Introduction
- 4. Scale and Optimize Hardware ● More users, more jobs, more resources ● Large investment in hardware ● Can’t keep upgrading and adding machines to solve problem forever ● Some tuning is needed to get things running
- 5. Users are more valuable than machines What do we do?
- 7. User Productivity ● Freedom to experiment and run jobs on the cluster ● Build tools to help developers. (Hadoop DSL, Resolvers for Pig/Hive) ○ Improve developer lifecycle ○ Also reduce unnecessary resource wastage
- 9. How easy is it to tune a job? ● Problems are not obvious ● Critical information is scattered ● Inter-related settings ● Large parameter space
- 10. Here’s what we learned!
- 11. Expert Intervention ● Not enough support resources available ● Poor coverage ● Difficult to prioritize efforts ● Delays user development Random Suggestions
- 12. Training is not at all easy ● Too many users ● Diverse backgrounds ● Scope is large and evolving ● Other responsibilities are more important
- 13. Scaling Productivity is Hard!
- 14. Dr. Elephant to the Rescue
- 15. What does Dr. Elephant do? ● Automated performance monitoring and tuning tool ● Help every user get the best performance from their jobs ● Highlights common mistakes ● Indicates best practices and tuning tips ● Provides a platform for other performance related tools ● Analyzes hundred thousand jobs every day
- 16. Architecture
- 17. Dashboard
- 18. Search
- 19. Job Page
- 20. MapReduce Report
- 21. Failed Job
- 22. Help Page
- 23. Tuning Tips
- 24. Awesome Features
- 25. Simplified analysis of a flow’s historical executions ● Monitoring performance, resource usage and many others ● Comparing flows against previous executions ● Impact of tuning a specific parameter or a changing a line of code
- 26. Flow History
- 27. Job History
- 28. Heuristics
- 29. How does a Heuristic work? ● Fetch Counters and Task Data ● Some logic to compute a value ● Compare value against threshold levels
- 30. Heuristic Severity Severity Color Description CRITICAL The job is in critical state and must be tuned SEVERE There is scope for improvement MODERATE There is scope for further improvement LOW There is scope for few minor improvements NONE The job is safe. No tuning necessary
- 31. Example | Mapper Data Skew
- 32. Mapper Skew Problem ● Number of Mappers depend on the number of splits ● Varying size of splits can cause skewness in the Mapper Input
- 33. Solution to Mapper Skewness ● Each Mapper should process the same amount of data ● Combine the small chunks and feed it to a single Mapper
- 34. Example | Spark Executor Load Balance
- 35. Spark Driver Executor 1 Executor 2 Executor 3 RDD Partition 1 Partition 2 Partition 3
- 37. Adding a New Heuristic 1. Create a new heuristic and test it. 2. Create a new view for the heuristic. For example, helpMapperSpill.scala.html 3. Add the details of the heuristic in the HeuristicConf.xml file. <heuristic> <applicationtype>mapreduce</applicationtype> <heuristicname>Mapper GC</heuristicname> <classname>com.linkedin.dre.mapreduce.heuristics.MapperGC</classname> <viewname>views.html.help.mapreduce.helpGC</viewname> </heuristic> 4. Run Dr. Elephant. It should now include the new heuristics.
- 38. Configuring Heuristics/Threshold levels <heuristics> <heuristic> <applicationtype>mapreduce</applicationtype> <heuristicname>Mapper Data Skew</heuristicname> <classname>com.linkedin.dre.mapreduce.heuristics.MapperDataSkew</classname> <viewname>views.html.help.mapreduce.helpMapperDataSkew</viewname> <params> <num_tasks_severity>10, 50, 100, 200</num_tasks_severity> <deviation_severity>2, 4, 8, 16</deviation_severity> <files_severity>1/8, 1/4, 1/2, 1</files_severity> </params> </heuristic> </heuristics>
- 39. Elephagent
- 40. Workflow monitoring and reports ● Performance characteristics change ○ Data Growth ○ Data distribution change ○ Hardware change ○ Incremental software change ● Monitor performance on each execution ● Compare behaviour across revisions ● Cost to Serve analysis
- 41. Production Reviews | JIRA Bot ● Separate cluster for critical workloads ● Audit before deployment ● Improved accuracy ● Faster turnaround ● Higher throughput
- 42. Future Plans
- 43. Upcoming ● Job Resource Usage and Wastage ● Job Wait time ● Real time analysis of a job ● Workflow DAG visualization ● Improved Spark heuristics
- 44. References Engineering Blog: engineering.linkedin.com/blog/2016/04/dr-elephant-open- source-self-serve-performance-tuning-hadoop-spark Open Source Github Link: github.com/linkedin/dr-elephant Mailing List: Dr-elephant-users Hadoop Summit 2015: https://www.youtube.com/watch?v=aL3OJ4YoxPA
- 45. Thank You
- 46. ©2014 LinkedIn Corporation. All Rights Reserved. ©2014 LinkedIn Corporation. All Rights Reserved. © 2016