Performance Tuning: You’re doing it wrong! by Kirk Pepperdine
This document outlines 10 common mistakes made when performing performance tuning: 1) Setting up a tiger team to diagnose issues by committee, 2) Giving a system more hardware without understanding the underlying bottleneck, 3) Having poor system visibility that provides too much unclear monitoring data, 4) Diving into code without properly identifying the primary bottleneck, 5) Not being able to profile a production system due to statistical sampling issues, 6) Not listening to signals from the system about where problems exist, 7) Assuming performance issues are due to known system behaviors without properly diagnosing, 8) Tuning based on settings from Google without understanding the specific system context, 9) Tuning based on folklore from blogs without empirical evidence, and 10) Not
Performance Tuning: You’re doing it wrong! by Kirk Pepperdine
- 1. Performance Tuning You’re doing it wrong!
- 2. Number 10: No performance requirements • No requirements == no problem
- 3. Number 9: Tuning by folklore • Performance happens in a context • admin manuals are devoid of context • blogs don’t speak to your context ® • Measure don’t guess • use measurements to guide all decisions
- 4. Number 8: Tuning by Google -XX:InitialHeapSize=1610612736 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=697933824 -XX:NewSize=697933824 -XX:OldSize=1395867648 -XX:OldPLABSize=16 -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:SurvivorRatio=1 -XX:TargetSurvivorRatio=90 -XX:-UseAdaptiveSizePolicy -XX:+PrintAdaptiveSizePolicy -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:-UseLargePagesIndividualAllocation -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled
- 5. Number 7: That is how my system works • Often the diagnosis takes 15 minutes • convincing the client that the implement is a problem can take hours!!!
- 6. Number 6: Not listening to the system • Presented groups of developers with a performance problem • not one group has solved it in 8 years!!! • once they learn to read the signals the problem can be diagnosed in minutes!
- 7. Number 5: Not being able to profile in prod • Would you let your doctor diagnose you by running the tests on a newt? • Profiling is statistical sampling technique • Performance bottleneck is a statistically significant event • Profilers dilute the answer with expensive to capture useless data points
- 8. Number 4: Diving into the code • “Know” what the problem is based on local knowledge • even if you tell them what the problem is they won’t change course!!! • In one exercise >97% of developers fail to identify and fix the primary bottleneck • even when they’ve been emphatically told where they will fail
- 9. Number 3: Poor system visibility • Monitoring pukes a ton of data into your lap • often leave people scratching their heads trying to understand where to start!!! • Often app is only logging • logs are optimized for development, not ops
- 10. Number 2: Give it more hardware • If you don’t understand your bottleneck….. • Won’t work if you can’t use the hardware you already have • Seemingly perfectly parallelizable batch job needed to run in 1 hour • single server took 4 hours • 4 servers took more than 8 hours to complete!!!
- 11. Number 1: Setting up a tiger team • Seriously!!! resort to diagnosis by committee? • Clear indication that your team is using technology that either don’t understand or can’t manage • Nice way to trigger “tribal” behavior.
- 12. jClarity Illuminate Performance Diagnostic Engine • Picks up where monitoring stops • Minimize ambiguity • uses heuristics to identify performance bottlenecks in live system • points out casual execution path! • Profiles with minimal impact on (already poor) performance