Python performance profiling
3 likes1,614 views
The document discusses performance profiling in Python, emphasizing the necessity of understanding bottlenecks in both development and production environments. It highlights tools and methods for profiling, such as hotshot pycallgraph and greenletprofiler, and the importance of logging slow database queries and API calls. The goal is to gather actionable insights to ensure code reliability and efficacy before deployment.
Python performance profiling
- 1. ©2013 DataStax Confidential. Do not distribute without consent. @rustyrazorblade Jon Haddad Technical Evangelist, DataStax Python Performance Profiling 1
- 2. What are our goals? • Understand potential bottlenecks in dev • Testing • Call graphs • Understand code once it's in production • Micro benchmarks • Automatic logging of slow DB queries / api calls • Gather evidence • No guessing • We need insight into both environments
- 3. Why do we need it in dev & prod? • Dev != production • No network latency on our desktops • Round trips are cheap in dev • Rarely hitting disk (DB fully in memory) • Zero CPU contention • Failure / failover rarely tested
- 5. Approaches in Dev • Unit / functional tests • Code coverage is important • if you’re not testing it, it’s probably broken • Must be reliable, repeatable • Always keep production in mind • Know your hardware • Load test regularly • Jenkins performance plugin
- 6. Finding slow tests is easy
- 7. Sometimes it's unavoidable… • Make sure you mark tests that are expected to be slow • These are frequently testing offline tasks in functional tests
- 9. pycallgraph • Understand code structure and flow • Summarize times • Darker colors represent more time spent
- 10. Blocking I/O • Usually the problem with web servers • Apps can be CPU bound but it's less frequent
- 11. Moving past blocking I/O • Event libraries! • libev most stable • gevent is a beautiful wrapper • Pool.map() is your friend • async can hide issues & make code harder to profile
- 12. Profiler - GreenletProfiler • Takes into account greenlets • Generates callgrind files • Mac Users: qcachegrind
- 13. In Production
- 14. Profile with minimal overhead • We need something really lightweight! • Our applications can time EVERYTHING • api requests • database queries • individual functions • small blocks of code • statsd is our friend • microtimers, counters • Integrates w/ librato, graphite
- 15. statsd + graphite / grafana
- 16. Logging • Log slow database queries / api calls automatically • Log & aggregate errors • What table was hit? • Read or write? • What was the query? • Can we duplicate? • Logstash / splunk / etc
- 17. ©2013 DataStax Confidential. Do not distribute without consent. 17