Experience with jemalloc
- 1. Experience with jemalloc Kit Chan (kichan@oath.com)
- 2. Problem – Difficult to debug memory leak in ATS Plugins • Plugin coded in C or C++ - easy to produce memory leak bugs • Hard to debug in large scale production system – Leak can take days or weeks to be noticeable – Can’t roll back • Don’t know which one. Multiple changes can be suspects • Critical feature cannot be rolled back
- 3. Options • Valgrind – Typically slows down by 10 to 20x • AddressSanitizer (ASAN) – Need to recompile. – Can slow down by 2x
- 4. Jemalloc for Memory Profiling • Compile and install jemalloc • Create a file (/usr/local/bin/start_ats.sh) with the following contents #!/bin/sh MALLOC_CONF="prof:true,prof_prefix:/tmp/jeprof.out,lg_prof_interval:34,lg_prof_sample:20" LD_PRELOAD=”/usr/local/lib/libjemalloc.so.2" export MALLOC_CONF export LD_PRELOAD /home/y/bin64/traffic_server "$@” – Interval between sampling – 2^20 = 1MB – Interval between file dump – 2^32 = 4GB – Prefix of file dump - /tmp/jeprof.out – Profiling is on. • Update “proxy.config.proxy_binary” to the file above in records.config • Other options available – please see jemalloc’s doc
- 5. Viewing the Results • Sample Usage jeprof --show_bytes --gif /usr/local/bin/traffic_server /tmp/jeprof.out.32201.3730.i3730.heap > /tmp/32201.3730.gif • Generate a gif file containing the call graph of the program • Other formats and options supported • Here is an example
- 7. Case Study #1 • ATS in front of multiple API Origins • Leak happened for several months. • Took about 2 weeks to be noticeable
- 10. Case Study #2 • ATS in front of multiple origins, serving HTML and JS/CSS/Images assets • Leak happened and took 12 hours to OOM • Multiple critical fixes out at the same time
- 12. Case Study #2 • Our own Brotli plugin did not release the encoder instance correctly
- 13. Problem – ATS not scaling up on more Cores/Better CPU
- 14. Memory operations are the issues
- 15. Plugins (ESI) are the problem
- 16. Jemalloc is the solution • CPU utilization can now stress to 90%+
- 17. Future – Running it on production – ATS 7.x allows us to turn off freelist – Tuning Options. E.g • lg_dirty_mult • lg_chunk
- 18. Conclusion • Jemalloc/Jeprof – good complementary tool for debugging memory leak • Improve scalability • Tunable
Editor's Notes
- #4: In the past valgrind is a very popular tool to debug memory leak. However it typically will slow down the process by 10 to 20X. In ATS there is effort to use ASAN to find memory leak problems. And there is a presentation in ATS Summit in 2015 to go into details on how to use Address Santizier. One problem with ASAN is that we need to recompile the binary. Still it is reported by with ASAN we can still experience a 2X slow down in performance. So it may still not be suitable for live debugging for critical system. Finally, we can always set up monitoring for the Ats process memory usage. Then we can trace back the changes that cause the memory to grow over a period of times. However, as stated above, there is still a lot of guess works needed to pinpoint the actual root cause. So we need something more. And Jemalloc comes to the rescue.
- #5: Please note that there are a few other options available for memory profiling and you can check it out in the jemalloc documents.