![Hipe
info
•
native, {hipe, [o3]}
•
doesn’t mix with NIFs
•
•
on_load
switching between non-native and native code is
expensive
•
different call stacks
•
might overload the code_server (bug?)
•
—enable-native-libs
•
hipe_bifs (sshhh)](https://image.slidesharecdn.com/performanceoptimization101-efsf2014-140307134433-phpapp02/85/Performance-optimization-101-Erlang-Factory-SF-2014-31-320.jpg)
Performance optimization 101 - Erlang Factory SF 2014
- 1. Performance Optimization 101 Louis-Philippe Gauthier Team leader @ AdGear Trader
- 2. Exercise
- 3. HTTP API server API GET /date - returns today’s date GET /time - returns the unix time in seconds
- 4. HTTP API server TODO • accepting connections • parsing http requests • routing • building responses
- 5. HTTP API server accepting connections
- 6. HTTP API server accepting connections
- 7. HTTP API server accepting connections • gen_tcp:controlling_process/2 is slow • spawn worker with ListenSocket • worker accepts and ack’s listener
- 8. HTTP API server accepting connections
- 9. HTTP API server accepting connections
- 10. HTTP API server accepting connections
- 11. HTTP API server accepting connections • use proc_lib instead of gen_server • socket options: • binary • {backlog, 4196} • {raw, 6, 9, <<30:32/native>>}
- 12. HTTP API server parsing request
- 13. HTTP API server parsing request
- 14. HTTP API server parsing request • binary matching is very powerful! • working with binaries is more memory efficient • binaries over 64 bytes are shared (not copied) • faster than the built-in http parser (BIF) when running on many cores and using hipe • keep state in a record • O(1) lookups
- 16. HTTP API server routing pattern matching is awesome!!
- 17. HTTP API server building response
- 18. HTTP API server building response
- 19. HTTP API server building response
- 20. HTTP API server building response • ETS is your friend! • cache time date in ETS public table • {read_concurrency, true} • if you store a binary over 64 bytes, it won’t get copied! • have a gen_server update the cache • every second for the time • every day for the date
- 21. HTTP API server building response • do not try to rewrite everything • use community projects and contribute back! • often your application will spend most of its time talking to external services • premature optimization is usually bad
- 22. Gotchas slow functions / modules • erlang:now/0 vs os:timestamp/0 • proplists:get_value() vs lists:keyfind() • timer:send_after() vs erlang:send_after() • gen_udp:send() vs erlang:port_command() • avoid erlang:controlling_process() if you can • avoid base64, string, unicode modules
- 23. Tools
- 24. Profiling info • useful to find slow code paths • fprof • • output is really hard to understand • • uses erlang:trace/3 erlgrind to read in kcachegrind eflame • also uses erlang:trace/3 • nice graphical output
- 25. Eflame info • flamechart.pl (from Joyent) • makes it visually easy to find slow function calls
- 26. Eflame how to
- 27. Eflame info
- 28. Micro benchmarks info • start with profiling • useful for experimentation and to validate hypothesis • small benchmarking library called timing • uses the excellent bear (statistics) library
- 30. Micro benchmarks info # parallel processes erlang:now/0 os:timestamp/0 1 0.99 0.87 10 22.87 2.54 100 168.23 16.99 1000 664.46 51.98
- 31. Hipe info • native, {hipe, [o3]} • doesn’t mix with NIFs • • on_load switching between non-native and native code is expensive • different call stacks • might overload the code_server (bug?) • —enable-native-libs • hipe_bifs (sshhh)
- 32. Hipe how to
- 33. NIFs info • function that is implemented in C instead of Erlang • can be dangerous… • • OOM (memory leak) • • crash VM (segfault) must return < 500 us (to be safe…) ideally should yield and use enif_consume_timeslice • • what is a reduction? dirty schedulers (R17) • finally!
- 34. Process Tuning info • tune min_heap_size on spawn • fullsweep_after if you have memory issues • • force gc +hms (set default min_heap_size)
- 36. Monitoring info • statsderl for application metrics • vmstats for VM metrics • system_stats for OS metrics • erlang:system_monitor/2 • entop for live system exploration
- 37. Statsderl info • statsd client • very cheap to call (async) • offers 3 kinds of metrics: • counters - for counting (e.g QPS) • gauges - for absolute values (e.g. system memory) • timers - similar to gauges but with extra statistics
- 38. Statsderl how to
- 39. VM Stats info • process count • messages in queues • run queue length • memory (total, proc_used, atom_used, binary, ETS) • scheduler utilization (per scheduler) • garbage collection (count, words reclaimed) • reductions • IO bytes (in/out)
- 40. VM Stats info
- 41. System Stats info • load1, load5, load15 • cpu percent • can be misleading because of spinning schedulers • virtual memory size • resident memory size • very useful to track those OOM crashes !
- 43. System Monitor info • monitoring for: • • busy_dist_port • long_gc • long_schedule • • busy_port large_heap riak_sysmon + lager / statsderl handler
- 45. Dashboard info
- 46. Entop info • top(1)-like tool for the Erlang VM • can be used remotely • gives per process: • • reductions • message queue length • ! pid / name heap size
- 47. Entop info
- 48. VM Tuning info • +K true (kernel polling) • +sct db (scheduler bind) • +scl false (disable load distribution) • +sfwi 500 (force sheduler wakeup NIFs) • +spp true (port parallelism) • +zdbbl (distribution buffer busy limit) • test with production load (synthetic benchmarks can be misleading)
- 49. Cset info • tool to help create cpusets • reduces non voluntary context-switches • reserve first two CPUs for interrupts and background jobs • reserve rest of CPUs for the Erlang VM • linux only
- 50. Cpuset how to
- 52. Other tools info • system limits • • • ulimit -n sysctl dtrace / systemtap • application + OS tracing
- 54. Thank you! github: lpgauth! irc: lpgauth (@erlounge)! twitter: lpgauth