When the OS gets in the way
2 likes210 views
The document discusses optimizing Linux operating system performance for low-latency applications, emphasizing the importance of isolating CPU resources, tuning parameters, and validating changes through testing. It provides practical strategies for reducing latency and jitter, such as using tools like lstopo, taskset, and csets to manage CPU affinity and resource allocation. Real-world examples demonstrate the impact of tuning on response times, highlighting the need for careful measurement and optimization during implementation.
![Rogue process
java
36049 [003] 3011858.780856: sched:sched_switch: java:
36049 [110] R ==> kworker/3:1:13991 [120]
kworker/3:1
13991 [003] 3011858.780861: sched:sched_switch:
kworker/3:1:13991 [120] S ==> java:36049 [110]](https://image.slidesharecdn.com/jaxdownload-160509143943/85/When-the-OS-gets-in-the-way-33-320.jpg)
![Some things can’t be deferred
kworker/3:1-13991 [003] 3013287.180771: funcgraph_entry: | process_one_work() {
kworker/3:1-13991 [003] 3013287.180772: funcgraph_entry: | cache_reap() {
kworker/3:1-13991 [003] 3013287.180772: funcgraph_entry: 0.137 us | mutex_trylock();
kworker/3:1-13991 [003] 3013287.180772: funcgraph_entry: 0.289 us | drain_array();
kworker/3:1-13991 [003] 3013287.180773: funcgraph_entry: 0.040 us | _cond_resched();
………
………
kworker/3:1-13991 [003] 3013287.180859: funcgraph_exit: +86.735 us | }
+86.735 us](https://image.slidesharecdn.com/jaxdownload-160509143943/85/When-the-OS-gets-in-the-way-36-320.jpg)
When the OS gets in the way
- 1. When the OS gets in the way (and what you can do about it) Mark Price @epickrram LMAX Exchange
- 2. When the OS gets in the way (and what you can do about it) Linux Mark Price @epickrram LMAX Exchange
- 3. ● Linux is an excellent general-purpose OS ● Many target platforms ● Scheduling is actually fairly complicated ● Low-latency is a special use-case ● We need to provide some hints It’s not the OS’s fault
- 4. Why should I care?
- 5. Useful in some scenarios ● Low latency applications ● Response times < 1ms ● Compute-intensive workloads ● Long-running jobs
- 6. A real-world scenario: LMAX System Latency = T1 - T0 Before tuning: 250us / 10+ms After tuning: 80us / <1ms (mean / max)
- 7. Jitter ● “slight irregular movement, variation, or unsteadiness, especially in an electrical signal or electronic device” ● Variation in response time latency ● Long-tail in response time
- 8. Dealing with it ● First take care of the low-hanging fruit ○ e.g. Garbage collection (gc-free / Zing) ○ e.g. Slow I/O ● Once response times are < 10ms the fun begins ● Make sure your code is running!
- 9. Measure first ● Need to validate changes are good ● End-to-end tests ● Using realistic load ● Change one thing and observe ● A refresher...
- 11. Multi-tasking ● num(tasks) > num(HyperThreads) ● OS must share out hardware resources ● Clever? Dumb? Fast? Slow? ● Fair...
- 12. Linux CFS ● Completely Fair Scheduler ● Maintains a task ‘queue’ per HT ● Runs the task with the lowest runtime ● Updates task runtime after execution ● Higher priority implies longer execution time ● Tasks are load-balanced across HTs
- 13. An example application ... Threads
- 14. … running on a language runtime
- 15. … running on an operating system
- 16. Optimise for locality - PCI/memory
- 18. How do I start?
- 19. ● BIOS settings for maximum performance ● That’s a whole other talk... Start with the metal
- 20. ● lstopo is a useful tool for looking at hardware ● Provided by the hwloc package ● Displays: ○ HyperThreads ○ Physical cores ○ NUMA nodes ○ PCI locality Discover what’s available
- 21. lstopo
- 23. ● Use isolcpus to reserve cpu resource ● kernel boot parameter ● isolcpus=0-5,10-13 ● Use taskset to pin your application to cpus: ● taskset -c 10-13 java … ● Set affinity of hot threads: ● sched_setaffinity(...) Reserve & use specific resource
- 24. Deploy the application sched_setaffinity() !{isolcpus} taskset
- 25. You have no load-balancer Pile-up
- 26. A solution: cpusets ● Create hierarchical sets of reserved resource ● CPU, memory ● Userland tools: cset (SUSE)
- 27. Isolate OS processes ● cset set --set=/system --cpu=6-9 ○ create a cpuset with cpus 6-9 ○ create it at the path /system ● cset proc --move --from-set=/ --to-set=/system ○ move all processes from / to /system ○ -k => move unbound kernel threads ○ --threads => move child threads ○ --force => erm... force
- 28. Run the application ● cset set --cpu=0-5,10-13 --set=/app ● cset proc --exec /app taskset -cp 10-13 java … ○ start a process in the /app cpuset ○ run the program on cpus 10-13 ● sched_setaffinity() to pin the hot threads to cpus 1,3,5
- 30. No more jitter?
- 31. ● Sampling tracer ● Static/dynamic trace points ● Very low overhead ● A good starting point for digging deeper ● perf list to view available trace points ● network, file-system, scheduler, etc perf_events
- 32. What’s happening CPU? ● perf record -e "sched:sched_switch" -C 3 ○ Sample task switches on CPU 3 ● perf report (best for multiple events) ● perf script (best for single events)
- 33. Rogue process java 36049 [003] 3011858.780856: sched:sched_switch: java: 36049 [110] R ==> kworker/3:1:13991 [120] kworker/3:1 13991 [003] 3011858.780861: sched:sched_switch: kworker/3:1:13991 [120] S ==> java:36049 [110]
- 34. ftrace ● Function tracer ● Static/dynamic trace points ● Higher overhead ● But captures everything ● Can provide function graphs ● trace-cmd is the usable front-end
- 35. So what is that kernel thread doing? ● trace-cmd record -P <pid> -p function_graph ○ Trace functions called by process <pid> ● trace-cmd report ○ Display captured trace data
- 36. Some things can’t be deferred kworker/3:1-13991 [003] 3013287.180771: funcgraph_entry: | process_one_work() { kworker/3:1-13991 [003] 3013287.180772: funcgraph_entry: | cache_reap() { kworker/3:1-13991 [003] 3013287.180772: funcgraph_entry: 0.137 us | mutex_trylock(); kworker/3:1-13991 [003] 3013287.180772: funcgraph_entry: 0.289 us | drain_array(); kworker/3:1-13991 [003] 3013287.180773: funcgraph_entry: 0.040 us | _cond_resched(); ……… ……… kworker/3:1-13991 [003] 3013287.180859: funcgraph_exit: +86.735 us | } +86.735 us
- 37. Things to look out for ● cache_reap() - SLAB allocator ● vmstat_update() - kernel stats ● other workqueue events ○ perf record -e “workqueue:*” -C 3 ● Interrupts - set affinity in /proc/irq ● Timer ticks - tickless mode ● CPU governor - set to performance ○ /sys/devices/system/cpu/cpuN/cpufreq/scaling_governor
- 38. Some numbers ● Inter-thread latency is a good proxy ● 2 busy-spinning threads passing a message ● Time taken between producer & consumer ● Record times over several seconds ● Compare tuned/untuned
- 39. Results == Latency (ns) == mean min 50.00% 90.00% 99.00% 99.90% 99.99% max untuned 466 200 464 608 768 992 2432 69632 tuned 216 128 208 288 336 544 1664 69632
- 40. tuned vs untuned
- 41. tuned vs untuned (log scale)
- 42. Results (loaded system) == Latency (ns) == mean min 50.00% 90.00% 99.00% 99.90% 99.99% max untuned 545 144 464 544 736 2944 294913 884739 tuned 332 216 336 352 448 544 704 36864
- 43. tuned vs untuned (loaded system)
- 44. Summary ● Select threads that need access to CPU ● Isolate CPUs from the OS ● Pin important threads to isolated CPUs ● Don’t forget interrupts ● There will be more things… ● Always test assumptions! ● Run validation tests to ensure tunings are as expected
- 45. Thank you ● lmax.com/blog/staff-blogs/ ● epickrram.blogspot.com ● github.com/epickrram/perf-workshop ● @epickrram