Speeding up ps and top
1 like1,059 views
The document outlines a presentation on improving the /proc/pid interface for process information retrieval, proposing a new interface called 'task_diag' for faster access and efficiency. It discusses limitations of the current system, introduces the concepts of checkpoint/restore in userspace using CRIU, and highlights performance comparisons demonstrating task_diag's superiority. The authors are key contributors to container technology, with a focus on enhancing systems for better process management within Linux environments.
![13
[Bad] solution 1: introduce task_diag
● Not obvious where to get pid and user
namespaces
● Impossible to restrict netlink sockets
– Credentials are saved when a socket is created
– Process can drop privileges, but netlink doesn't care
– The same socket can be used to get process
attributes and to set ip addresses](https://image.slidesharecdn.com/task-diag-160124034003/85/Speeding-up-ps-and-top-13-320.jpg)
Speeding up ps and top
- 1. Speeding up ps and top Kirill Kolyshkin, Andrey Vagin SCALE 14x, 23 Jan 2016 Pasadena, CA
- 2. 2 Agenda ● Intro {Virtuozzo, OpenVZ, CRIU} ● Limitations of current /proc/PID interface ● Similar problems solved before ● Proposed solutions (yabad and good ones) ● Performance results
- 3. 3 ● Leading provider of secure, production-ready containers, hypervisors, and virtualized storage ● An industry pioneer, first containers in 2001 ● Powering some of world’s largest cloud networks – over 5 million mission critical cloud workloads ● 700+ worldwide partners
- 4. 4 ● Founded in 1997, “spun off” in Dec 2015 ● HQ in Seattle, offices in London, Moscow, Munich ● Over 170 employees, including 100+ engineers, 15 kernel hackers ● Contributor/sponsor of key open source initiatives 1997 2008 2015 2016 “A rose by any other name…”
- 5. 5 $ whoami ● Linux user since 1995 – Slackware on floppy disks, kernels 1.0.9 and 1.1.50 ● Developing VEs containers since 2002 – vzctl and vzpkg ● Leading OpenVZ from 2005 till 2015 ● SCALE user speaker since SCALE4x (2004) ● Twitter: @kolyshkin
- 6. 6 ● Full (system) containers for Linux ● Developed since 1999, open source since 2005 ● Live migration since 2007 ● ~2000 Linux kernel patches – enabling LXC, Docker, CoreOS… – biggest contributor to containers ● Now reborn as Virtuozzo 7, more open than ever OpenVZ
- 7. 7 CRIU: Checkpoint / Restore In Userspace ● About 3 y.o, ver 1.8 Dec 2015 ● Replaces OpenVZ in-kernel c/r ● Saves and restores sets of running processes ● Integrated into Docker, LXC ● Not just for live migration! – save HPC job or game, update kernel or hardware, balance load, speed-up boot, reverse debug, inject faults
- 8. 8 Ideas behind CRIU ● We can't merge kernel c/r upstream, so... hack it! Redo the whole thing in userspace ● Use existing interfaces where available – /proc, ptrace, netlink, parasite code injection ● Amend the kernel where necessary – only ~170 kernel patches – kernel v3.11+ is sufficient (if CONFIG_CHECKPOINT_RESTORE is set)
- 9. 9 Current interface: /proc/PID/* $ ls /proc/self/ attr cwd loginuid numa_maps schedstat task autogroup environ map_files oom_adj sessionid timers auxv exe maps oom_score setgroups uid_map cgroup fd mem oom_score_adj smaps wchan clear_refs fdinfo mountinfo pagemap stack cmdline gid_map mounts personality stat comm io mountstats projid_map statm coredump_filter latency net root status cpuset limits ns sched syscall
- 10. 10 Limitations of /proc/PID interface ● Requires at least three syscalls per each process – open(), read(), close() ● Variety of formats, mostly text based ● Not enough information (/proc/PID/fd/*) ● Some formats are non-extendable – /proc/PID/maps where the last column is optional ● Sometimes slow due to extra attributes – /proc/PID/smaps vs /proc/PID/maps ●
- 11. 11 /proc/PID/smaps 7f1cb0afc000-7f1cb0afd000 rw-p 00021000 08:03 656516 /usr/lib64/ld-2.21.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd wr mr mw me dw ac sd $ time cat /proc/*/maps > /dev/null real 0m0.061s user 0m0.002s sys 0m0.059s $ time cat /proc/*/smaps > /dev/null real 0m0.253s user 0m0.004s sys 0m0.247s
- 12. 12 Similar problem: info about sockets ● /proc – /proc/net/netlink – /proc/net/unix – /proc/net/tcp – /proc/net/packet ● Problems: not enough info, complex format, all-or-nothing ● Solution: use netlink, generalize tcp_diag as sock_diag – the extendable binary format – allows to specify a group of attributes and sockets
- 13. 13 [Bad] solution 1: introduce task_diag ● Not obvious where to get pid and user namespaces ● Impossible to restrict netlink sockets – Credentials are saved when a socket is created – Process can drop privileges, but netlink doesn't care – The same socket can be used to get process attributes and to set ip addresses
- 14. 14 A new interface for processes ● /proc/task_diag is a transaction file – write request → read response ● Netlink message format: binary and extendable ● Get information about a specified set of processes ● Optimal grouping of attributes – Any attribute in a group can't affect a response time ● Information about one process can be split into a few messages (16KB message size) ● Work in progress, anything may change!
- 15. 15 nlmsg_len nlmsg_type nlmsg_flags nlmsg_seq nlmsg_id nlattr_len nlattr_type payload nlattr_len nlattr_type payload Netlink message and attributes ● Simple and flexible message-based protocol ● Easy to add a new group ● Easy to add new attribute
- 16. 16 Ways to specify sets of processes ● TASK_DIAG_DUMP_ALL – Dump all processes ● TASK_DIAG_DUMP_ALL_THREAD – Dump all threads ● TASK_DIAG_DUMP_CHILDREN – Dump children of a specified task ● TASK_DIAG_DUMP_THREAD – Dump threads of a specified task ● TASK_DIAG_DUMP_ONE – Dump one task
- 17. 17 Groups of attributes ● TASK_DIAG_BASE – PID, PGID, SID, TID, comm ● TASK_DIAG_CRED – UID, GID, groups, capabilities ● TASK_DIAG_STAT – per-task and per-process statistics (same as taskstats, not avail in /proc) ● TASK_DIAG_VMA – mapped memory regions and their access permissions (same as maps) ● TASK_DIAG_VMA_STAT – memory consumption for each mapping (same as smaps)
- 18. 18 Performance: ps Get pid, tid, pgid and comm for 50000 processes $ time ./task_proc_all a real 0m0.279s user 0m0.013s sys 0m0.255s $ time ./task_diag_all a real 0m0.051s user 0m0.001s sys 0m0.049s A few times faster ;)
- 19. 19 Performance: using perf tool > Using the fork test command: > 10,000 processes; 10k proc with 5 threads = 50,000 tasks > reading /proc: 11.3 sec > task_diag: 2.2 sec > > @7,440 tasks, reading /proc is at 0.77 sec and task_diag at 0.096 > > 128 instances of sepcjbb, 80,000+ tasks: > reading /proc: 32.1 sec > task_diag: 3.9 sec > > So overall much snappier startup times. // David Ahern
Editor's Notes
- #5: “a rose by any other name” – you know your Shakespear, right?
- #6: Kernel 1.0.9 did not have support for IDE CDROM, and it took me a week to compile the 1.1.50 kernel that had it (as each kernel compilation was an overnight job). SCALE speaker in 2004. How many of you were at SCALE4x? What makes it more interesting is that time I came all the way from Moscow, Russia, and it was my first time in U.S.
- #7: OpenVZ, my beloved child
- #9: We failed to merge in-kernel c/r because that kernel code is very invasive, touching every kernel subsystem, no kernel maintainer wanted that in their code
- #10: More than 40 files and 10 directories for each process.
- #11: Variety of formats – no one wants to spend their life writing parsers for all these formats An example of non-extendable format is /proc/*/maps – last field is file name, and it is ... optional!
- #14: Another bad example of using netlink: taskstats
- #16: The structure is pretty generic, this is what makes this format extendable.