Linux Resource Management - Мариян Маринов (Siteground)
0 likes129 views
The document discusses Linux resource management techniques like ulimits, quotas, CPU affinity, and cGroups. It provides examples of configuring ulimits and CPU affinity at the process level using tools like ulimit, taskset, and proc files. It also explains how cGroups can be used to control resources like CPU, memory, block I/O for groups of processes in a hierarchy.
![● login (on tty, via PAM)
● KDM, GDM, XDM & etc. (locally via PAM)
● ssh (remotely, via PAM and shell)
● pam_limits
– /etc/security/limits.conf
– /etc/security/limits.d/
● shell (sh, bash, zsh, csh, tcsh)
– /etc/profile.d/limits.[tcz]sh
ulimitsulimits how-tohow-to](https://image.slidesharecdn.com/linuxresourcelimits-171127081837-171204072916/85/Linux-Resource-Management-Siteground-9-320.jpg)
Linux Resource Management - Мариян Маринов (Siteground)
- 1. Linux Resource Linux Resource ManagementManagement Marian HackMan Marinov Chief System Architect mm@siteground.com
- 2. Who am I?Who am I? ● Chief System Architect - SiteGroundChief System Architect - SiteGround ● Linux System Administrator since 1996Linux System Administrator since 1996 ● Teaching LSA and NetSec at FMI SofiaTeaching LSA and NetSec at FMI Sofia ● Organizing OpenFest and othersOrganizing OpenFest and others
- 3. ● ulimitulimit ● quotaquota ● CPU affinity per-device and per-processCPU affinity per-device and per-process ● cGroupscGroups
- 4. cpu time (seconds, -t) unlimitedcpu time (seconds, -t) unlimited scheduling priority (-e) 0scheduling priority (-e) 0 real-time priority (-r) 0real-time priority (-r) 0 file size (blocks, -f) unlimitedfile size (blocks, -f) unlimited pending signals (-i) 96832pending signals (-i) 96832 open files (-n) 1024open files (-n) 1024 file locks (-x) unlimitedfile locks (-x) unlimited pipe size (512 bytes, -p) 8pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200POSIX message queues (bytes, -q) 819200 max user processes (-u) 200max user processes (-u) 200 max locked memory (kbytes, -l) 64max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimitedmax memory size (kbytes, -m) unlimited virtual memory (kbytes, -v) unlimitedvirtual memory (kbytes, -v) unlimited core file size (blocks, -c) 0core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimiteddata seg size (kbytes, -d) unlimited stack size (kbytes, -s) 8192stack size (kbytes, -s) 8192 ulimitsulimits
- 5. app1 userXuserX user procsuser procs userX 1userX 1 tty:tty: core file size (blocks, -c) 0core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimiteddata seg size (kbytes, -d) unlimited scheduling priority (-e) 0scheduling priority (-e) 0 file size (blocks, -f) unlimitedfile size (blocks, -f) unlimited pending signals (-i) 96832pending signals (-i) 96832 max locked memory (kbytes, -l) 64max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimitedmax memory size (kbytes, -m) unlimited open files (-n) 1024open files (-n) 1024 pipe size (512 bytes, -p) 8pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0real-time priority (-r) 0 stack size (kbytes, -s) 8192stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimitedcpu time (seconds, -t) unlimited max user processes (-u) 200max user processes (-u) 200 virtual memory (kbytes, -v) unlimitedvirtual memory (kbytes, -v) unlimited file locks (-x) unlimitedfile locks (-x) unlimited ulimitsulimits
- 6. app2 app1 userXuserX userXuserX user procsuser procs userX 2userX 2 tty:tty: core file size (blocks, -c) 0core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimiteddata seg size (kbytes, -d) unlimited scheduling priority (-e) 0scheduling priority (-e) 0 file size (blocks, -f) unlimitedfile size (blocks, -f) unlimited pending signals (-i) 96832pending signals (-i) 96832 max locked memory (kbytes, -l) 64max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimitedmax memory size (kbytes, -m) unlimited open files (-n) 1024open files (-n) 1024 pipe size (512 bytes, -p) 8pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0real-time priority (-r) 0 stack size (kbytes, -s) 8192stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimitedcpu time (seconds, -t) unlimited max user processes (-u) 200max user processes (-u) 200 virtual memory (kbytes, -v) unlimitedvirtual memory (kbytes, -v) unlimited file locks (-x) unlimitedfile locks (-x) unlimited ulimitsulimits
- 7. app2 app1 app3 userXuserX userXuserX userXuserX user procsuser procs userX 3userX 3 tty:tty: core file size (blocks, -c) 0core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimiteddata seg size (kbytes, -d) unlimited scheduling priority (-e) 0scheduling priority (-e) 0 file size (blocks, -f) unlimitedfile size (blocks, -f) unlimited pending signals (-i) 96832pending signals (-i) 96832 max locked memory (kbytes, -l) 64max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimitedmax memory size (kbytes, -m) unlimited open files (-n) 1024open files (-n) 1024 pipe size (512 bytes, -p) 8pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0real-time priority (-r) 0 stack size (kbytes, -s) 8192stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimitedcpu time (seconds, -t) unlimited max user processes (-u) 200max user processes (-u) 200 virtual memory (kbytes, -v) unlimitedvirtual memory (kbytes, -v) unlimited file locks (-x) unlimitedfile locks (-x) unlimited ulimitsulimits
- 8. core file size (blocks, -c) 0core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimiteddata seg size (kbytes, -d) unlimited scheduling priority (-e) 0scheduling priority (-e) 0 file size (blocks, -f) unlimitedfile size (blocks, -f) unlimited pending signals (-i) 96832pending signals (-i) 96832 max locked memory (kbytes, -l) 64max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimitedmax memory size (kbytes, -m) unlimited open files (-n) 1024open files (-n) 1024 pipe size (512 bytes, -p) 8pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0real-time priority (-r) 0 stack size (kbytes, -s) 8192stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimitedcpu time (seconds, -t) unlimited max user processes (-u) 200max user processes (-u) 200 virtual memory (kbytes, -v) unlimitedvirtual memory (kbytes, -v) unlimited file locks (-x) unlimitedfile locks (-x) unlimited app2 app1 app3 userXuserX userXuserX userXuserX user procsuser procs userX 4userX 4 app4 userXuserX ssh:ssh: tty:tty: ulimitsulimits
- 9. ● login (on tty, via PAM) ● KDM, GDM, XDM & etc. (locally via PAM) ● ssh (remotely, via PAM and shell) ● pam_limits – /etc/security/limits.conf – /etc/security/limits.d/ ● shell (sh, bash, zsh, csh, tcsh) – /etc/profile.d/limits.[tcz]sh ulimitsulimits how-tohow-to
- 10. $ cat /proc/self/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 200 200 processes Max open files 1024 4096 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 200 200 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us ulimitsulimits how-tohow-to
- 11. $ cat /proc/self/limits on older kernels: $ echo -n "Max open files=2000:6000" > /proc/self/limits $ prlimit ulimitsulimits how-tohow-to
- 12. Other kernel limits ● fs.file-max - max fd for the machine ● fs.nr_open - max fd per process ● fs.mount-max - max mounted filesystems ● kernel.threads-max
- 13. ● Dedicate a CPU to HW device ● Dedicate a CPU to a process ● taskset mask cmd ● /proc/interrupts – /proc/irq/NUM/smp_affinity – /proc/irq/NUM/smp_affinity_list – /proc/irq/NUM/affinity_hint CPU AffinityCPU Affinity
- 14. ● Dedicate a CPU to HW device ● Dedicate a CPU to a process core0 core1 core2 core3 eth0 1Gbps eth4 10Gbps megaraid 6Gbps CPU AffinityCPU Affinity
- 15. ● Dedicate a CPU to HW device ● Dedicate a CPU to a process core0 core1 core2 core3 eth0 1Gbps eth1 10Gbps eth2 10Gbps megaraid 6Gbps core0 - eth1 10Gbps core1 - eth2 10Gbps core3 - megaraid 6Gbps core4 - eth0 & processes CPU AffinityCPU Affinity
- 16. taskset example root@terion:~# taskset -p 2727 pid 2727's current affinity mask: ff root@terion:~# taskset -pc 3 2727 pid 2727's current affinity list: 0-7 pid 2727's new affinity list: 3 root@terion:~# taskset -p 2727 pid 2727's current affinity mask: 8 root@terion:~# ps axf|grep 2727 2727 ? Ss 2:06 /usr/sbin/acpid root@terion:~#
- 17. irq affinity example root@terion:~# cat /proc/interrupts CPU0 CPU1 16: 3567385 0 IO-APIC 16-fasteoi ehci_hcd:usb1 17: 4567 0 IO-APIC 17-fasteoi snd_hda_intel: 23: 50797 0 IO-APIC 23-fasteoi ehci_hcd:usb2 25: 78045696 0 PCI-MSI 512000-edge ahci 36: 12 0 PCI-MSI 409600-edge eth0 37: 169256226 0 PCI-MSI 1572864-edge iwlwifi 38: 3515939 0 PCI-MSI 524288-edge nvidia
- 18. irq affinity example root@terion:~# cd /proc/irq/37 root@terion:/proc/irq/37# cat smp_affinity ff root@terion:/proc/irq/37# cat smp_affinity_list 0-7 root@terion:/proc/irq/37# echo 3 > smp_affinity_list root@terion:/proc/irq/37# cat smp_affinity 08 root@terion:/proc/irq/37# cat smp_affinity_list 3 root@terion:/proc/irq/37#
- 19. Other resource limitations can be enforced using virtualization technologies like KVM, Xen, etc.
- 20. What if you want to set a limit to a group of processes?
- 21. ● CPUSET ● CPU ● CPUACCT ● MEMORY ● BLKIO ● DEVICES ● freezer ● net_cls ● net_prio ● perf_event ● hudgetlb cGroupscGroups
- 22. cGroupscGroups ● freezer ● net_cls ● net_prio ● perf_event ● hudgetlb ● CPUSET ● CPU ● CPUACCT ● MEMORY ● BLKIO ● DEVICES
- 23. ● cGroups have hierarchy // /user1/user1 /user2/user2 /user1/user3/user1/user3 cGroupscGroups
- 24. root@goblin:/cgroup# ls -1 cpuset* cpuset.cpus cpuset.mems cpuset.cpu_exclusive cpuset.mem_exclusive cpuset.effective_cpus cpuset.effective_mems ... cGroupscGroups CPUSETCPUSET
- 25. root@goblin:/cgroup# ls -1 cpu.* cpu.cfs_period_us cpu.cfs_quota_us cpu.rt_period_us cpu.rt_runtime_us cpu.shares cpu.stat cGroupscGroups CPUCPU
- 26. root@goblin:/cgroup# ls -1 cpuacct.* cpuacct.stat cpuacct.usage cpuacct.usage_percpu cpuacct.usage_all cpuacct.usage_percpu_sys cpuacct.usage_percpu_user cpuacct.usage_sys cpuacct.usage_user CPUACCTCPUACCTcGroupscGroups
- 27. memory.memsw.failcnt memory.memsw.limit_in_bytes memory.memsw.max_usage_in_bytes memory.memsw.usage_in_bytes memory.limit_in_bytes memory.usage_in_bytes memory.soft_limit_in_bytes memory.max_usage_in_bytes memory.move_charge_at_immigrate memory.failcnt memory.numa_stat memory.stat memory.oom_control memory.pressure_level memory.swappiness memory.use_hierarchy cGroupscGroups MEMORYMEMORY
- 29. blkio.weight blkio.weight_device blkio.leaf_weight blkio.leaf_weight_device BLKIOBLKIOcGroupscGroups
- 30. cGroupscGroups root@goblin:/cgroup# ls -1 devices.* devices.allow devices.deny devices.list DEVICESDEVICES
- 31. Marian HackMan Marinov Chief System Architect mm@siteground.com QuestionsQuestions