Linux Server Deep Dives (DrupalCon Amsterdam)
- 1. Linux Server Deep Dives Amin Astaneh Drupalcon Amsterdam 2019
- 2. Who Am I? ● Senior Manager, SRE, Acquia ● Acquian since December 2010 ● Champion DevOps, SRE, operational, agile best practices
- 3. WARNING: This is NOT Your Usual Linux Talk We won’t be talking about the usual suspects: ● top, ps, uptime ● sar, mpstat, iostat ● not even strace or lsof ● (well, maybe a little strace) Past talk: https://bit.ly/2BNzNy5
- 4. Goal: An Introduction to Advanced Tooling ● Tools: perf_events, ebpf ● Origins and capabilities ● How to install these tools ● Demo of examples that you can use today Aim is to provide inspiration on simple yet powerful ways to troubleshoot Drupal from the infrastructure and performance side. The classic tools answer what resources are being used. These tools answer how resources are being used in much greater detail.
- 5. Before We Begin: Tool Caveats 1) These tools can introduce a performance overhead. Keep that in mind when deciding to analyse your production workloads. Run in non-production where possible. 2) Some tools require you to rebuild your services in order to use them. Eg: mysqld, php, etc 3) Some tools require you to install debug packages to be useful. 4) These tools require root access.
- 6. Before We Begin: The Environment For This Talk ● Ubuntu 18.04 VM ● Drupal 8.7.8 installed running the Umami demo site ● Modest resources (1 core, 1GB RAM, 10GB HDD) ● No fancy caching like Varnish or Memcached
- 7. Before We Begin: Some Operating System Basics Let’s talk about system calls (aka: syscalls). It’s how programs interact with the kernel (in this case, Linux) to perform tasks, such as: ● read or write to a file ● database calls, memcached, HTTP ● executing other programs If you want the full list, run `man 2 syscalls`. If you want to read about a specific one, run `man 2 name_of_syscall`.
- 10. perf_events ● It’s been around since 2009 ● Part of the linux kernel since 2.6.31 ● Originally called Performance Counters for Linux ● Enables capture of analysis of broad performance-related kernel events ● Not very well documented :( ● To install: linux-tools package
- 11. The Extended Berkeley Packet Filter (eBPF) The Berkeley Packet Filter was originally simply that: a packet filter. However, there are certain characteristics of the project as it evolved since 2014 that expanded upon its originally-intended usage: ● Filters were implemented as programs that ran in a kernel-mode VM; ● “BPF guarantees that the programs loaded into the kernel cannot crash, and cannot run forever” ● eBPF programs can access in-kernel debugging features such as kprobes
- 12. What Does This Mean For eBPF? You can use eBPF for in-depth performance analysis of a running server, not just its network stack. The toolkit provided by the BPF compiler collection (BCC) provides us an accessible wealth of observability tools. It also provides the means to write your own tools.
- 13. Installing BCC ● Ubuntu: sudo apt-get install bpfcc-tools linux-headers-$(uname -r) ● RHEL: yum install bcc-tools ● Amazon Linux: yum install bcc
- 15. `perf` tool Allows you to monitor for specific OS events to trace/analyse ● counters- number of occurances that something happens ● tracing- real time tracking of events (usually syscalls) ● probing- monitor and capture specific events on the server ● reporting- analyse captured data
- 16. `perf stat` Example of counters: perf stat -e 'syscalls:sys_enter_*' COMMAND (lists the count of system calls for COMMAND) Example: let’s see what a `drush status` does to our system: sudo perf stat -e 'syscalls:sys_enter_*' drush status 2>&1 | grep -v ' 0 ' Why does this matter? A certain module or feature may be badly performing, and now you can find out why.
- 17. `perf trace` ● Say hello to a more performant replacement to strace! ● System call tracers print what is happening in real time ● Tracing PHP processes can be really useful for troubleshooting performance problems quickly when you don’t have an APM installed ● perf trace has less overhead than strace, by a LOT
- 18. `perf trace` overhead ● How do we test that? ● Using dd, we can see that perf trace has a 2.5x slowdown ● Strace had a 62x showdown. # dd if=/dev/zero of=/dev/null bs=512 count=10000k 5242880000 bytes (5.2 GB) copied, 3.53031 s, 1.5 GB/s # perf stat -e 'syscalls:sys_enter_*' dd if=/dev/zero of=/dev/null bs=512 count=10000k 5242880000 bytes (5.2 GB) copied, 9.14225 s, 573 MB/s # strace -c dd if=/dev/zero of=/dev/null bs=512 count=10000k 5242880000 bytes (5.2 GB) copied, 218.915 s, 23.9 MB/s
- 19. perf trace You can see all syscalls on the system with `perf trace`! For a single process, run `perf trace -p <PID>` or `perf trace <COMMAND>`
- 20. `perf record` ● You can sample all CPU activity on the system: ● perf record -a -F 1000 sleep 10 ○ Record activity on all processors, 1000 times per second, for 10 seconds ● Then you can generate a report on the output data ● perf report Note that you need to install debug packages in order to drill down into specific library calls! (PKG-dbg, or PKG-dbgsym)
- 21. `perf top` ● Like the top command, but for kernel-level events ● Plain `perf top` will tell you what userspace and kernel functions are using the most resources ● What is generating network traffic on the server? ● perf top -e net:net_dev_xmit -ns comm,pid
- 22. Dynamic Tracing with `perf probe` This allows you to monitor for invokation of specific kernel functions. ● create a probe: perf probe --add <FUNCTION> ● record probe behavior: perf record -e probe:<FUNCTION> -aR sleep 1 ● list probes: perf probe -l ● delete probes: perf probe -d <EVENT> You probably won’t use these when getting started, but know that this exists.
- 23. Trace HTTP Outbound Connections in Real Time Use tcpconnect to detect external calls performed by Drupal, cronjobs, etc. Could also help in detecting intruders! # tcpconnect PID COMM IP SADDR DADDR DPORT 1957 php-fpm 4 192.168.122.229 143.204.214.36 80
- 24. Trace HTTP Requests in Real Time Use tcptracer to detect all TCP connections on your server. A very easy way to find abusive or high-throughput HTTP clients as they happen!
- 25. How long do your HTTP client connections last? tcplife prints out the latency and data transfers for each connection, which again can be useful for analysing what your clients are doing.
- 26. Trace File Accesses On Web Server Use statsnoop to detect all file information accesses on your server (stat family of syscalls) # statsnoop | grep sites/default/files | egrep 'jpg|png|pdf|mp4' PID COMM IP SADDR DADDR DPORT 1957 php-fpm 4 192.168.122.229 143.204.214.36 80
- 27. Monitor file reads and writes! Use filetop to find how which specific files are getting the most activity!
- 28. How large are your per-process I/O operations? bitesize prints histograms of storage I/O operations for each process. May be useful to find programs that are doing excessive or inefficient operations.
- 29. How Long Does it Take For Filesystem Operations? ext4dist, xfsdist, zfsdist, etc will generate histograms of how long it takes to perform reads and write operations on the filesystem. This really breaks down the performance characteristics of the filesystem beyond what iostat will tell you.
- 30. Find out if you need more memory! More operating system theory: A ‘page fault’ means that a access to data required reading from the disk rather than what was in the page cache (stored in RAM). This is particularly important on servers expected to serve a lot of file data, eg: a file server. Too little memory for page cache affects performance. This is in a way similar to nginx or varnish miss rates. The cachestat tool enables you to monitor for this condition. The dcstat tool is useful for directory cache.
- 31. Trace Creation of New Processes Use pidpersec to determine the rate of new process creation. High values may be revealing that something is wrong with custom code such as cronjobs or scripts on the server.
- 32. Trace Creation of New Processes Use execsnoop to detect all new processes on your server. Quite useful for following up after use of pidpersec. # execsnoop-bpfcc PCOMM PID PPID RET ARGS date 2647 2499 0 /bin/date sleep 2648 2499 0 /bin/sleep 1 date 2649 2499 0 /bin/date sleep 2650 2499 0 /bin/sleep 1
- 33. Spy On a User Session! Use ttysnoop to watch another person’s shell session! # to find the ttys in use ps auxww --forest | egrep --color ‘^|pts’ # then to trace ttysnoop /dev/pts/X
- 34. Spy On All User Sessions! Similarly, you can use bashreadline to see all programs that have been invoked from a bash shell. Useful for analysing how jump hosts are being used.
- 35. Spy On SSL/TLS Connections! sslsniff will print the data being written to and read from SSL_write() and SSL_read() functions, basically intercepting encrypted traffic on the server!
- 36. In Summary ● perf_events and eBPF are pretty awesome additions to your toolkit ● You can see more details on Linux server activity than ever before ● You can start using these tools today :D ● Test in non-production first ● Have fun!
- 37. Further Reading
- 38. Further Reading ● Buy Brendan Gregg’s book on eBPF ○ http://www.brendangregg.com/bpf-performance-tools-book.html ● Perf Events Reference ○ https://perf.wiki.kernel.org/index.php/Main_Page ● bcc Github Project ○ https://github.com/iovisor/bcc ● Julia Evans’ Perf Cheatsheet ○ https://jvns.ca/perf-cheat-sheet.pdf ● Linux Syscall References ○ `man 2 syscalls` ○ `man 2 <SYSCALL>`
- 39. Thank You! Amin Astaneh Twitter: @aastaneh Email: amin@aminastaneh.net
- 40. Join us for contribution opportunities Mentored Contribution First Time Contributor Workshop General Contribution #DrupalContributions
- 41. What did you think? https://drupal.kuoni-congress.info/2019/program/ https://www.surveymonkey.com/r/DrupalConAmsterdam