Dynamic Instrumentation- OpenEBS Golang Meetup July 2017
0 likes550 views
The document discusses dynamic instrumentation and tracing for programming, emphasizing its application for debugging and understanding large codebases without risking program stability in production. It highlights various tools and techniques, such as kprobes, uprobes, ftrace, and SystemTap, that facilitate both user-space and kernel tracing on Linux systems. Additionally, it mentions the evolution of tracing with eBPF and its integration with other tracing tools, providing a flexible and efficient approach for developers.
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017
- 1. Dynamic Instrumentation @JeffryMolanus @openebs Golang Meetup Bangalore XXIV 15 July 2017
- 2. Tracing • To record information about a programs execution • Useful for understanding code, in particular a very large code base • Used during debugging, statistics, so on and so forth • Dynamic tracing is the ability to ad-hoc add or remove certain instrumentation without making changes to the code that is subject to tracing or restarting the program or system • In general, tracing should not effect the stability of the program that is being traced in production, during development its less of importance • When no tracing is enabled there should be no overhead; when enabled the overhead depends on what is traced and how • User land tracing requires abilities in kernel (which is the focus of this talk) • user space tracing has a little more overhead due to the induced context switch
- 3. Tracers on other platforms • Illumos/Solaris and FreeBSD • Dtrace, very powerful and production safe used for many years • Compressed Type Format (CTF) data is available in binaries and libraries, no need for debug symbols to work with the types • Solaris uses the same CTF data for type information for debugging • Event Tracing for Windows (EWT) • Linux • Requires debug symbols to be downloaded depending on what you trace and how specific you want to trace • With DWARF data more can be done then with plain CTF however
- 4. Basic architecture of tracing • There are generally, two parts of tracing in Linux • Frontend tools to work/consume with/the in kernel tracing facilities • We will look briefly in ftrace, systemtap and BCC • Backend subsystems • Kernel code that executes what ever code you want to be executed on entering the probes function or address • kprobes, probes, tracepoints, sysdig
- 5. ftrace • Tracepoints; static probes defined in the kernel that can be enabled at run time • ABI is kept stable by kernel • static implies you have to know what you want to trace while developing the code • Makes use of sysfs interface to interact with it • Several wrappers exist to make things a little easier • tracecmd and kernelshark (UI) • Also check the excellent stuff from Brendan Gregg
- 7. Trace points in sysfs
- 8. kernelshark
- 9. kprobes • kprobes is defined in multiple sub categories • jprobes: trace function entry (optimised for function entry, copy stack) • kretprobes: trace function return • kprobes: trace at any arbitrary instruction in the kernel • To use it one has to write a kernel module which needs to be loaded at run time • this is not guaranteed to be safe • A kprobe replaces the traced instruction with a break point instruction • On entry, the pre_handler is called after instrumenting, the post handler
- 10. kprobes
- 11. Kprobe example
- 12. Kprobe example
- 13. jprobes • Note: function prototype needs to match the actual syscall
- 14. utrace/uprobes • Roughly the the same as the kprobe facility in the kernel but focused on user land tracing • current ptrace() in linux is implemented using the utrace frame work • tools like strace and GDB use ptrace() • Allows for more sophisticated tooling, one of which is uprobes • Trace points are placed on the an inode:offset tuple • All binaries that map that address will have a SW breakpoint injected at that address
- 15. ftrace & user space • The same ftrace interface is available for working with uprobes • Behind the scene the kernel does the right thing (e.g use kprobe, tracepoints, or uprobes) • The same sysfs interface is used, general work flow: • Find address to place the probe on • Enable probing • Disable probing • View results (flight recorder)
- 18. eBPF• Pretty sure everyone here has used BPF likely with out knowing • tcpdump uses BPF • eBPF is enhanced BPF • sandboxed byte code executed by kernel which is safe and user defined • attach eBPF to kprobes and uprobes • certain restrictions in abilities
- 19. BCC • BPF Compiler Collection, compiles code for the in kernel VM to be executed • Several high level wrappers for Python, lua and GO • Code is still written in C however
- 20. Recap • Several back-end tracing capabilities in the kernel • Tracepoints, kprobes, jprobes, kretprobes and uprobes • eBPF allows attachment to kprobe, uprobes and tracepoints for safe execution • Linux tracing world can use better generic frontends for adhoc tracing • Best today are perf and systemtap (IMHO) • Who wants to write C when you want to print a member of a complex struct? (ply)
- 21. Systemtap • High level scripting language to work with the aforementioned tracing capabilities of Linux • Flexible as it allows for writing scripts that can trace specific lines within a file (debug symbols) • Next to tracing, it can also make changes to running programs when run in “guru mode” • Resulting scripts from systemtap are kernel modules that are loaded in to the kernel (kprobe and uprobes) • Adding a eBPF target is in the works as currently, systemtap may result in unremovable modules or sudden death of traced processes
- 22. stp files • Example script oneliner: • stap -e ‘probe syscall.open { printf(“exec %s, file%s, execname(), filename) }’ • stap -L ‘syscall.open' • syscall.open: __nr:long name:string filename:string flags:long flags_str:string mode:long argstr:string • List user space functions in process “trace” • stap -L ‘process(“./trace").function("*")' • .call and .return probes for each function
- 23. List probes
- 24. Tracing line numbers • What's the value of ret after line 35? • Could be done by tracing ret values, but that is not the purpose of this exercise • gcc -g -O0 • full debug info
- 25. Tracing line number • .statement(“main@code/talk/trace.c:36”) { … }
- 28. Downstack • All functions being called by a function
- 29. Tracing go
- 30. Cant trace return values
- 31. Calling convention • AMD64 calling conventions • RDI, RSI, RDX, RCX, R8 and R9 • Go is based on PLAN9 which uses a different approach therefore tracing does not work as well as one would like it to be (yet) • This also goes for debuggers • Perhaps Go will start using the X86_64 ABI as it moves forward or all tools and debuggers will add specific PLAN9 support • https://go-review.googlesource.com/#/c/28832/ (ABI change?) • GO bindings to the BCC tool chain • Allows for creating eBPF tracing tools written in go • but still requires writing the actual trace logic in C
- 32. Summary • Dynamic tracing is an invaluable tool for understanding code flow • To verify hypotheses around software bugs or understanding • Ability to make changes to code on the fly with out recompiling (guru mode) • Under constant development most noticeable the eBPF/BCC work