DISTRIBUTED PERFORMANCE ANALYSIS USING INFLUXDB AND THE LINUX EBPF VIRTUAL MACHINE
1 like441 views
The document discusses the use of eBPF (extended Berkeley Packet Filter) for distributed performance analysis, detailing its functionalities, processes, and various applications. eBPF serves as a tracing framework that enables dynamic instrumentation of the Linux kernel to monitor events and trace behavior without requiring extensive modifications to the kernel itself. It also highlights practical examples of using eBPF to trace database queries, network activity, and device I/O latency, among others.
![© 2018 InfluxData. All rights reserved.8
In today’s world: tcpdump
Documentation about the instruction set: https://www.kernel.org/doc/Documentation/networking/filter.txt
# tcpdump -d 'ip and tcp port 80'
(000) ldh [12]
(001) jeq #0x800 jt 2 jf 12
(002) ldb [23]
(003) jeq #0x6 jt 4 jf 12
(004) ldh [20]
(005) jset #0x1fff jt 12 jf 6
(006) ldxb 4*([14]&0xf)
(007) ldh [x + 14]
(008) jeq #0x50 jt 11 jf 9
(009) ldh [x + 16]
(010) jeq #0x50 jt 11 jf 12
(011) ret #262144
(012) ret #0
-d stands for: Dump the compiled packet-matching code in a human readable form to standard output and stop.
Is src (x+14) on port 80 (0x50)?
Is src (x+16) on port 80 (0x50)?
Is it an ethernet IP IPv4 packet?](https://image.slidesharecdn.com/distributedperformanceanalysisusinginfluxdbandthelinuxebpfvirtualmachine-181122135831/85/DISTRIBUTED-PERFORMANCE-ANALYSIS-USING-INFLUXDB-AND-THE-LINUX-EBPF-VIRTUAL-MACHINE-8-320.jpg)
![In today’s world: seccomp
gcc -lseccomp seccomp-test.c
./a.out
hey there!
something's gonna happen!!
[1] 19463 killed ./a.out](https://image.slidesharecdn.com/distributedperformanceanalysisusinginfluxdbandthelinuxebpfvirtualmachine-181122135831/85/DISTRIBUTED-PERFORMANCE-ANALYSIS-USING-INFLUXDB-AND-THE-LINUX-EBPF-VIRTUAL-MACHINE-9-320.jpg)
DISTRIBUTED PERFORMANCE ANALYSIS USING INFLUXDB AND THE LINUX EBPF VIRTUAL MACHINE
- 1. © 2018 InfluxData. All rights reserved.1 Distributed Performance Analysis Using InfluxDB and the Linux eBPF Virtual Machine
- 2. eBPF (extended BPF) Extended Berkeley Packet Filter
- 3. © 2018 InfluxData. All rights reserved.3 BPF is a Tracing Framework* Used to access kernel trace backend instrumentation tools *Actually, it’s not just that. And there’s also XDP.
- 4. © 2018 InfluxData. All rights reserved.4 Static tracepoints # cat /sys/kernel/debug/tracing/available_events sched: task: signal: timer: workqueue: irq: kvm: # ls /sys/kernel/debug/tracing/events/irq/ enable filter irq_handler_entry irq_handler_exit softirq_entry softirq_exit softirq_raise tcp:
- 5. © 2018 InfluxData. All rights reserved.5 Dynamic trace functionalities uprobes kprobes XDP
- 6. © 2018 InfluxData. All rights reserved.6 eBPF program lifecycle Kernel space User space User program eBPF bytecode Static verifier BPF BPF_MAP_* Results (bpf_lookup_elem) uprobes kprobes tracepoints BPF_PROG_LOAD see man 2 bpf The mustache parrot warns! eBPF programs can’t be turing complete! XDP socketfilter
- 8. © 2018 InfluxData. All rights reserved.8 In today’s world: tcpdump Documentation about the instruction set: https://www.kernel.org/doc/Documentation/networking/filter.txt # tcpdump -d 'ip and tcp port 80' (000) ldh [12] (001) jeq #0x800 jt 2 jf 12 (002) ldb [23] (003) jeq #0x6 jt 4 jf 12 (004) ldh [20] (005) jset #0x1fff jt 12 jf 6 (006) ldxb 4*([14]&0xf) (007) ldh [x + 14] (008) jeq #0x50 jt 11 jf 9 (009) ldh [x + 16] (010) jeq #0x50 jt 11 jf 12 (011) ret #262144 (012) ret #0 -d stands for: Dump the compiled packet-matching code in a human readable form to standard output and stop. Is src (x+14) on port 80 (0x50)? Is src (x+16) on port 80 (0x50)? Is it an ethernet IP IPv4 packet?
- 9. In today’s world: seccomp gcc -lseccomp seccomp-test.c ./a.out hey there! something's gonna happen!! [1] 19463 killed ./a.out
- 10. © 2018 InfluxData. All rights reserved.10 10 More practical examples? ¨ Trace file opens by filename ¨ Trace queries done against a database, like InfluxDB or MySQL ¨ Trace TCP retransmissions ¨ Trace all commands done in a bash shell ¨ Trace block device I/O latency over time ¨ JVM events ¨ Go Runtime Events ¨ Firewalls, packet rewriting, dropping etc..
- 11. High-level APIs are there!
- 12. © 2018 InfluxData. All rights reserved.12 Write every executed bash command to Influx Kernel space User space main.go eBPF bytecode Static verifier BPF BPF_PERF_OUTPUT Results (bpf_open_perf_buffer) uretprobe get_return_value BPF_PROG_LOAD LLVM Backend “ebpf” Results channel Results consumer consumes writes Populates the channel Populates the userspace map
- 13. © 2018 InfluxData. All rights reserved.13 As a Daemonset in a Kubernetes Cluster! Kernel space (for privileged POD) Pod User space main.go eBPF bytecode Static verifier BPF BPF_PERF_OUTPUT Results (bpf_open_perf_buffer) uretprobe get_return_value BPF_PROG_LOAD LLVM Backend “ebpf” Results channel Results consumer consumes writes Populates the channel Populates the userspace map
- 14. © 2018 InfluxData. All rights reserved.14
- 15. © 2018 InfluxData. All rights reserved.15
- 16. © 2018 InfluxData. All rights reserved.16 References 1. https://www.iovisor.org/ 2. https://github.com/cilium/cilium 3. https://github.com/iovisor/gobpf 4. https://landlock.io/ 5. https://github.com/iovisor/bpftrace 6. https://github.com/iovisor/bpf-docs 7. https://medium.com/@fntlnz/load-xdp-programs-using-the-ip-iproute2-command-502043898263 8. https://www.youtube.com/watch?v=JRFNIKUROPE 9. https://cilium.readthedocs.io/en/latest/bpf/ 1. iovisor BCC 2. Cilium: HTTP, gRPC, and Kafka Aware Security and Networking for Containers with BPF and XDP 3. iovisor/gobpf - To load eBPF programs using Go 4. Landlock LSM 5. iovisor bpftrace 6. iovisor BPF docs 7. Blog post on how to load xdp programs using iproute2 8. BPF Tracing Talk from Brendan Gregg 9. Cilium documentation for BPF
- 17. Thank you!lorenzo@influxdata.com twitter.com/fntlnz github.com/fntlnz If you’re that kind of person can find my pgp key here: https://fntlnz.wtf/downloads/pubkey-B2400EE4.asc