eBPF Powered Distributed Kubernetes Performance Analysis - Lorenzo Fontana, InfluxData
2 likes790 views
The document discusses the extended Berkeley Packet Filter (eBPF) and its application in performance analysis within Kubernetes environments. It outlines various eBPF functionalities like static and dynamic tracing, practical use cases, and mentions tools such as bpftrace and kubectl trace. Additionally, it provides references for further reading on eBPF and related projects.
eBPF Powered Distributed Kubernetes Performance Analysis - Lorenzo Fontana, InfluxData
- 1. @fntlnz eBPF Powered Kubernetes Performance Analysis
- 2. @fntlnz eBPF (extended BPF) Extended Berkeley Packet Filter
- 3. © 2018 InfluxData. All rights reserved.3 BPF is a Tracing Framework* Used to access kernel trace backend instrumentation tools *Actually, it’s not just that. And there’s also XDP.
- 4. © 2018 InfluxData. All rights reserved.4 Static tracepoints sched: task: signal: timer: workqueue: irq: kvm: tcp: @fntlnz
- 5. © 2018 InfluxData. All rights reserved.5 Dynamic trace functionalities uprobes kprobes XDP @fntlnz
- 6. © 2018 InfluxData. All rights reserved.6 Aggregate events at kernel side and deal with just a few events instead of thousands of them @fntlnz
- 7. © 2018 InfluxData. All rights reserved.7 eBPF program lifecycle Kernel space User space User program eBPF bytecode Static verifier BPF BPF_MAP_* Results (bpf_lookup_elem) uprobes kprobes tracepoints BPF_PROG_LOAD see man 2 bpf The mustache parrot warns! eBPF programs can’t be turing complete! XDP socketfilter
- 9. © 2018 InfluxData. All rights reserved.9 In today’s world: tcpdump -d stands for: Dump the compiled packet-matching code in a human readable form to standard output and stop. Documentation about the instruction set: https://www.kernel.org/doc/Documentation/networking/filter.txt Is it an ethernet IP IPv4 packet? Is src (x+14) on port 80 (0x50)? Is dst (x+16) on port 80 (0x50)?
- 10. @fntlnz In today’s world: seccomp
- 11. © 2018 InfluxData. All rights reserved.11 11 @fntlnz More practical examples? ¨ Trace file opens by filename ¨ Trace queries done against a database, like InfluxDB or MySQL ¨ Trace TCP retransmissions ¨ Trace all commands done in a bash shell ¨ Trace block device I/O latency over time ¨ JVM events ¨ Go Runtime Events ¨ Firewalls, packet rewriting, dropping etc..
- 12. @fntlnz High-level APIs are there!
- 13. © 2018 InfluxData. All rights reserved.13 13 @fntlnz iovisor/gobpf
- 14. © 2018 InfluxData. All rights reserved.14 14 @fntlnz iovisor/bpftrace Program from file Inline programs
- 15. © 2018 InfluxData. All rights reserved.15 15 @fntlnz bpftrace probe types
- 17. © 2018 InfluxData. All rights reserved.17 17 @fntlnz github.com/fntlnz/kubectl-trace The kubectl trace plugin Your bpftrace program The node where to run it in your cluster Attach the terminal to the program’s TTY
- 18. © 2018 InfluxData. All rights reserved.18 18 @fntlnz github.com/fntlnz/kubectl-trace Run program from file Ctrl-C tells the program to Plot the results using hist() The output histogram
- 19. © 2018 InfluxData. All rights reserved.19 19 @fntlnz kubectl trace and visidata!
- 20. @fntlnz
- 21. © 2018 InfluxData. All rights reserved.21 Any BPF Books to recommend ? David and Jessie are writing one!!
- 22. © 2018 InfluxData. All rights reserved.22 References 1. https://www.iovisor.org/ 2. https://github.com/cilium/cilium 3. https://github.com/iovisor/gobpf 4. https://landlock.io/ 5. https://github.com/iovisor/bpftrace 6. https://github.com/iovisor/bpf-docs 7. https://medium.com/@fntlnz/load-xdp-programs-using-the-ip-iproute2-command-502043898263 8. https://www.youtube.com/watch?v=JRFNIKUROPE 9. https://cilium.readthedocs.io/en/latest/bpf/ 1. iovisor BCC 2. Cilium: HTTP, gRPC, and Kafka Aware Security and Networking for Containers with BPF and XDP 3. iovisor/gobpf - To load eBPF programs using Go 4. Landlock LSM 5. iovisor bpftrace 6. iovisor BPF docs 7. Blog post on how to load xdp programs using iproute2 8. BPF Tracing Talk from Brendan Gregg 9. Cilium documentation for BPF
- 23. @fntlnz Thank you!lorenzo@influxdata.com twitter.com/fntlnz github.com/fntlnz If you’re that kind of person can find my pgp key here: https://fntlnz.wtf/downloads/pubkey-B2400EE4.asc