A little systemtap
2 likes329 views
This document discusses the Linux tracing tool systemtap. It provides an overview of systemtap and what it can be used for, including tracing system calls, kernel functions, and application functions. It also discusses how systemtap works, how it uses debugging symbols, and how RPMs handle separate debug information files. Several examples are given of using systemtap probes to trace requests for Nginx, cURL, Redis, MySQL, and TCP retransmissions. The document concludes by mentioning using DTrace for other languages beyond C, such as MySQL, Python, and Java.
![跟踪内核态
probe kernel.function("tcp_retransmit_skb") {
rto = tcp_get_info_rto($sk)
saddr = format_ipaddr(__ip_sock_saddr($sk), __ip_sock_family($sk))
daddr = format_ipaddr(__ip_sock_daddr($sk), __ip_sock_family($sk))
sport = __tcp_sock_sport($sk)
dport = __tcp_sock_dport($sk)
lastrto = record[saddr, sport, daddr, dport]
state = tcp_ts_get_info_state($sk)
if (lastrto != rto) {
if (lastrto) {
printf("%s:%d => %s:%d STATE:%s RTO:%d -> %d (ms)n", saddr, sport,
daddr, dport, tcp_sockstate_str(state), lastrto/1000, rto/1000)
} else {
printf("%s:%d => %s:%d STATE:%s RTO:%d (ms)n", saddr, sport,
daddr, dport, tcp_sockstate_str(state), rto/1000)
}
}
record[saddr, sport, daddr, dport] = rto
}](https://image.slidesharecdn.com/alittlesystemtap-161125092737/85/A-little-systemtap-6-320.jpg)
![跟踪应⽤用程序
probe process(“/usr/bin/redis-server”).function(“getConmmand") {
printf(“id:%d %s %sn", $c->id, user_string($c->argv[0]-
>ptr), user_string($c->argv[1]->ptr))
}
id:2 get abcd
id:2 get a](https://image.slidesharecdn.com/alittlesystemtap-161125092737/85/A-little-systemtap-7-320.jpg)
![ngx-req-watch
[root@localhost systemtap-toolkit]# ./ngx-req-watch -p 5614
WARNING: watching /opt/nginx/sbin/nginx(8521 8522 8523 8524) requests
nginx(8523) GET URI:/123?a=123 HOST:127.0.0.1 STATUS:200 FROM 127.0.0.1 FD:16 RT:
0ms
nginx(8523) GET URI:/123?a=123 HOST:127.0.0.1 STATUS:200 FROM 127.0.0.1 FD:16 RT:
0ms](https://image.slidesharecdn.com/alittlesystemtap-161125092737/85/A-little-systemtap-19-320.jpg)
![libcurl-watch-req
[root@localhost systemtap-toolkit]# ./libcurl-watch-req
WARNING: Tracing libcurl (0) ...
curl(23759) URL:http://www.google.com RT:448(ms) RTCODE:0
curl(23767) URL:http://www.facebook.com/asdfasdf RT:596(ms) RTCODE:0
curl(23769) URL:https://www.facebook.com/asdfasdf RT:902(ms) RTCODE:0](https://image.slidesharecdn.com/alittlesystemtap-161125092737/85/A-little-systemtap-20-320.jpg)
![redis-watch-req
[root@localhost systemtap-toolkit]# ./redis-watch-req -p 23261
WARNING: watching /usr/bin/redis-server(23261) requests
redis-server(23261) RT:30(us) REQ: id:2 fd:5 ==> get a #-1 RES: #9
redis-server(23261) RT:23(us) REQ: id:2 fd:5 ==> set a #12 RES: #5](https://image.slidesharecdn.com/alittlesystemtap-161125092737/85/A-little-systemtap-21-320.jpg)
![pdomysql-watch-query
[root@localhost systemtap-toolkit]# ./pdomysql-watch-query -l /usr/lib64/php/modules/
pdo_mysql.so
Tracing pdo-mysql (0)
php-fpm(12896) 172.17.10.196:3306@root: SELECT * from person RT:0(ms) RTCODE:1
php-fpm(12896) 172.17.10.196:3306@root: SELECT * from person RT:8(ms) RTCODE:1
php-fpm(12896)172.17.10.196:3306@root: SELECT sleep(5) RT:5012(ms) RTCODE:1](https://image.slidesharecdn.com/alittlesystemtap-161125092737/85/A-little-systemtap-22-320.jpg)
![TCP-Retrans
[root@localhost systemtap-toolkit]# ./tcp-retrans
Printing tcp retransmission
10.0.2.15:49896 -> 172.17.9.41:80 state:TCP_SYN_SENT rto:0 -> 1000 ms
10.0.2.15:49896 -> 172.17.9.41:80 state:TCP_SYN_SENT rto:1000 -> 2000 ms
10.0.2.15:49896 -> 172.17.9.41:80 state:TCP_SYN_SENT rto:2000 -> 4000 ms](https://image.slidesharecdn.com/alittlesystemtap-161125092737/85/A-little-systemtap-23-320.jpg)
A little systemtap
- 2. 什什么是 systemtap Linux 下万能的 Trace ⼯工具。
- 5. 跟踪系统调⽤用 stap -e 'probe syscall.open {printf("%s(%d) is opening %sn", execname(), pid(), user_string($filename))}' vminfo(710) is opening /var/run/utmp less(11933) is opening /root/.lesshst cat(12198) is opening /etc/ld.so.cache cat(12198) is opening /lib64/libc.so.6 cat(12198) is opening /usr/lib/locale/locale-archive cat(12198) is opening /tmp/123
- 6. 跟踪内核态 probe kernel.function("tcp_retransmit_skb") { rto = tcp_get_info_rto($sk) saddr = format_ipaddr(__ip_sock_saddr($sk), __ip_sock_family($sk)) daddr = format_ipaddr(__ip_sock_daddr($sk), __ip_sock_family($sk)) sport = __tcp_sock_sport($sk) dport = __tcp_sock_dport($sk) lastrto = record[saddr, sport, daddr, dport] state = tcp_ts_get_info_state($sk) if (lastrto != rto) { if (lastrto) { printf("%s:%d => %s:%d STATE:%s RTO:%d -> %d (ms)n", saddr, sport, daddr, dport, tcp_sockstate_str(state), lastrto/1000, rto/1000) } else { printf("%s:%d => %s:%d STATE:%s RTO:%d (ms)n", saddr, sport, daddr, dport, tcp_sockstate_str(state), rto/1000) } } record[saddr, sport, daddr, dport] = rto }
- 7. 跟踪应⽤用程序 probe process(“/usr/bin/redis-server”).function(“getConmmand") { printf(“id:%d %s %sn", $c->id, user_string($c->argv[0]- >ptr), user_string($c->argv[1]->ptr)) } id:2 get abcd id:2 get a
- 8. systemtap 安装 yum install -y systemtap 早先的内核版本依赖utrace补丁 3.50 使⽤用内核 uprobes 和 uretprobes
- 12. stap -L 'process("/usr/bin/redis- server").function("zzlPrev")' process("/usr/bin/redis-server").function("zzlPrev@/usr/ src/debug/redis-2.8.19/src/t_zset.c:720") $zl:unsigned char* $eptr:unsigned char** $sptr:unsigned char**
- 13. 分离的 debug info 因为程序带上调试符号会⾮非常⼤大，⽽而且不不是所有的⽤用户需要调试符号。 所以有了了分离的 debug info。
- 14. RPM的机制 /usr/lib/rpm/find-debuginfo.sh --strict-build-id xx extracting debug info from xx
- 15. RPM的机制 奥秘在于⼆二进制中的 build-id objcopy xx xx.debug strip --strip-debug ./xx
- 16. RPM的机制 奥秘在于程序中的 build-id objcopy xx xx.debug strip --strip-debug ./xx objdump -s -j .note.gnu.build-id /usr/bin/redis-server
- 19. ngx-req-watch [root@localhost systemtap-toolkit]# ./ngx-req-watch -p 5614 WARNING: watching /opt/nginx/sbin/nginx(8521 8522 8523 8524) requests nginx(8523) GET URI:/123?a=123 HOST:127.0.0.1 STATUS:200 FROM 127.0.0.1 FD:16 RT: 0ms nginx(8523) GET URI:/123?a=123 HOST:127.0.0.1 STATUS:200 FROM 127.0.0.1 FD:16 RT: 0ms
- 20. libcurl-watch-req [root@localhost systemtap-toolkit]# ./libcurl-watch-req WARNING: Tracing libcurl (0) ... curl(23759) URL:http://www.google.com RT:448(ms) RTCODE:0 curl(23767) URL:http://www.facebook.com/asdfasdf RT:596(ms) RTCODE:0 curl(23769) URL:https://www.facebook.com/asdfasdf RT:902(ms) RTCODE:0
- 21. redis-watch-req [root@localhost systemtap-toolkit]# ./redis-watch-req -p 23261 WARNING: watching /usr/bin/redis-server(23261) requests redis-server(23261) RT:30(us) REQ: id:2 fd:5 ==> get a #-1 RES: #9 redis-server(23261) RT:23(us) REQ: id:2 fd:5 ==> set a #12 RES: #5
- 22. pdomysql-watch-query [root@localhost systemtap-toolkit]# ./pdomysql-watch-query -l /usr/lib64/php/modules/ pdo_mysql.so Tracing pdo-mysql (0) php-fpm(12896) 172.17.10.196:3306@root: SELECT * from person RT:0(ms) RTCODE:1 php-fpm(12896) 172.17.10.196:3306@root: SELECT * from person RT:8(ms) RTCODE:1 php-fpm(12896)172.17.10.196:3306@root: SELECT sleep(5) RT:5012(ms) RTCODE:1
- 23. TCP-Retrans [root@localhost systemtap-toolkit]# ./tcp-retrans Printing tcp retransmission 10.0.2.15:49896 -> 172.17.9.41:80 state:TCP_SYN_SENT rto:0 -> 1000 ms 10.0.2.15:49896 -> 172.17.9.41:80 state:TCP_SYN_SENT rto:1000 -> 2000 ms 10.0.2.15:49896 -> 172.17.9.41:80 state:TCP_SYN_SENT rto:2000 -> 4000 ms
- 24. beyond c land
- 25. MySQL DTRACE
- 26. Python DTRACE
- 27. Java DTRACE
- 28. Q&A