strace for Perl Mongers
5 likes2,490 views
This document discusses using strace to trace system calls made by Perl programs. It provides examples of using strace to trace a web application's interactions with a database and memcached. Specifically, it shows strace output for MySQL and memcached requests, and analyzes strace logs to diagnose a slow cache retrieval issue and the "thundering herd problem".
![A. HTTP鮧¸Çž*]r
HTTPë¸Üò¸ž4e_tl
”„](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-17-320.jpg)
![Q. HTTP鮧¸Çž*]r
HTTPë¸Üò¸ž4eoru
RPR_tl”„](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-18-320.jpg)
![ŠNgm®]](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-20-320.jpg)
![Web¡ÖézV›Ì
1. HTTP鮧¸Çž*]—
2. _zw‰•zÈèÝXN—
MySQL„z¡®º¸
memcached„z¡®º¸
(Æ–¾z^|logz„Y‹etc)
3. /_zw‰•zÈèÝXN—
4. HTTPë¸Üò¸ž4e](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-21-320.jpg)
![¶¸Åß²–ê§w™e—t...
1. int fd = accept(...); // 鮧¸Ç*]r
2. read(fd, ...);{{{// 鮧¸ÇzÆ?Ÿs
3. _zw‰•zÈèÝXN—
socket, connect, write, read, close
open, write, read, close
4. /_zw‰•zÈèÝXN—
5. write(fd, ...);{{{// ë¸Üò¸4e](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-22-320.jpg)
![a•w»ƒe—t
1. int fd = accept(...); // 鮧¸Ç*]r
2. read(fd, ...);{{{// 鮧¸ÇzÆ?Ÿs
3. ‰•zÈèÝ /
4. write(fd, ...);{{{// ë¸Üò¸4e](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-23-320.jpg)
![fdžðRzXè2z2˜
Ô £êzf#
opens4orYkfdXclosewUa˜—Šs
®è£¡òÇ´£Èsz¼°ÂÇzf#
sockets4orYkfdXclosewUa˜—Šs
´–Ï´£ÈwV]—®è£¡òǼ°ÂÇzf#
accepts4orYkfdXclosewUa˜—Šs](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-30-320.jpg)
![Web¡ÖézstracežŒÔz¬Úò
// こっから
accept(4, {sa_family=AF_INET, ...)}, [16]) = 5
(中略)
read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308
(中略)
write(5, HTTP/1.1 200 OKrnDate: Sun, 10 A..., 2218) = 2218
// ここまでが一つのリクエスト/レスポンス
// keepaliveしてないならここで close](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-31-320.jpg)
![Web¡ÖézV›Ì($)
lPkPzÝ „Ðzó
sþUwv—
HTTP鮧¸Çž*]—
_zw‰•zÈèÝXN—
MySQL„z¡®º¸
memcached„z¡®º¸
(Æ–¾z^|logz„Y‹etc)
/_zw‰•zÈèÝXN—
HTTPë¸Üò¸ž4e](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-34-320.jpg)
![¶öÜ£òÇ - MySQL
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 10
fcntl(10, F_SETFL, O_RDONLY) = 0
fcntl(10, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(10, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(10, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(10, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr(10.0.0.1
fcntl(10, F_SETFL, O_RDWR) = 0
poll([{fd=10, events=POLLIN|POLLPRI}], 1, 4000) = 1 ([{fd=10, revents=POLLIN}
setsockopt(10, SOL_SOCKET, SO_RCVTIMEO, 200334110000000setsockopt(10, SOL_SOCKET, SO_SNDTIMEO, 200334110000000setsockopt(10, SOL_IP, IP_TOS, [8], 4) = 0
setsockopt(10, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(10, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-35-320.jpg)
![¶öÜ£òÇ - memcached
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 8
fcntl(8, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(8, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(8, {sa_family=AF_INET, sin_port=htons(11211), sin_addr=inet_addr(10.0.0.1
poll([{fd=8, events=POLLOUT}], 1, 250) = 1 ([{fd=8, revents=POLLOUT}])
getsockopt(8, SOL_SOCKET, SO_ERROR, [-511598409602301952], [4]) = 0
setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0
sendmsg(8, {msg_name(0)=NULL, msg_iov(6)=[{set, 3}, { csrf_token:, 20},
read(8, 0x59f9f50, 1536) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=8, events=POLLIN}], 1, 300) = 1 ([{fd=8, revents=POLLIN}])
read(8, STOREDrn, 1536)](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-36-320.jpg)
![+180ms
tN—APIzë¸Üò¸X|éé–¸ w180msËB
50ms or diev•4x[•P¶vvPtP]vPëØê](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-40-320.jpg)
![accept(2) and prefork server
DÖ캸X•Ôw´pzsocket fdžacceptc|i˜j
˜Õì®
Linux kernelX”cvwqueueingcr[˜—zs|鮧
¸ÇXakÔwOYÖX—Ö캸{´pl]
(ˆæ)
http://d.hatena.ne.jp/naoya/20070311/1173629378](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-51-320.jpg)
![accept(2) thundering herd
accept(2)zó–XÖ캸Dt鮧¸ÇDzÀ]‘s
Pf
loadXÖX—
vmstatsf—t...
ustsytcsz–Xg–ÖX— (50, 50, 100000tW)
Ö캸zcontext switch
TeZXOÀWt›˜—](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-53-320.jpg)
![7•
xæz¡–¬Å®ÀâltÇtuRc”RŽvPDXcr
P—
Monoceros‹kPwaccepte—Ö캸t鮧¸Çž
{ve—Ö캸žñÀw
]—tWƒ
event-driventÝêÀÖ캸{øPgXuP with Perl
IO::AIO (AnyEvent::IOzbackend) tWŽÇt¶ò¯êÖ
캸¼völok–...](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-57-320.jpg)
![stracez”Pt_™
̼XÓPs|Ö캸wattache—l]sef˜
—
Web¡Öé°–¶æòzDÓmX
W—](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-60-320.jpg)
![stracelt
W•vP_t
¶¸Åß²–êž]›vPvCPUÏ¥òÈvÏ7v{v
¡Öé°–¶æòszÏ7loop, deep recursion
µzuP¡ê³é¹ßtW
-d NYTProf
Çèò¸Ü–Çä’AszþU
tcpdumptWtsharktWngreptWžPŠc“R](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-61-320.jpg)
![straces
W—_t
鮧¸Ç*]r|ë¸Üò¸4eŠszÔì–
Æ–¾¡®º¸zöA
öUwDxmemcachedW•getcr—W|DxDB‹w
Por—W
öUwcachea˜rP—data
öUwPícrP—SELECT
iz)|ŽP z6crPvP“”](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-62-320.jpg)
![H
|k–¼l]ustrace{¸ÿzÖêdvP
žT|ÚàW•ä—tWiŸv_tŽvP
sŽ|žP[X
WorP˜|V7€¦
Web¡ÖézDÓmX
W—](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-63-320.jpg)
![ˆæ`
The Linux Programing Interface
ŠPc|K¨„€h(÷§cWvP]u...)Vee
à˜§hŽN—]u1000Ù–·Ý|pŠ–„€s{v[
ï1
ÀóŒ«Pvc|éÔ ëò¸tcrŽ6q
äz˜{Veel]u|±¿Œz{ÞXûPzs
pRzLinuxÖì¯èÞò¯
tWz[X?|{PPWŽc
˜vP](https://image.slidesharecdn.com/straceforperlmongers-140830075349-phpapp02/85/strace-for-Perl-Mongers-64-320.jpg)
strace for Perl Mongers
- 1. strace for Perl Mongers zentooo @ DeNA
- 2. ß…Ãu
- 4. °ntDtÒ–ê±
- 6. p€g§a†ó
- 10. ¶¸Åß²–ê{|¡Öé°–¶æòX j[OSz API File I/O (open, lseek, write, read, close) Network (socket, bind, listen, accept, connect) Memory (brk, sbrk, mmap) Process (fork, wait, kill) Thread (pthread_create)
- 14. ¡Öé°–¶æò{OSzzzÖ
- 15. ±¿seX
- 18. Q. HTTP鮧¸Çž*]r HTTPë¸Üò¸ž4eoru RPR_tl”„
- 20. ŠNgm®]
- 21. Web¡ÖézV›Ì 1. HTTP鮧¸Çž*]— 2. _zw‰•zÈèÝXN— MySQL„z¡®º¸ memcached„z¡®º¸ (Æ–¾z^|logz„Y‹etc) 3. /_zw‰•zÈèÝXN— 4. HTTPë¸Üò¸ž4e
- 22. ¶¸Åß²–ê§w™e—t... 1. int fd = accept(...); // 鮧¸Ç*]r 2. read(fd, ...);{{{// 鮧¸ÇzÆ?Ÿs 3. _zw‰•zÈèÝXN— socket, connect, write, read, close open, write, read, close 4. /_zw‰•zÈèÝXN— 5. write(fd, ...);{{{// ë¸Üò¸4e
- 23. a•w»ƒe—t 1. int fd = accept(...); // 鮧¸Ç*]r 2. read(fd, ...);{{{// 鮧¸ÇzÆ?Ÿs 3. ‰•zÈèÝ / 4. write(fd, ...);{{{// ë¸Üò¸4e
- 24. DEMO #1 - just to strace
- 25. stracežŒÔz¬Úò (1) ↓システムコール名 返り値↓ read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308 ↑引数（バッファなどの場合はよしなに展開される）
- 26. stracežŒÔz¬Úò (2) fd (Ô £êÆ¢¸®éÖ¾) w¶öe— ↓コレ read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308
- 27. fd (Ô £êÆ¢¸®éÖ¾) Ô £êÌÂÇî–®¼°ÂÇvuž³ÎBckÖ9 „zÜ£ò¾ (0, 1, 2) = (stdin, stdout, stderr) i˜’ø{«zw•˜—
- 28. fd with file int fd = open(/tmp/yapc2014, ...); read(fd, buffer); write(fd, buffer); close(fd);
- 29. fd with client socket int fd = socket(); connect(fd, addr, ...); read(fd, buffer); write(fd, buffer); close(fd);
- 30. fdžðRzXè2z2˜ Ô £êzf# opens4orYkfdXclosewUa˜—Šs ®è£¡òÇ´£Èsz¼°ÂÇzf# sockets4orYkfdXclosewUa˜—Šs ´–Ï´£ÈwV]—®è£¡òǼ°ÂÇzf# accepts4orYkfdXclosewUa˜—Šs
- 31. Web¡ÖézstracežŒÔz¬Úò // こっから accept(4, {sa_family=AF_INET, ...)}, [16]) = 5 (中略) read(5, GET / HTTP/1.0rnHost: xxx-xxxxxx..., 131072) = 308 (中略) write(5, HTTP/1.1 200 OKrnDate: Sun, 10 A..., 2218) = 2218 // ここまでが一つのリクエスト/レスポンス // keepaliveしてないならここで close
- 32. DEMO #2 - strace web app
- 33. N[Šs2˜vzsº“Ž Twiggy, Twiggy::Preforkvuevent-drivenv´–Ï (epoll_wait) multi socket listencr—ÔzStarlet (select)
- 34. Web¡ÖézV›Ì($) lPkPzÝ „Ðzó sþUwv— HTTP鮧¸Çž*]— _zw‰•zÈèÝXN— MySQL„z¡®º¸ memcached„z¡®º¸ (Æ–¾z^|logz„Y‹etc) /_zw‰•zÈèÝXN— HTTPë¸Üò¸ž4e
- 35. ¶öÜ£òÇ - MySQL socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 10 fcntl(10, F_SETFL, O_RDONLY) = 0 fcntl(10, F_GETFL) = 0x2 (flags O_RDWR) fcntl(10, F_GETFL) = 0x2 (flags O_RDWR) fcntl(10, F_SETFL, O_RDWR|O_NONBLOCK) = 0 connect(10, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr(10.0.0.1 fcntl(10, F_SETFL, O_RDWR) = 0 poll([{fd=10, events=POLLIN|POLLPRI}], 1, 4000) = 1 ([{fd=10, revents=POLLIN} setsockopt(10, SOL_SOCKET, SO_RCVTIMEO, 200334110000000setsockopt(10, SOL_SOCKET, SO_SNDTIMEO, 200334110000000setsockopt(10, SOL_IP, IP_TOS, [8], 4) = 0 setsockopt(10, SOL_TCP, TCP_NODELAY, [1], 4) = 0 setsockopt(10, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
- 36. ¶öÜ£òÇ - memcached socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 8 fcntl(8, F_GETFL) = 0x2 (flags O_RDWR) fcntl(8, F_SETFL, O_RDWR|O_NONBLOCK) = 0 connect(8, {sa_family=AF_INET, sin_port=htons(11211), sin_addr=inet_addr(10.0.0.1 poll([{fd=8, events=POLLOUT}], 1, 250) = 1 ([{fd=8, revents=POLLOUT}]) getsockopt(8, SOL_SOCKET, SO_ERROR, [-511598409602301952], [4]) = 0 setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0 sendmsg(8, {msg_name(0)=NULL, msg_iov(6)=[{set, 3}, { csrf_token:, 20}, read(8, 0x59f9f50, 1536) = -1 EAGAIN (Resource temporarily unavailable) poll([{fd=8, events=POLLIN}], 1, 300) = 1 ([{fd=8, revents=POLLIN}]) read(8, STOREDrn, 1536)
- 37. DEMO #3 - strace web app with DB and memcached
- 38. Öì¯èß{oktV–w ”WvP •d®§éžDxPcr— ork”–cache misscr— Ï7wretrycr— TCP ConnectionžPŠ›grvP TCP ConnectionmŸtˆ˜rvP
- 39. öº #1 - memcached retry problem
- 40. +180ms tN—APIzë¸Üò¸X|éé–¸ w180msËB 50ms or diev•4x[•P¶vvPtP]vPëØê
- 42. M[èo strace -p $pid -e 'trace=sendmsg' (Cache::Memcached::FastXsendmsgžžRz{—orPk)
- 43. Ë sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13} sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13} sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13} sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_local:, 13} sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote1:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:, sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { cache_remote2:,
- 44. ̾|•c èÂЖá·ä–êwretryAÿ (maxs3x) XâorPk Cache::Memcached::Fastzget{cache misstiz)z§ è–žæ*sYvP cache missz_w3xretrycrk retry¹{20msec 20 msec × 3 (retry) × 3 = 180 msec izhit rate{uRvŸƒtPRø{N–ppŽget,{ retrycvP7cžâ˜rwE
- 45. Ë - with timestamp 22:45:34.517703 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.537967 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.558936 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.579275 sendmsg(6, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, { 22:45:34.587428 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:34.608296 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:34.628673 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:34.649146 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:35.135129 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:35.155516 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:35.176027 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3}, 22:45:35.196453 sendmsg(11, {msg_name(0)=NULL, msg_iov(4)=[{get, 3},
- 47. ÙžpWvPÖê strace tcpdump gdb
- 48. Ùžp[Žz zM6 ¡Öézì¯ Ôw{¡Öéz²–È
- 49. öº #2 - accept(2) thundering herd
- 50. What is thundering herd The thundering herd problem occurs when a large number of processes waiting for an event are awoken when that event occurs, but only one process is able to proceed at a time. - Wikipedia *[zÖ캸X§zÇé«–s•Ôw”YceX|ö Uz{v{´zÖ캸scW훘vP (Šk{í› ˜—…Y) Ôwé¼–¸XÏ7wv—þU ”[N—z{Æ–¾zremote cacheXexpireckÔw´D wDB„z®§éX—|tW
- 51. accept(2) and prefork server DÖ캸X•Ôw´pzsocket fdžacceptc|i˜j ˜Õì® Linux kernelX”cvwqueueingcr[˜—zs|鮧 ¸ÇXakÔwOYÖX—Ö캸{´pl] (ˆæ) http://d.hatena.ne.jp/naoya/20070311/1173629378
- 52. accept(2) and event-driven/ multi-process server DÖ캸X•Ôw´pzsocket fdžepoll_waitsÁ listen socketXreadIÿ = acceptXIÿwvokÔFs epoll_waitcr—ÀrzÖ캸XOYÖX— ´pzÖ캸Xacceptw´ )zÖ캸{óorî (EAGAIN)
- 53. accept(2) thundering herd accept(2)zó–XÖ캸Dt鮧¸ÇDzÀ]‘s Pf loadXÖX— vmstatsf—t... ustsytcsz–Xg–ÖX— (50, 50, 100000tW) Ö캸zcontext switch TeZXOÀWt›˜—
- 54. DEMO #3 - EAGAIN and again and again
- 57. 7• xæz¡–¬Å®ÀâltÇtuRc”RŽvPDXcr P— Monoceros‹kPwaccepte—Ö캸t鮧¸Çž {ve—Ö캸žñÀw ]—tWƒ event-driventÝêÀÖ캸{øPgXuP with Perl IO::AIO (AnyEvent::IOzbackend) tWŽÇt¶ò¯êÖ ìº¸¼völok–...
- 58. Štwâ–Še
- 59. stracez›—Pt_™ ß¿váQXVVP (ø˜˜|uRwsŽv—X) kŠwO_—,zÌÎw{¹PrvP (Devel::KYTProftW sì¯cck[X”Pƒ¥
- 60. stracez”Pt_™ ̼XÓPs|Ö캸wattache—l]sef˜ — Web¡Öé°–¶æòzDÓmX W—
- 61. stracelt W•vP_t ¶¸Åß²–êž]›vPvCPUÏ¥òÈvÏ7v{v ¡Öé°–¶æòszÏ7loop, deep recursion µzuP¡ê³é¹ßtW -d NYTProf Çèò¸Ü–Çä’AszþU tcpdumptWtsharktWngreptWžPŠc“R
- 62. straces W—_t 鮧¸Ç*]r|ë¸Üò¸4eŠszÔì– Æ–¾¡®º¸zöA öUwDxmemcachedW•getcr—W|DxDB‹w Por—W öUwcachea˜rP—data öUwPícrP—SELECT iz)|ŽP z6crPvP“”
- 63. H |k–¼l]ustrace{¸ÿzÖêdvP žT|ÚàW•ä—tWiŸv_tŽvP sŽ|žP[X WorP˜|V7€¦ Web¡ÖézDÓmX W—
- 64. ˆæ` The Linux Programing Interface ŠPc|K¨„€h(÷§cWvP]u...)Vee à˜§hŽN—]u1000Ù–·Ý|pŠ–„€s{v[ ï1 ÀóŒ«Pvc|éÔ ëò¸tcrŽ6q äz˜{Veel]u|±¿Œz{ÞXûPzs pRzLinuxÖì¯èÞò¯ tWz[X?|{PPWŽc ˜vP
- 65. ™G (?) Web+DBz2014ñ10KÿwMÌ(Perl Hackers Hub)ž„Wg rá[_twv–Šck
- 66. `s×N–XtR`bPŠc k