Unix Programming with Perl 2

Unix Programming with Perl 2 DeNA Co., Ltd. Kazuho Oku

Writing correct code tests aren’t enough tests don’t ensure that the code is correct writing correct code requires… knowledge of perl and knowledge of the OS Oct 15 2011 Unix Programming with Perl 2

Last Year’s Talk Covered these aspects of Unix programming using Perl $! and Errno how to evaluate the errors file handles their internals interaction w. fork(2) Unix signals various signals and how to handle them writing cancellable code Oct 15 2011 Unix Programming with Perl 2

Last Year’s Slides http://www.slideshare.net/kazuho/unix-programming-with-perl Oct 15 2011 Unix Programming with Perl 2

Today’s Talk will cover more advanced topics inter-process communication Unix signals and race condition Oct 15 2011 Unix Programming with Perl 2

IPC::Open3 Oct 15 2011 Unix Programming with Perl 2

IPC::Open3 - Pipe and Deadlock # check syntax errors in perl scripts by using perl –c my @cmd = ( $^X, (map { "-I$_" } grep { !ref $_ } @INC), '-c', $file, ); my $pid = open3(my $cin, my $cout, my $cerr, @cmd); while (waitpid($pid, 0) != $pid) {} ok((WIFEXITED($?) && WEXITSTATUS($?) == 0), $file); http://d.hatena.ne.jp/tokuhirom/20100813/1281666615 Oct 15 2011 Unix Programming with Perl 2

IPC::Open3 - Pipe and Deadlock Q. generally speaking, this code might block. But when? my $pid = open3(my $cin, my $cout, my $cerr, @cmd); while (waitpid($pid, 0) != $pid) {} Oct 15 2011 Unix Programming with Perl 2

IPC::Open3 - Pipe and Deadlock Q. generally speaking, this code might block. But when? my $pid = open3(my $cin, my $cout, my $cerr, @cmd); while (waitpid($pid, 0) != $pid) {} A1. blocks if the child process reads from STDIN Oct 15 2011 Unix Programming with Perl 2

IPC::Open3 - Pipe and Deadlock Close STDIN of the child before calling waitpid my $pid = open3(my $cin, my $cout, my $cerr, @cmd); close $cin; while (waitpid($pid, 0) != $pid) {} Oct 15 2011 Unix Programming with Perl 2

IPC::Open3 - Pipe and Deadlock Q. generally speaking, this code might block. But when? my $pid = open3(my $cin, my $cout, my $cerr, @cmd); while (waitpid($pid, 0) != $pid) {} A1. blocks if the child process reads from STDIN A2. blocks if the child process writes long data to STDOUT or STDERR Oct 15 2011 Unix Programming with Perl 2

Size of the pipe buffer Pipe has size limit cannot write infinitely unless the other peer reads from pipe But actually, how large is the size limit? Oct 15 2011 Unix Programming with Perl 2

Size of the pipe buffer (2) Checking the size of the pipe buffer for (my $sz = 1; ; $sz++) { my @cmd = ( $^X, '-e', qq(print "1"x$sz), ); my $pid = open3(my $cin, my $cout, 0, @cmd) or die $!; while (waitpid($pid, 0) != $pid) {} print "size: $sz\n"; } Oct 15 2011 Unix Programming with Perl 2

Size of the pipe buffer (3) Size of the pipe buffer was… Linux 2.6.32 (x86-64): 65,536 bytes Mac OS X 10.6: 16,384 bytes Size may vary old versions of Linux: 4,096 bytes TCP streams and Unix sockets have configurable buffer size as well Oct 15 2011 Unix Programming with Perl 2

IPC::Open3 - Pipe and Deadlock How should we avoid deadlocks? if we do not need the output of the child process (ex. perl –c), could this be the right answer? my $pid = open3(my $cin, my $cout, my $cerr, @cmd); close $cout; close $cerr; while (waitpid($pid, 0) != $pid) {} ok((WIFEXITED($?) && WEXITSTATUS($?) == 0), $file); Oct 15 2011 Unix Programming with Perl 2

IPC::Open3 - Pipe and Deadlock No. The child process may get killed while trying to write to the output streams closed by the parent process my $pid = open3(my $cin, my $cout, my $cerr, @cmd); close $cout; close $cerr; # if child process tries to write at this moment, it # will be killed by SIGPIPE while (waitpid($pid, 0) != $pid) {} # and as a result, $? may become different ok((WIFEXITED($?) && WEXITSTATUS($?) == 0), $file); Oct 15 2011 Unix Programming with Perl 2

IPC::Open3 - Pipe and Deadlock So how about this? we read all data from the child process my $pid = open3(my $cin, my $cout, my $cerr, @cmd); while (<$cout>) {} while (<$cerr>) {} while (waitpid($pid, 0) != $pid) {} Oct 15 2011 Unix Programming with Perl 2

IPC::Open3 - Pipe and Deadlock No. Will deadlock if the child process writes more bytes than the pipe buffer size to STDERR my $pid = open3(my $cin, my $cout, my $cerr, @cmd); while (<$cout>) {} # may enter deadlock at this point while (<$cerr>) {} while (waitpid($pid, 0) != $pid) {} Oct 15 2011 Unix Programming with Perl 2

IPC::Open3 - Pipe and Deadlock Tip: passing undef as CHLD_ERR will send all output to CHLD_OUT my $pid = open3(my $cin, my $cout, undef , @cmd); while (<$cout>) {} while (waitpid($pid, 0) != $pid) {} Oct 15 2011 Unix Programming with Perl 2

IPC::Open3 – Better to use temporary files Advice: don’t use pipes, use temporary files (unless you need to read the output of the child process while it is running) my $cout = File::Temp->new(); my $pid = do { local *COUT = $cout; open3(my $cin, ’>&COUT’, 0, @cmd); } or die $!; while (waitpid($pid, 0) != $pid) {} seek($cout, 0, SEEK_SET) or die $!; # seek to the start ... Oct 15 2011 Unix Programming with Perl 2

IPC::Open3 vs. open |- Q. Why use IPC::Open3 instead of open ’-| cmd > tmpfile’? A. to skip the shell invocation for speed and security Oct 15 2011 Unix Programming with Perl 2

IPC::Open3 vs. open |- (2) Escaping for open |- is difficult # using IPC::Open3 my @cmd = ( $prog, ’-e’, $arg2, ); my $pid = open3(my $cin, my $cout, 0, @cmd) or die $!; # using open |- my $cmd = ”$prog –e $arg”; # need to escape $arg open my $fh, ’|-’,”$cmd > ” . $tempfh->filename or die $!; Oct 15 2011 Unix Programming with Perl 2

Avoid shell invocation Shell invocation is evil ex. iT○ns upgrade accidentally removes user files failed to quote usernames with a whitespace Direct invocation is safer system($args) => system(@args) open | => IPC::Open2 or IPC::Open3 Oct 15 2011 Unix Programming with Perl 2

Rewriting open |- using IPC::Open3 # the original open(my $fh,’|-’, $cmd) or die $!; print $fh ”hello\n”; close $fh; # TODO: check $? # using IPC::Open3 my $pid = open3(my $fh,’>&STDOUT’, ’>&STDERR’, @cmd) or die $!; print $fh ”hello\n”; close $fh; while (waitpid($pid, 0) != $pid) {} # TODO: check $? Oct 15 2011 Unix Programming with Perl 2

Rewriting open -| using IPC::Open3 # the original open(my $fh,’-|’, $cmd) or die $!; my $line = <$fh>; close $fh; # TODO: check $? # using IPC::Open3 my $pid = open3(my $cin, my $fh, ’>&STDERR’, @cmd) or die $!; close $cin; my $line = <$fh>; close $fh; while (waitpid($pid, 0) != $pid) {} # TODO: check $? Oct 15 2011 Unix Programming with Perl 2

Signal and Race Condition Oct 15 2011 Unix Programming with Perl 2

How to sleep until receiving a signal? Is the code correct? my $gothup = 0; local $SIG{HUP} = sub { $gothup++ }; while (! $gothup) { sleep(30 * 60); # sleep for 30 minutes do_cleanup(); # do some periodical tasks } print ”SIGHUP!\n”; Oct 15 2011 Unix Programming with Perl 2

How to sleep until receiving a signal? (2) a race condition exists my $gothup = 0; local $SIG{HUP} = sub { $gothup++ }; while (! $gothup) { # What happens if a signal is arrives here? # sleep() will sleep 30 minutes since it never # gets interrupted by the signal sleep(30 * 60); # sleep for 30 minutes do_cleanup(); # do some periodical tasks } print ”SIGHUP!\n”; Oct 15 2011 Unix Programming with Perl 2

Use POSIX::pselect SIGHUP is blocked outside of pselect (and thus no race conditions) my $blocked = POSIX::SigSet->new(SIGHUP); my $unblocked = POSIX::SigSet->new(); my $gothup = 0; local $SIG{HUP} = sub { $gothup++ }; sigprocmask(SIG_BLOCK, $blocked, $unblocked); while (! $gothup) { pselect(undef, undef, undef, 30 * 60, $unblocked); do_cleanup(); # do some periodical tasks } print ”SIGHUP!\n”; Oct 15 2011 Unix Programming with Perl 2

The problem of pselect Pselect has race condition on many environments, implemented like… sub select { my ($rset, $wset, $eset, $secs, $mask) = @_; my $oldmask = POSIX::SigSet->new(); sigprocmask(SIG_SETMASK, $mask, $oldmask); $oldmask); my $ret = select($rset, $wset, $eset, $secs); sigprocmask(SIG_SETMASK, $oldmask); $ret; } osx has the problem, glibc on linux does not have the problem but bionic (android) has the problem, … Oct 15 2011 Unix Programming with Perl 2

Using eval & die does not solve the problem my $blocked = POSIX::SigSet->new(SIGHUP); my $unblocked = POSIX::SigSet->new(); sigprocmask(SIG_BLOCK, $blocked, $unblocked); $SIG{HUP} = sub { die ’sighup:’ }; while (! $gothup) { { local $@; eval { sigprocmask(SIG_SETMASK, $unblocked); # what if the signal is delivered at the very moment the # perl interpreter calls sleep(3)? sleep(30 * 60); # sleep for 30 minutes }; $gothup = $@ =~ /^sighup:/; sigprocmask(SIG_SETMASK, $blocked); } do_cleanup(); # do some periodical tasks } Oct 15 2011 Unix Programming with Perl 2

The fix – call syswrite on signal # set unsafe signal handler (see perldoc perlipc) that converts a # signal to a message (Q. proper error handling as advised in this # year and last year’s slides are missing. Can you name them?) socketpair(my $sig_rdr, my $sig_wtr, AF_UNIX, SOCK_STREAM, PF_UNSPEC) or die $!; POSIX::sigaction(SIGHUP, POSIX::SigAction->new(sub { syswrite($sig_wtr, "1", 1) == 1 or die $!; } })); while (1) { my $rfds = ''; vec($rfds, fileno($sig_rdr), 1) = 1; if (select($rfds, undef, undef, 30 * 60) > 0) { sysread($sig_rdr, my $buf, 1) == 1 or die $!; last; } do_cleanup(); # do some periodical tasks } Oct 15 2011 Unix Programming with Perl 2

Summary Oct 15 2011 Unix Programming with Perl 2

Summary buffer size is not infinite, be aware of deadlocks on inter-process / network communication avoid shell invocation, use system(@args) or IPC::Open3 be careful of race conditions when handling Unix signals Oct 15 2011 Unix Programming with Perl 2

Unix Programming with Perl 2

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Unix Programming with Perl 2 (20)

More from Kazuho Oku (20)

Recently uploaded (20)

Unix Programming with Perl 2