progress

Adding Progress Bars to Your Scripts http://oreilly.com/pub/h/943

Sign In/My Account | View Cart
Press
Book List Learning Lab PDFs O'Reilly GearNewsletters Jobs
Room

Buy the book!

HACK Adding Progress Bars to Your Scripts
Give a visual indication that a download is progressing smoothly
#18 The Code
[Discuss (3) | Link to this hack]

With all this downloading, it's often helpful to have some visual representation of its progress. In By Kevin Hemenway,
most of the scripts in this book, there's always a bit of visual information being displayed to the Tara Calishain
screen: that we're starting this URL here, processing this data there, and so on. These helpful bits October 2003
usually come before or after the actual data has been downloaded. But what if we want visual More Info
feedback while we're in the middle of a large MP3, movie, or database leech?

If you're using a fairly recent vintage of the LWP library, you'll be able to interject your own subroutine to run at regular
intervals during download. In this hack, we'll show you four different ways of adding various types of progress bars to
your current applications. To get the most from this hack, you should have ready a URL that's roughly 500 KB or larger;
it'll give you a good chance to see the progress bar in action.

The Code

The first progress bar is the simplest, providing only a visual heartbeat so that you can be sure things are progressing and
not just hanging. Save the following code to a file called progress_bar.pl and run it from the command line as perl
scriptnameURL, where URL is the online location of your appropriately large piece of sample data:

#!/usr/bin/perl -w
#
# Progress Bar: Dots - Simple example of an LWP progress bar.
# http://disobey.com/d/code/ or contact morbus@disobey.com.
#
# This code is free software; you can redistribute it and/or
# modify it under the same terms as Perl itself.
#

use strict; $|++;
my $VERSION = "1.0";

# make sure we have the modules we need, else die peacefully.
eval("use LWP 5.6.9;"); die "[err] LWP 5.6.9 or greater required.n" if $@;

# now, check for passed URLs for downloading.
die "[err] No URLs were passed for processing.n" unless @ARGV;

# our downloaded data.
my $final_data = undef;

# loop through each URL.
foreach my $url (@ARGV) {
print "Downloading URL at ", substr($url, 0, 40), "... ";

# create a new useragent and download the actual URL.
# all the data gets thrown into $final_data, which
# the callback subroutine appends to.
my $ua = LWP::UserAgent->new( );
my $response = $ua->get($url, ':content_cb' => &callback, );
print "n"; # after the final dot from downloading.
}

# per chunk.
sub callback {
my ($data, $response, $protocol) = @_;
$final_data .= $data;
print ".";
}

None of this code is particularly new, save the addition of our primitive progress bar. We use LWP's standard get method,
but add the :content_cb header with a value that is a reference to a subroutine that will be called at regular intervals as
our content is downloaded. These intervals can be suggested with an optional :read_size_hint, which is the number
of bytes you'd like received before they're passed to the callback.

1 di 5 20/06/2009 11.56


Search In this example, we've defined that the data should be sent to a subroutine named callback. You'll notice that the
Go routine receives the actual content, $data, that has been downloaded. Since we're overriding LWP's normal
$response->content or :content_file features, we now have to take full control of the data. In this hack, we
Hacks Site store all our results in $final_data, but we don't actually do anything with them.
• List of Titles
• Got a Hack? Most relevant, however, is the print statement within the callback routine. This is our first pathetic attempt at visual
• Suggestion Box feedback during the downloading process: every time a chunk of data gets sent our way, we spit out a dot. If the total data
size is sufficiently large, our screen will be filled with dots, dots, and more dots:
Resource Centers
Bioinformatics
Downloading URL at http://disobey.com/large_file.mov...
C/C++ ..........................................................................
Databases ..........................................................................
Digital Media ..........................................................................
Enterprise Development ..........................................................................
Game Development ..........................................................................
Java .....................................................................
Linux/Unix
Macintosh/OS X While useful, it's certainly not very pretty, and it can be especially disruptive for large files (the previous example is the
.NET output of downloading just 700 KB). Instead, how about we use a little primitive animation?
Open Source
Oracle If you've worked in the shell or installed various programs (or even a retail version of Linux), you may have seen rotating
Perl cursors built from ASCII letters. These cursors could start at , erase that character, draw a |, erase, /, erase, -, and then
Python to restart the loop. Individually, and without the benefit of a flipbook, these look pretty boring. Onscreen, however, they
Scripting
create a decent equivalent to an hourglass or spinning ball.
Security
Software Development
Modify the previous script, adding the highlighted lines:
SysAdmin/Networking
Web
...
Web Services
# our downloaded data.
Windows
my $final_data = undef;
Wireless
XML
# your animation and counter.
my $counter; my @animation = qw( | / - );
Book Series
Annoyances # loop through each URL.
CD Bookshelves foreach my $url (@ARGV)
Cookbooks ...
Developer's Notebooks
Hacks This initializes a counter and creates an array that contains the frames of our animations. As you can see, we use the same
Head First frames we discussed earlier. If you don't like `em, customize your own (perhaps . i l i). The last change we need to
In A Nutshell make is in our callback routine. Swap out the existing print "." with:
Missing Manuals
Pocket References print "$animation[$counter++]b";
Personal Trainer $counter = 0 if $counter == scalar(@animation);
Technology & Society
And that's it. For each chunk of data we receive, the next frame of the animation will play. When our counter is the same
Publishing Partners
as the number of frames, we reset and begin anew. Obviously, we can't show a readily apparent example of what this looks
Mandriva
No Starch Press
like, so try it at your leisure.
Paraglyph Press
PC Publishing
We can still do better, though. We've certainly removed the distracting dot distortion, but we're still left with only simple
Pragmatic Bookshelf output; we don't have raw information on how far we've gone and how far still to go. The following code provides a
SitePoint progress meter with a visual percentage bar, as well as a numerical reading:
Syngress Publishing

Online Publications
LinuxDevCenter.com
MacDevCenter.com
ONJava.com
ONLamp.com
OpenP2P.com
Perl.com
WebServices.XML.com
WindowsDevCenter.com
XML.com

Special Interest
Beta Chapters
Events
From the Editors List
Letters
MAKE
Open Books
tim.oreilly.com

Special Sales
Academic
Corporate Services
Government

Inside O'Reilly

2 di 5 20/06/2009 11.56


About O'Reilly
Bookstores #!/usr/bin/perl -w
Contact Us #
International # Progress Bar: Wget - Wget style progress bar with LWP.
Register Your Book # http://disobey.com/d/code/ or contact morbus@disobey.com.
User Groups # Original routine by tachyon at http://tachyon.perlmonk.org/
Writing for O'Reilly #
#

use strict; $|++;

eval("use LWP 5.6.9;"); die "[err] LWP 5.6.9 or greater required.n" if $@;


# happy golucky variables.
my $final_data; # our downloaded data.
my $total_size; # total size of the URL.

print "Downloading URL at ", substr($url, 0, 40), "...n";

# the callback subroutine appends to. before that,
# though, get the total size of the URL in question.
my $result = $ua->head($url);
my $remote_headers = $result->headers;
$total_size = $remote_headers->content_length;

# now do the downloading.
my $response = $ua->get($url, ':content_cb' => &callback );
}

# per chunk.
sub callback {
print progress_bar( length($final_data), $total_size, 25, '=' );
}

# wget-style. routine by tachyon
# at http://tachyon.perlmonk.org/
sub progress_bar {
my ( $got, $total, $width, $char ) = @_;
$width ||= 25; $char ||= '=';
my $num_width = length $total;
sprintf "|%-${width}s| Got %${num_width}s bytes of %s (%.2f%%)r",
$char x (($width-1)*$got/$total). '>',
$got, $total, 100*$got/+$total;
}

You'll notice right off the bat that we've added another subroutine at the bottom of our code. Before we get into that, check
out our actual LWP request. Instead of just asking for the data, we first check the HTTP headers to see the size of the file
we'll be downloading. We store this size in a $total_size variable. It plays an important part in our new subroutine,
best demonstrated with a sample:

|=============> | Got 422452 bytes of 689368 (61.28%)

This is sprintf magic at work, thanks to a little magic from tachyon over at Perl Monks
(http://www.perlmonks.org/index.pl?node_id=80749). As each chunk of data gets sent to our callback, the display is
updated both as a bar and as a byte count and percentage. It's a wonderful piece of work and my preferred progress bar as
of this writing. As you can see in the progress_bar line of the callback, you can modify the width as well as the
character.

So far, we've rolled our own, but there is a module on CPAN, Term::ProgressBar
(http://search.cpan.org/author/FLUFFY/Term-ProgressBar), that takes care of the lion's share of the work for us. It has a
bit more functionality than sprintf, such as titling the progress bar, including an ETA, and growing to the length of the
user's terminal width. Here it is in action:

3 di 5 20/06/2009 11.56


#!/usr/bin/perl -w
#
# Progress Bar: Term::ProgressBar - progress bar with LWP.
# http://disobey.com/d/code/ or contact morbus@disobey.com.
# Original routine by tachyon at http://tachyon.perlmonk.org/
#
#

use strict; $|++;

eval("use LWP 5.6.9;");
die "[err] LWP is not the required version.n" if $@;
eval("use Term::ProgressBar;"); # prevent word-wrapping.
die "[err] Term::ProgressBar not installed.n" if $@;


# happy golucky variables.
my $final_data = 0; # our downloaded data.
my $total_size; # total size of the URL.
my $progress; # progress bar object.
my $next_update = 0; # reduce ProgressBar use.

print "Downloading URL at ", substr($url, 0, 40), "...n";

# the callback subroutine appends to. before that,
# though, get the total size of the URL in question.
my $result = $ua->head($url);
my $remote_headers = $result->headers;
$total_size = $remote_headers->content_length;

# initialize our progress bar.
$progress = Term::ProgressBar->new({count => $total_size, ETA => &return;
'linear'});
$progress->minor(0); # turns off the floating asterisks.
$progress->max_update_rate(1); # only relevant when ETA is used.

# now do the downloading.
my $response = $ua->get($url, ':content_cb' => &callback );

# top off the progress bar.
$progress->update($total_size);
}

# per chunk.
sub callback {

# reduce usage, as per example 3 in POD.
$next_update = $progress->update(length($final_data))if length($final_data) >= $ne
}

And here's its output:

13% [======== ]9m57s Left

More examples are available in the Term::ProgressBar documentation.

Comment on this hack
You must be logged in to the O'Reilly Network to post a comment.

Showing
messages 1 through 3 of 3.

Warning
2005-11-06 04:02:02 tiberido [Reply | View]

4 di 5 20/06/2009 11.56


I got an

Undefined subroutine &main::return called at ScriptProgBar.pl line 44.

In the last script (Term::ProgressBar). I deleted &return

$progress = Term::ProgressBar->new({count => $total_size, ETA => 'linear'});

An now the script works fine.

Best regards,

Tiberido

Net::FTP Progress Bar
2005-07-20 14:22:52 E=Mc² [Reply | View]

Anyway of adding this to and ftp get? or is there any other way to add a progress bar to an ftp get you can
use the ftp->size but I cant seem to get the display to update as I cant apend per bytes any help or
suggestions

Term::ProgressBar
2005-01-28 21:50:28 kiwiberryfox [Reply | View]

I was going through the codes for different Progress bars and I noticed
a typo in the last Progress bar code for
href="http://hacks.oreilly.com/pub/h/943">Term::ProgressBar. Line 45

<hr />

$progress = Term::ProgressBar->new({count => $total_size, ETA => &return;
'linear'});
$progress->minor(0);

<hr />
You should replace the semicolon with a comma.
<hr />
$progress = Term::ProgressBar->new({count => $total_size, ETA => &return,
'linear'});
$progress->minor(0);

Thanks,
Kiwiberryfox

Showing messages 1 through 3 of 3.

O'Reilly Home | Privacy Policy
© 2007 O'Reilly Media, Inc.
Website: webmaster@oreilly.com | Customer Service: orders@oreilly.com | Book issues: booktech@oreilly.com

All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners.

5 di 5 20/06/2009 11.56

progress

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to progress (20)

Recently uploaded (20)

progress