Test tutorial

Test::Tutorial
chromatic and Michael G Schwern

What Is Testing?
q Check that your code does what it's supposed to do.
q At varying levels of granularity.
q In varying environments.
q At various points in its development.

What Is Automated Testing?
q Programs to check other programs.
q As with any rote task, you let the computer do it.
x Humans will forget rote tests, computers will not
q Press a button and walk away.
x No human required (very important)
q Manually running tests is a waste of your time.
q Tests should run as close to instantaneous as possible
x so you won't have an excuse not to run them
x so you'll run them as often as possible
x Instant feedback

Testing Promotes Automation
q Testable code is decoupled
q Testable code is scriptable

Why Test?
q no missing functionality
q no accidental functionality
q when your tests pass, you're done

More informative bug reports
q Better to get the diagnostic output of your tests than "It doesn't
work"
q Easily generated by end users
x "Run the tests and send me the output"
q Helps IMMENSELY with porting (this is why my code works on
VMS)
x You can often port code without ever using the machine you're
porting to

More More Reasons
q Most of the time spent on a project is debugging and bug fixing.
x Worse, it often comes at the end (hidden cost)
x "Oh, I'm 99% done, I just need to do some testing"
q Testing as you go will increase your development time, but
reduce debugging time.
x It will let you estimate more realistically
x Increased project visibility
x Reduced debug time once you get used to it

The Real Reason For Writing Tests
q Confidence.
x No fear of change
x No fear of lurking bugs
x No fear of breaking old things
x No fear that new things don't work
Ë Knowing when things don't work.
q So you can play and experiment without worry.
q Enable refactoring

Testing Is Laziness
q Take an O(n) amount of work and make it O(1)
x Instead of walking through the code by hand at each change
x Teach the computer to do that.

Textbook Testing
q Traditional testing philosophy says things like
Test all subroutines
Test all branches of all conditions
Test the boundary conditions of all inputs
...
q This is just big and scary and too much.
q We're lazy, and we think we can still be effective with much
less work.

XP Testing
q XP says to write your tests before you write your code.
x It's hard enough to get people to write tests at all.
x Changing their coding philosophy at the same time is worse.
q If you can do Test First, excellent.
q If you're not already testing, this is a chance to start some new
habits...

On Test-First Programming
q Think of it as coding to teeny, tiny, mini-iterations.
q Break each task into boolean expressions.
q Ask "What feature do I need next?"
x Test the smallest and most immediate element of the overall task.
x Take small steps!

The two test-first questions
q "How can I prove that this feature works?"
x Write the simplest test that will fail unless the feature works.
x The test must fail.
q "What is the least amount of code I can write to pass the test?"
x The simpler the test, the simpler the code you need.
x The test must now pass.
q This produces known good code and a comprehensive test
suite.
q Be sure to run the entire test suite after you implement a task.
q Don't be afraid of baby steps. That's the point.

Test Bugs
q Another good philosophy is to test bugs and new features.
q Every time you find a bug, write a test for it.
q Every time you add a new feature, write a test for it.
q In the process, you might test a few other related things.
q This is the simplest way to retrofit tests onto existing code.

Effects Of Testing Bugs
q This has pleasant effects:
x Slowly grows your test suite
x Focuses tests on the parts of the code which have the most bugs
q You're allowed to make mistakes, but ONLY ONCE. Never
twice.
q A disproportionate amount of bugs use the same logic.
x One test can catch lots of bugs

Knowing You're Done
t/Your-Code.t......ok
All tests successful.
q For once, your computer is telling you something good.
q Instant, positive feedback

There Is No Magic
q You may have seen this from h2xs.
######## We start with some black magic to print on failure.
# Change 1..1 below to 1..last_test_to_print .
# (It may become useful if the test is moved to ./t subdirec
BEGIN { $| = 1; print "1..1n"; }
END {print "not ok 1n" unless $loaded;}
use Foo;
$loaded = 1;
print "ok 1n";
######## End of black magic.
q Testing really isn't this frightening.

And Now For Something Completely Different

The Most Basic Perl Test Program
#!/usr/bin/perl -w
print "1..1n";
print 1 + 1 == 2 ? "ok 1n" : "not ok 1n";
q Since 1 + 1 is 2, this prints:
1..1
ok 1
x "1..1" I'm going to run one test.
x "ok 1" The first test passed.

Perl's Testing Protocol
q There are two parts to running a test in Perl.
x Your test
x Test::Harness
q The output of your test is piped to Test::Harness.
q Test::Harness interprets your output and reports.
$ perl -MTest::Harness -wle 'runtests @ARGV' contrived.t
contrived....ok
Files=1, Tests=1, 0 wallclock secs ( 0.02 cusr + 0.02 csys

There's TMTOWTDI and there's this...
q Here's some of the many ways people write their tests:
x t/op/sysio.t
print 'not ' unless (syswrite(O, $a, 2) == 2);
print "ok 20n";
x ext/Cwd/t/cwd.t
print +($getcwd eq $start ? "" : "not "), "ok 4n";
x t/pod/plainer.t
unless( $returned eq $expected ) {
print map { s/^/#/mg; $_; }
map {+$_} # to avoid readonly values
"EXPECTED:n", $expected, "GOT:n", $returned;
print "not ";
}
printf "ok %dn", ++$test;
q Maintenance nightmare.

I'm ok, you're ok
#!/usr/bin/perl -w
use Test::Simple tests => 1;
ok( 1 + 1 == 2 );
q "ok" is the backbone of Perl testing.
x If the expression is true, the test pass.
x False, it fails.
q Every conceivable test can be performed just using ok().

YOU FAILED!!!
#!/usr/bin/perl -w
ok( 1 + 1 == 2 );
ok( 2 + 2 == 5 );
q from that comes:
1..2
ok 1
not ok 2
# Failed test (contrived.t at line 5)
# Looks like you failed 1 tests of 2.
x "1..2" I'm going to run two tests.
x "ok 1" The first test passed.
x "not ok 2" The second test failed.
x Some helpful commentary from Test::Simple

Date::ICal
q We're going to write some tests for Date::ICal.
x It's real code.
x It's sufficiently complex.
x Everyone understands dates.
Ë Some people even have them.

Where To Start?
q This is the hardest part.
q Retrofitting a test suite onto old code sucks.
x Marching through the testing swamps.
q Write tests from the start and your life will be easier.
q In any event, begin at the beginning.

new()
q Since Date::ICal is OO, the beginning is when you make an
object.
x (white-lie: the beginning is when you load the module)
#!/usr/bin/perl -w
use Date::ICal;
my $ical = Date::ICal->new; # make an object
ok( defined $ical ); # check we got something
ok( $ical->isa('Date::ICal') ); # and it's the right class
q This produces:
1..2
ok 1
ok 2
q This is your first useful test.

Names
q "ok 2" isn't terribly descriptive.
x what if you have 102 tests, what did #64 do?
q Each test can be given a little description.
ok( defined $ical, 'new() returned something' );
ok( $ical->isa('Date::ICal'), " and it's the right class" )
q This outputs
1..2
ok 1 - new() returned something
ok 2 - and it's the right class

What's In A Name
q Two views on names.
x A name is a descriptive tag so you can track the test output back to
the code which produced it. (the original purpose)
x A name is a short description of what was tested.
q There's a subtle difference.
q Don't pull your hair out over it.
x More importantly, don't pull other people's hair out over it.

Test The Manual
q Simplest way to build up a test suite is to just test what the
manual says it does.
x Also a good way to find mistakes/omissions in the docs.
x You can take this five steps further and put the tests IN the
manual. Test::Inline, later.
q If the docs are well written, they should cover usage of your
code.
x You do have docs, right?

SYNOPSIS
q A good place to start.
x A broad overview of the whole system
q Here's a piece of Date::ICal's SYNOPSIS.
SYNOPSIS
use Date::ICal;
$ical = Date::ICal->new( year => 1964, month => 10, day =
hour => 16, min => 12, sec => 47, tz => '0530' );
$hour = $ical->hour;
$year = $ical->year;
q Oddly enough, there is a bug in this.

SYNOPSIS test
use Date::ICal;
$ical = Date::ICal->new(
year => 1964, month => 10, day => 16, hour => 16,
min => 12, sec => 47, tz => '0530' );
ok( $ical->sec == 47, ' sec()' );
ok( $ical->min == 42, ' min()' );
ok( $ical->hour == 10, ' hour()' );
ok( $ical->day == 16, ' day()' );
ok( $ical->month == 10, ' month()' );
ok( $ical->year == 1964, ' year()' );

SYNOPSIS results
1..8
ok 3 - sec()
not ok 4 - min()
# Failed test (ical.t at line 14)
not ok 5 - hour()
# Failed test (ical.t at line 15)
ok 6 - day()
ok 7 - month()
ok 8 - year()
q Whoops, failures!
q We know what and where it failed, but not much else.
q How do you find out more?
x Throw in print statements
x Run in the debugger.
q That sounds like work.

Test::More
q Test::Simple is deliberately limited to one function.
q Test::More does everything Test::Simple does.
x You can literally s/use Test::Simple/use Test::More/
q It provides more informative ways to say "ok".

is() you is() or is() you isnt() my $baby;
q Test::More's is() function:
x declares that something is supposed to be something else
x "Is this, that?"
is( $this, $that );
# From
ok( $ical->day == 16, ' day()' );
# To
is( $ical->day, 16, ' day()' );

ok() to is()
q Here's the test with ok() replaced with is() appropriately.
use Test::More tests => 8;
use Date::ICal;
$ical = Date::ICal->new(
year => 1964, month => 10, day => 16, hour => 16,
min => 12, sec => 47, tz => '+0530' );
is( $ical->sec, 47, ' sec()' );
is( $ical->min, 42, ' min()' );
is( $ical->hour, 10, ' hour()' );
is( $ical->day, 16, ' day()' );
is( $ical->month, 10, ' month()' );
is( $ical->year, 1964, ' year()' );
q "Is $ical->sec, 47?"
q "Is $ical->min, 12?"

Diagnostic Output
1..8
ok 3 - sec()
not ok 4 - min()
# Failed test (- at line 13)
# got: '12'
# expected: '42'
not ok 5 - hour()
# got: '21'
# expected: '10'
ok 6 - day()
ok 7 - month()
ok 8 - year()
q $ical->min returned 12 instead of 42.
q $ical->hour returned 21 instead of 10.

Interpreting The Results
q Turns out, there is no 'tz' argument to new()!
x And it didn't warn us about bad arguments
q The real argument is 'offset'
x So the synopsis is wrong.
x This is a real bug I found while writing this
q Damn those tests.

When to use is()
q Use instead of ok() when you're testing "this equals that".
x Yes, there is an isnt() and isn't().
q is() does a string comparison which 99.99% of the time comes
out right.
x cmp_ok() exists to test with specific comparison operators

Tests Are Sometimes Wrong
q The previous example was supposed to be highly contrived to
illustrate that tests are sometimes wrong.
q When investigating a test failure, look at both the code and the
test.
q There's a fine line of trusting your testing code.
x Too much trust, and you'll be chasing phantoms.
x Too little trust, and you'll be changing your tests to cover up bugs.

How Can I Be Sure The Test Is Right?
q Write the test
q Run it and make sure the new test fails
q Add the new feature / fix the bug
q Run the test and make sure the new test passes.
q Some development systems, such as Aegis, can enforce this
process.
q It's difficult to do this when writing tests for existing code.
x Another reason to test as you go

Version Control and Testing
q VC & testing work well.
x Run the tests, make sure they pass
x Make sure everything is checked in.
x Write tests for the bug / feature.
Ë Make sure they fail.
x Fix your bug / write your feature
x Run the tests.
Ë If they pass, commit. You're done.
Ë If they fail, look at the diff. The problem is revealed by that change.
q The smaller the change, the better this works.
q You are using version control, right?

Testing vs Brooks's Law
q Tests catch damage done by a new programmer immediately
q Easier for other developers to help you
x They can pre-test their patches.
x Even if you write perfect code, the rest of us don't.

Testing Lots Of Values
q Date handling code is notorious for magic dates that cause
problems
x 1970, 2038, 1904, 10,000. Leap years. Daylight savings.
q So we want to repeat sets of tests with different values.

It's Just Programming
use Date::ICal;
my %ICal_Dates = (
'19971024T120000' => # from the docs.
[ 1997, 10, 24, 12, 0, 0 ],
'20390123T232832' => # after the Unix epoch
[ 2039, 1, 23, 23, 28, 32 ],
'19671225T000000' => # before the Unix epoch
[ 1967, 12, 25, 0, 0, 0 ],
'18990505T232323' => # before the MacOS epoch
[ 1899, 5, 5, 23, 23, 23 ],
);
while( my($ical_str, $expect) = each %ICal_Dates ) {
my $ical = Date::ICal->new( ical => $ical_str, offset => 0 )
ok( defined $ical, "new(ical => '$ical_str')" );
is( $ical->year, $expect->[0], ' year()' );
is( $ical->month, $expect->[1], ' month()' );
is( $ical->day, $expect->[2], ' day()' );
is( $ical->hour, $expect->[3], ' hour()' );
is( $ical->min, $expect->[4], ' min()' );
is( $ical->sec, $expect->[5], ' sec()' );
}

The Good News
q If you can write good code, you can learn to write good tests.
q Just a while loop.
q Easy to throw in more dates.

The Bad News
q You have to keep adjusting the # of tests when you add a date.
x use Test::More tests => ##;
q There are some tricks:
# For each date, we run 8 tests.
use Test::More tests => keys %ICal_Dates * 8;
q There's also 'no_plan':
use Test::More 'no_plan';

Plan? There Ain't No Plan!
q The plan exists for protection against:
x The test dying
x Accidentally not printing tests to STDOUT
x Exiting early
q The first two have other protections, and the third will shortly.
x So the plan isn't as useful as it used to be
q Newer versions of Test::Harness allow the plan to be at the
end:
ok 1
ok 2
ok 3
1..3
q This allows Test::More to count your tests for you.
x You have to upgrade Test::Harness for this to work.

Boundary tests
q Almost bad input
q Bad input
q No input
q Lots of input
q Input that revealed a bug

Bad Input Can Do Bad Things
q Garbage in / Error out
x graceful exceptions, not perl errors
x helpful warnings, not uninitialized value warnings
q Make sure bad input causes predictable, graceful failure.

Basic bad input example
local $!;
ok( !open(FILE, "I_dont_exist"), 'non-existent file' );
isnt( $!, 0, ' $! set' );
q Note, the exact value of $! is unpredictable.

Tests with warnings
q Test::More used to have a problem testing undefined values
is( undef, undef, 'undef is undef' );
q The test will pass, but there would be warnings.
x The user will see them, but the test will not.
q There's a whole bunch of these in Test-Simple/t/undef.t

Catching Warnings
q Use $SIG{__WARN__}.
my $warnings = '';
local $SIG{__WARN__} = sub { $warnings . join '', @_ };
is( undef, undef, 'undef is undef' );
is( $warnings, '', ' no warnings' );
q Use the same technique to check for expected warnings.

Dealing With Death
q Use eval BLOCK.
local $@;
eval {
croak "Wibble";
};
like( $@, qr/^Wibble/ );
q Use the same technique to check that things didn't die.
x Useful for past bugs where certain inputs would cause a fatal error.

Acceptance, Regression, Unit, Functional...
q Same thing, just a matter of timing.
q Unit: Detailed tests of individual parts
x Unit tests are easy(er)
x So we think of the rest in terms of unit tests
q Functional: Tests of your API
x Blackbox unit tests
q Integration: Testing that the pieces work together
x Just unit testing bigger units
q Acceptance: Tests defining your requirements
x Customer driven unit tests
q Regression: Tests for backwards compatibility
x Old tests never die, they just become regression tests
q All can be done with the same techniques

Blackbox vs Glassbox
q No, they're not window managers.
q Blackbox tests use only the public, documented API.
x No cheating
x You have to forget how the code is implemented
x More closely approximates real world usage
x Immune from internal changes
x Often forces you to make the public API more flexible
q Glassbox tests can use whatever you want.
x Cheat, steal, lie, violate encapsulation
x Often necessary to test certain 'untestable' parts
x May be broken by internal changes, undermines the test suite.
q Blackbox is preferred where possible, glassbox is sometimes
necessary.
x Sometimes you can just peek inside the box.

Test::More toys
q Test::More has 13 ways to say ok.
x It also has a wonderful man page.
q Here's some of the most common.

like()
q Next to is() and ok(), you'll be using like() the most.
like( $this, qr/that/ );
q This is the same as:
ok( $this =~ /that/ );
q It has nicer diagnostics:
not ok 1
# Failed test (contrived.t at line 2)
# 'wibble'
# doesn't match '(?-xism:woof)'
q Because qr// was added in 5.005, it understands a string that
looks like a regex for older perls.
like( $this, '/that/' );
q There is an unlike() which is the !~ version.

isa_ok()
q We've been doing this a lot.
ok( defined $ical, "new(ical => '$ical_str')" );
q You do this so much in OO code, there's a special function.
isa_ok( $ical, 'Date::ICal' );
q It works on references, too.
isa_ok( $foo, 'ARRAY' ); # is $foo an array ref?
q It also has nice diagnostics.
not ok 1 - The object isa Date::ICal
# The object isn't a 'Date::ICal' it's a 'ARRAY'

can_ok()
q A test for $obj->can($some_method)
ok( $obj->can('foo'), 'foo() method inherited' );
q Simple but useful test can be like:
# Does the Foo class have these methods?
can_ok( 'Foo', qw(this that whatever wibble) );
x Might seem silly, but can catch stupid mistakes like forgetting a "=cut"
q Takes an object or a class.
q Also useful for checking your functions are exported
use Text::Soundex;
can_ok(__PACKAGE__, 'soundex');

use_ok()
q The real first thing you test is if the module loaded.
BEGIN { use_ok( 'Date::ICal' ); }
q Has to be inside a BEGIN block to act like a real 'use'.
q Remember the black magic? That's what it was doing.

is_deeply()
q For comparing complex data structures
x Hashes, lists, hash of lists of hashes of lists of scalar references...
my %expect = ( this => 42, that => [qw(1 2 3)] );
my %got = some_function();
is_deeply( %got, %expect );
q Will show you where the two structures start to diverge
not ok 1
# Structures begin differing at:
# $got->{that}[2] = '3'
# $expected->{that}[2] = Does not exist
q In CS this is really a "shallow comparison" and is() is "deep".
x So the name is wrong because Schwern failed CS.
q A stopgap measure
x Currently doesn't handle circular structures (patches welcome)
q Waiting for someone to step up to the plate and write Test::Set

diag()
q Test::More's functions are pretty good about providing
diagnostics.
q Sometimes you need more...
q diag() lets you display whatever diagnostic information you
want.
x Guaranteed not to interfere with Test::Harness
x Not a test function
x Will not display inside a TODO block
q Useful for giving suggestions about tricky failures

Odd User Reactions
q Sometimes users react rather oddly to tests.
x won't report failures
x will react to failures as if the test caused the bug!
x will report "the tests failed" and leave off all the diagnostics
x won't run the tests at all

Getting People to RUN Your Tests
q Once you've gotten people writing tests...
q ...your next problem is getting them to RUN them

Make It Simple
q Preferably ONE command.
x no user interaction (or smart defaults)
x 'make test'
x 'quicktest' CVS integration.

Test On Commit
q Make running the tests part of your commit policy
x Automate with CVS commit actions (CVSROOT/modules)
x Use a system such as Aegis

Daily Smoke Test
q Run the whole battery of tests against the latest code every
day, automatically
x CPAN::Smoke is one example

Test Before Release
q Automatically run tests as part of your release process.
x 'make disttest'
x your release process is automated, right?

Testing Is Eating Your Own Dog Food
q It forces you to use your own API.
q Code that's hard to test may be hard to use.
q This often makes your API more flexible.
x Tends to get rid of constants and assumptions

'make test'
schwern@blackrider:~/src/devel/File-chdir$ make test
PERL_DL_NONLAZY=1 /usr/local/bin/perl5.6.1 -Iblib/arch
-Iblib/lib -I/usr/local/perl5.6.1/lib/5.6.1/ppc-linux-64int
-I/usr/local/perl5.6.1/lib/5.6.1 -e
'use Test::Harness qw(&runtests $verbose); $verbose=0;
runtests @ARGV;' t/*.t
t/array.............ok
t/chdir.............ok
t/var...............ok
Files=3, Tests=48, 2 wallclock secs
( 1.71 cusr + 0.38 csys = 2.09 CPU)
q When you run 'make test' on a CPAN module, you're using:
ExtUtils::MakeMaker
Test::Harness
your test

What in the hell is all that mess?
PERL_DL_NONLAZY=1
q magic to force XS code to strictly check shared libraries
-Iblib/lib -Iblib/lib
q Changes @INC to use the module you're about to install
-I/usr/local/perl5.6.1/lib/5.6.1/ppc-linux-64int ...
q Mistake. Code specific for testing Perl itself that leaked out.
q Causes problems with core modules on CPAN.
q Fixed in latest versions of MakeMaker.

The mess continued...
-e 'use Test::Harness qw(&runtests $verbose);
q import runtests and $verbose
$verbose=0
q This is really $verbose=$(TEST_VERBOSE)
runtests @ARGV;' t/*.t
q Pass in all your tests to Test::Harness::runtests()

Still more mess...
t/array.............ok
t/chdir.............ok
t/var...............ok
q Your tests are all ok
q It's Miller Time.
( 1.71 cusr + 0.38 csys = 2.09 CPU)
q Benchmark of how long your tests took to run. May go away.

New MakeMaker Is A Little Different
$ make test
PERL_DL_NONLAZY=1 /usr/local/bin/perl
"-MExtUtils::Command::MM" "-e"
"test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/array....ok
t/chdir....ok
t/var......ok
( 2.27 cusr + 0.48 csys = 2.75 CPU)
q The -I$(PERL_LIB) -I$(PERL_ARCH) mistake is gone
q The hanging Test::Harness wires have been put away
q Mostly done for non-Unix platforms.

test.pl caveat
q Some modules put tests in test.pl.
q Do not do that.
q 'make test' does not parse the output which means...
x 'make test' won't exit with non-zero on failure.
x Things like the CPAN shell won't know there was a failure.
x Historical accident, MakeMaker predates Test::Harness.

Testing and Perl versions
q Test::Simple/More will be in 5.8.0.
q Test.pm was put in 5.4.5.
q Test::Harness has been around so long nobody remembers
who wrote it.
x pre-5.6.1 will not support TODO tests or no_plan.
q They're all available from CPAN.
q They all work back to 5.4.0.
q They all work on every platform.

Testing, CPAN Modules, PREREQ_PM
q Some people worry about having too many prereqs on their
CPAN modules
x Don't want to add prereqs on testing modules
q A prereq of Test::More in turn prereqs & upgrades
Test::Harness.
q Even though Test::More isn't yet in core, it's already widely
installed.
Acme::ComeFrom, Acme::Magpie, Acme::Time::Asparagus,
Acme::USIG, Acme::Your, Alzabo, Apache::ConfigParser,
Apache::DefaultCharset, Apache::GuessCharset, Apache::RSS,
Apache::Session::CacheAny,
Apache::Session::Generate::ModUniqueId,
Apache::Session::Generate::ModUsertrack,
Apache::Session::PHP, Apache::Session::SQLite,
Apache::Singleton, Apache::StickyQuery, App::Info,
Archive::Any, Astro::Funtools::Parse, Attribute::Profiles,
Attribute::Protected, Attribute::Unimplemented, CPAN,
Business::Tax::Vat, Cache::Mmap, Carp::Assert, CDDB::File,
CGI::Application, CGI::FormMagick, CGI::Untaint,
CGI::Untaint::creditcard, CGI::Untaint::email,
CGI::Untaint::uk_postcode, Class::DBI,

More modules with Test::Simple/Test::More
prerequisites
Class::DBI::FromCGI, Class::DBI::Join, Class::DBI::mysql,
Class::DBI::SQLite, Class::Factory, Class::Observable,
Class::PseudoHash, Class::Trigger, CompBio, File::Random,
Crypt::CAST5_PP, Crypt::OOEnigma, Data::BFDump,
Data::BT:PhoneBill, Date::Chinese, Date::DayOfWeek,
Date::Discordian, Date::Easter, Date::Passover, Date::ICal,
Date::ISO, Date::Japanese, Date::Leapyear, Date::Range,
Date::Range::Birth, Date::Roman, Date::Set,
Date::SundayLetter, Devel::Caller, Devel::LexAlias,
Devel::Profiler, Devel::Tinderbox::Reporter, DNS::Singleton,
Email::Find, Email::Valid::Loose, Encode::Punycode,
Getopt::ArgvFile, GraphViz::Data::Structure, Hash::Merge,
HTML::Calendar::Simple, HTML::DWT, HTML::ERuby,
HTML::FromANSI, HTML::LBI, HTML::Lint, HTML::TableParser,
HTML::Template::JIT, HTML::TextToHTML, I18N::Charset,
IDNA::Punycode, Ima::DBI, Image::DS9, Inline::TT,
IO::File::Log, Lingua::Pangram, Lingua::SoundChange,
Lingua::Zompist::Barakhinei, Lingua::Zompist::Cadhinor,
Lingua::Zompist::Kebreni, Lingua::Zombist::Verdurian,
Locale::Maketext::Lexicon, Log::Dispatch::Config,
Log::Dispatch::DBI, Mail::Address::MobileJp,

Everybody's Depending on Us!
Mail::Address::Tagged, Mail::ListDetector,
Mail::ListDetector::Detector::Fml, MARC::Record,
Math::Currency, Module::CoreList, Module::InstalledVersion,
SPOPS, Net::DNS, Net::DNS::Zonefile, Net::ICal,
Net::IDN::Nameprep, Net::IP::Match, Net::Services,
Net::Starnet::DataAccounting, Net::Telnet::Cisco,
OutNet::BBS, PerlPoint::Package, PHP::Session,
Pod::Coverage, Test::Inline,
POE::Component::IKC::ReallySimple, POE::Component::RSS,
POE::Component::SubWrapper, POE::Session::Cascading,
Proc::InvokeEditor, Regexp::English, Regexp::Network,
Spreadsheet::ParseExcel::Simple, Storable, Sub::Context,
Sub::Parameters, Term::Cap, Term::TtyRec, Test::Class,
Test::Exception, Test::Mail, CGI::Application, Text::Quote,
Text::WikiFormat, Tie::Array::Iterable, Tie::Hash::Approx,
uny2k, WWW::Automate, WWW::Baseball::NPB, WWW::Page::Author,
WWW::Page::Host, WWW::Page::Modified, WWW::Search,
XML::Filter::BufferText, XML::SAX::Writer, XML::XPath::Simpl
XML::XSLT, XTM, XTM::slides
q So the prerequisite will likely already be resolved.
q Brought to you by Schwern Of Borg.

t/lib trick
q If you still don't want to have prerequisites on testing modules
x Copy Test/Builder.pm & Test/More.pm into t/lib/
x Slap a "use lib 't/lib'" on your tests
x distribute the whole thing
q Who does this?
x CGI, CPANPLUS, MakeMaker, parrot, Test::Harness
q Caveats
x You'll be adding to Test::More's takeover of search.cpan.org
x Adds 18K to your tarball.
x Can't use TODO or no_plan.

Make the GUI layer thin
q GUIs, CGI programs, etc... are hard to test.
q Make the problem as small as possible.
x Separate the form from the functionality.
x Put as much code into format agnostic libraries as possible
x Large, stand-alone programs (especially CGIs) ring alarm bells.
q You might wind up with a small amount that still needs to be
tested by hand.
x At least you don't have to test the whole thing by hand.

Testing Web Stuff
q WWW::Automate is your friend.
x LWP with lots of help.
x Easily deals with forms
x "Click" on buttons
x Follow links
x Has a "back" button
q Makes simulating a real web site user easier.

Domain Specific Test Libraries
q WWW::Automate
x Technically not a test library, but sooooo useful
q Test::Exception
q Test::Differences
x Testing large blocks of text and complicated structures
q Test::Unit
x Straight XUnit port to Perl
x Great for those used to JUnit & PyUnit
q Test::Class
x XUnit, but adapted to Perl
x Inherited tests
q Test::MockObject
q Test::Inline
x Embed tests in your documentation
q Test::Mail

Test::Builder
q Usually you want Test::More's general functions + domain
specific ones.
x Unfortunately, sometimes test libraries don't play well together
x Who owns the test counter?
x Who prints the plan?
q Test::Builder is a single backend to solve that problem.
x Singleton object to handle the plan and the counter
x Test::More-like methods you can write wrappers around
q Test libraries built on Test::Builder will work together.
Test::Exception, Test::Class, Test::MockObject,
Test::Inline, Test::Mail, Test::More, Test::Simple
q Attend "Writing A Test Library" for more information

Passing Tests Should PASS
q One must trust their test suite, else it will be ignored.
q When it fails, it should indicate a real problem.
q "Expected failures" sap that trust.
x "Oh, don't worry, that test always fails on Redhat 6.2"
x If a failure sometimes isn't really a failure, when do you know a real
failure?
q "Expected failures" make test automation impossible.
x Programs don't know "well, the test failed but it really passed"
x Joe CPAN module installer also doesn't know that.
q Get your test suite at 100% and keep it there.
x That's worth saying again.
q STAY AT 100% PASSING!

Failure Is An Option
q There are three varieties of test failure, and several solutions.
x A failure indicating a mistake/bad assumption in the test suite.
Ë You fix it.
x A real failure indicating a bug or missing feature.
Ë You fix it, or...
Ë You put off fixing it and...
Ë comment out the test (blech) or...
Ë declare it "TODO"
x A failure due to an assumption about the environment.
Ë You can't fix it, so you "skip" it.

It'll Never Work
q Sometimes, a test just doesn't make sense in certain
environments.
q Some examples...
x Features which require a certain version of perl
x Features which require perl configured a certain way (ex. threads)
x Features which are platform specific
x Features which require optional modules

Skipping Tests
q Let's assume we have a test for an HTML generator.
q Let's also assume that if we have HTML::Lint, we want to lint
the generated code.
require HTML::Lint;
my $lint = HTML::Lint->new;
isa_ok( $lint, 'HTML::Lint' );
$lint->parse( $some_html );
is( $lint->errors, 0, 'No errors found in HTML' );
q Since HTML::Lint is optional, this test will fail if you don't have
it.
x But it's not a real failure, else HTML::Lint isn't really optional.
x So the user shouldn't hear about it.

# SKIP
q You can explicitly skip a set of tests rather than run them.
1..2
ok 1
ok 2 # SKIP no beer
x Test #1 passed.
x Test #2 was skipped because there is no beer.
q A skipped test means the test was never run.

SKIP: block
q Test::More can cause an entire block of code not to run at all.
SKIP: {
eval { require HTML::Lint };
skip "HTML::Lint not installed", 2 if $@;
my $lint = new HTML::Lint;
isa_ok( $lint, "HTML::Lint" );
$lint->parse( $html );
is( $lint->errors, 0, "No errors found in HTML" );
}
x if we don't have HTML::Lint, the skip() function is run.
x skip() prevents anything further in the SKIP block to be run.
x the number indicates how many tests you would have run.
q The appropriate number of 'ok's will be output.
ok 23 # SKIP HTML::Lint not installed
ok 24 # SKIP HTML::Lint not installed

skipall
q In some cases you want to skip a whole test file.
use Test::More;
if( $^O eq 'MSWin32' ) {
plan tests => 42;
}
else {
plan skip_all => 'Win32 specific test';
}
q Test::More will exit at the skip_all.
q On non-Win32, the output will be:
1..0 # skip Win32 specific test
q Test::Harness will interpret this as a skipped test.

Procrastination Codified
q It's good to write the test before you add a new feature.
q It's good to write a test as soon as you receive a bug report.
q It's bad to release code with failing tests.
q This would seem to be a contradiction.
x Either you fix all your bugs and add all your features immediately
x Or you comment out your failing tests.
q Option #3, for the professionally lazy:
x Declare your failing tests to be "todo"
q This allows one to build a test suite without having to fix all the
bugs you find right away.

TODO Test
TODO: {
local $TODO = 'URI::Geller not quite working';
my $card = 'Eight of clubs';
is( URI::Geller->your_card, $card, 'Is this your card?' );
my $spoon;
URI::Geller->bend($spoon);
is( $spoon, 'bent', 'Spoon bending' );
}
q Output will be something like:
not ok 23 - Is this your card
# TODO URI::Geller not quite working
not ok 24 - Spoon bending
# TODO URI::Geller not quite working

Automated TODO List
q TODO reverses the sense of the test
x 'not ok' will be treated as a quiet success
x 'ok' Test::Harness will warn you of an "unexpected success"
q It's a TODO list
x Write your tests before your feature/bug fix
x Each 'unexpected success' is an item off your todo list
x Remove the TODO wrapper
q You can release at any point and not have to cull your test
suite
q Keeps users from seeing "expected failures"
q Each open bug can have a test.
x Sometimes bugs get accidentally fixed

Keep Test Scripts Small
q Many testing questions start with
x "I've got this test script with 1400 tests..."
q Big tests are
x Hard to maintain
x Hard to decouple
x Hard to read
x Take a long time to run
x Have all the same problems as big subroutines
q Keep them small & focused.
x One function or set of functions per script
x One aspect per script
x Put complicated tests in their own script
x Put slow tests in their own script
q Test::Simple/More's tests are a good example

Big FTP/XML program example
q Common testing problem. You have a big program which...
x Downloads an XML file via FTP
x Parses the XML
x Generates HTML
q How do you test that?

Programs Are Hard, Libraries Are Easy
q The smaller the piece, the better.
q The more flexible the piece, the better.
q The more hooks into the guts, the better.
x Libraries of functions can have small, flexible pieces.
x Programs are, by definition, monolithic.
q Extract pieces out of your program and put it into a library
x Then test the library
x Side-benefit, you'll have improved your code
q Take the FTP, XML parsing and HTML generation code out of
the program.

Separate Form And Functionality
q HTML is hard to test
x It changes a lot
x It's hard to parse
q Instead of going from XML straight to HTML
q ...go from XML -> agnostic format -> HTML
x Test the XML -> agnostic part
x Test the agnostic -> HTML part
q Much easier to test when only one of the input/output pair is
formatted.
q ...and you'll have improved the flexibility of your code.

Mock Code
q Sometimes you just can't run a piece of code in a test
x Maybe there's no network connection
x Maybe the test is destructive (system("/sbin/shutdown now"))
q Going to the extreme edge of glassbox testing, replacing code
for testing

System call / Power manager example
q Say you have to test a power management daemon
q One of the things it does is puts the computer to sleep
q How do you test that?
sub should_i_sleep {
my($power_remaining) = @_;
system("/sbin/snooze") if $power_remaining < $Min_Power;
return 1;
}

First, Isolate The Untestable Part
sub should_i_sleep {
my($power_remaining) = @_;
snooze if $power_remaining < $Min_Power;
return 1;
}
sub snooze {
system("/sbin/snooze");
}
q Test snooze() by hand once.
x It's small, so you can get away with it

Then, Replace The Untestable Part
{
my @snooze_args = ();
my $snooze_called = 0;
local *Power::Manager::snooze = sub {
$snooze_called++;
@snooze_args = @_; # trap the arguments
return 0; # simulate successful system call
};
should_i_sleep($Min_Power - 1);
is( $snooze_called, 1, 'snooze called once' );
is( @snooze_args, 0, ' called properly' );
}
q Check that it was called.
q Check that it got the right arguments
q By changing the return value to non-zero we can simulate a
failure.
q Very, very powerful technique.

Forcing Failure
q How will your program react if, say, the database connection
fails?
use DBI;
{
local *DBI::connect = sub {
return 0;
};
...test for graceful failure here...
}
...test for graceful recovery here...

He's Your Dog, Charlie Brown
q Don't leave testing for the QA guys
x too much delay
x too much animosity
q You know your code, you can test it
x and you can fix it
x and you wrote it, so it's your bug :P

Further Reading
q perl-qa@perl.org
q http://archive.develooper.com/perl-qa@perl.org/
q "Perl Debugged"
q "Writing Solid Code"

Thanks
q Norman Nunley
q Andy Lester
q Barrie Slaymaker
q H. Merijn Brand
q Jarkko Hietaniemi
q Tatsuhiko Miyagawa
q Tels
q Rafael Garcia-Suarez
q Abhijit Menon-Sen
q Curtis Poe & OTI
q Beer and Root Beer (fuel of champions)

Test tutorial

More Related Content

What's hot (20)

Similar to Test tutorial (20)

Recently uploaded (20)

Test tutorial