Structure and Interpretation of Test Cases

Structure and
Interpretation
of Test Cases
@KevlinHenney

Programs must be
written for people to
read, and only
incidentally for
machines to execute.

Write Tests for People
Gerard Meszaros

So who should you be writing the
tests for? For the person trying to
understand your code.
Good tests act as documentation
for the code they are testing.
Gerard Meszaros

Very many people say "TDD"
when they really mean, "I have
good unit tests" ("I have GUTs"?).
Alistair Cockburn
The modern programming professional has GUTs

good unit tests" ("I have GUTs

A unit test is a test of behaviour
whose success or failure is wholly
determined by the correctness of
the test and the correctness of
the unit under test.
Kevlin Henney
https://www.theregister.co.uk/2007/07/28/what_are_your_units/

Unit testable
Necessarily not unit testable
Should be unit testable... but isn’t

The programmer in me made unit testing more
about applying and exercising frameworks.
I had essentially reduced my concept of unit
testing to the basic mechanics of exercising
xUnit to verify the behavior of my classes.
My mindset had me thinking far too narrowly
about what it meant to write good unit tests.
Tod Golding
Tapping into Testing Nirvana

about what it meant to write good unit tests

From time to time I hear people
asking what value of test coverage
(also called code coverage) they
should aim for, or stating their
coverage levels with pride.
Such statements miss the point.
Martin Fowler
http://martinfowler.com/bliki/TestCoverage.html

I expect a high level of coverage.
Sometimes managers require one.
There's a subtle difference.
Brian Marick
http://martinfowler.com/bliki/TestCoverage.html

public static bool IsLeapYear(int year) 

[Test]
public void TestIsLeapYear() 

[Test]
public void TestIsLeapYearIsCorrect() 

[Test]
public void Test1() 
[Test]
public void Test2() 

[Test]
public void TestLeapYears() 
[Test]
public void TestNonLeapYears() 

[Test]
public void TestLeapYears()
{
Assert.IsTrue(IsLeapYear(2016));
Assert.IsTrue(IsLeapYear(2000));
}
[Test]
public void TestNonLeapYears() 

[Test]
public void Test2016IsALeapYear() 
[Test]
public void Test2000IsALeapYear() 
[Test]
public void Test2018IsNotALeapYear() 
[Test]
public void Test1900IsNotALeapYear() 

[Test]
public void _2016IsALeapYear() 
[Test]
public void _2000IsALeapYear() 
[Test]
public void _2018IsNotALeapYear() 
[Test]
public void _1900IsNotALeapYear() 

[Test]
public void _2016_is_a_leap_year() 
[Test]
public void _2000_is_a_leap_year() 
[Test]
public void _2018_is_not_a_leap_year() 
[Test]
public void _1900_is_not_a_leap_year() 

_2016_is_a_leap_year
_2018_is_not_a_leap_year

Years_divisible_by_4_are_leap_years
Years_not_divisible_by_4_are_not_leap_years
Years_divisible_by_100_are_not_leap_years

Years_divisible_by_100_are_not_leap_years

Years_divisible_by_4_but_not_by_100_are_leap_year
Years_divisible_by_100_but_not_by_400_are_not_lea

Years_divisible_by_4_but_not_by_100_are_leap_years
Years_divisible_by_100_but_not_by_400_are_not_leap_years

IsLeapYear_YearNotDivisibleBy4_ReturnsFalse
IsLeapYear_YearDivisibleBy4ButNotBy100_ReturnsTrue
IsLeapYear_YearDivisibleBy100ButNotBy400_ReturnsFalse
IsLeapYear_YearDivisibleBy400_ReturnsTrue

Propositions
are vehicles
for stating
how things are
or might be.

Thus only indicative
sentences which it
makes sense to think
of as being true or as
being false are
capable of expressing
propositions.

public static bool IsLeapYear(int year)
{
return
year % 4 == 0 &&
year % 100 != 0 ||
year % 400 == 0;
}

{
return year % 4 == 0;
}

For tests to drive development they must
do more than just test that code performs
its required functionality: they must clearly
express that required functionality to the
reader.
That is, they must be clear specifications
of the required functionality.
Nat Pryce & Steve Freeman
Are your tests really driving your development?

namespace Leap_year_spec
{
[TestFixture]
public class A_year_is_a_leap_year
{
[Test]
public void if_it_is_divisible_by_4_but_not_by_100() 
[Test]
public void if_it_is_divisible_by_400() 
}
[TestFixture]
public class A_year_is_not_a_leap_year
{
[Test]
public void if_it_is_not_divisible_by_4() 
[Test]
}
}

{
[TestFixture]
public class A_year_is_a_leap_year 
[TestFixture]
{
[Test]
public void if_it_is_not_divisible_by_4()
{
Assert.IsFalse(IsLeapYear(2018));
}
[Test]
public void if_it_is_divisible_by_100_but_not_by_400()
{
}
}
}

{
[TestFixture]
[TestFixture]
{
[Test]
{
Assert.IsFalse(isLeapYear(2018));
}
[Test]
{
}
}
}

{
[TestFixture]
[TestFixture]
{
[Test]
{
}
[Test]
{
}
}
}

{
[TestFixture]
[TestFixture]
{
[Test]
{
}
[Test]
}
}

{
[TestFixture]
[TestFixture]
{
[TestCase(2018)]
[TestCase(2017)]
[TestCase(42)]
[TestCase(1)]
public void if_it_is_not_divisible_by_4(int year)
{
Assert.IsFalse(IsLeapYear(year));
}
[Test]
}
}

{
[TestFixture]
[TestFixture]
{
[Test]
public void if_it_is_not_divisible_by_4(
[Values(2018, 2017, 42, 1)] int year)
{
Assert.IsFalse(IsLeapYear(year));
}
[Test]
}
}

{
[TestFixture]
public class A_year_is_a_leap_year
{
[Test]
public void if_it_is_divisible_by_4_but_not_by_100(
[Values(2016, 1984, 4)] int year) 
[Test]
public void if_it_is_divisible_by_400(
[Range(400, 2400, 400)] int year) 
}
[TestFixture]
{
[Test]
public void if_it_is_not_divisible_by_4(
[Values(2018, 2017, 42, 1)] int year) 
[Test]
public void if_it_is_divisible_by_100_but_not_by_400(
[Values(2100, 1900, 100)] int year) 
}
}

{
[TestFixture]
public class A_year
{
[Test]
public void is_either_a_leap_year_or_not([Range(1, 10000)] int year) 
}
}

{
[TestFixture]
public class A_year
{
[Test]
public void is_either_a_leap_year_or_not([Range(1, 10000)] int year)
{
Assert.AreEqual(
year % 4 == 0 && year % 100 != 0 || year % 400 == 0,
IsLeapYear(year));
}
}
}

{
[TestFixture]
public class A_year
{
[Test]
public void is_either_a_leap_year_or_not([Range(1, 10000)] int year)
{
Assert.AreEqual(LeapYearExpectation(year), IsLeapYear(year));
}
public static bool LeapYearExpectation(int year)
{
return year % 4 == 0 && year % 100 != 0 || year % 400 == 0;
}
}
}

{
return year % 4 == 0 && year % 100 != 0 || year % 400 == 0;
}

All happy families are alike;
each unhappy family is
unhappy in its own way.
Leo Tolstoy
Anna Karenina

{
[TestFixture]
[TestFixture]
public class A_year_is_not_a_leap_year 
[TestFixture]
public class A_year_is_not_supported
{
[Test]
public void if_it_is_0()
{
Assert.Throws<ArgumentException>(() => IsLeapYear(0));
}
}
}

{
[TestFixture]
[TestFixture]
[TestFixture]
{
[Test]
{
Assert.Catch<ArgumentException>(() => IsLeapYear(0));
}
}
}

{
[TestFixture]
[TestFixture]
[TestFixture]
{
[Test]
{
Assert.Catch<ArgumentException>(() => IsLeapYear(0));
}
[Test]
public void if_it_is_negative(
[Values(-1, -4, -100, -400)] int year)
{
Assert.Catch<ArgumentException>(() => IsLeapYear(year));
}
}
}

{
[TestFixture]
[TestFixture]
[TestFixture]
public class A_year_is_supported
{
[Test]
public void if_it_is_positive(
[Values(1, int.MaxValue)] int year)
{
Assert.DoesNotThrow(() => IsLeapYear(year));
}
}
[TestFixture]
public class A_year_is_not_supported 
}

Stack
{new, push, pop, depth, top}

An abstract data type defines a
class of abstract objects which
is completely characterized by
the operations available on
those objects.
Barbara Liskov
Programming with Abstract Data Types

 T •  Stack
{
new: Stack[T],
push: Stack[T]  T → Stack[T],
pop: Stack[T] ⇸ Stack[T],
depth: Stack[T] → Integer,
top: Stack[T] ⇸ T
}

template<typename T>
class stack
{
public:
stack();
void push(const T & new_top);
void pop();
std::size_t depth() const;
T top() const;
private:
...
};

TEST_CASE(“test stack::stack”) 
TEST_CASE(“test stack::push”) 
TEST_CASE(“test stack::pop”) 
TEST_CASE(“test stack::depth”) 
TEST_CASE(“test stack::top”) 

TEST_CASE(“stack tests”)
{
SECTION(“test constructor”) 
SECTION(“test push”) 
SECTION(“test pop”) 
SECTION(“test depth”) 
SECTION(“test top”) 
}

{
SECTION(“constructor”) 
SECTION(“push”) 
SECTION(“pop”) 
SECTION(“depth”) 
SECTION(“top”) 
}

class stack
{
public:
stack() = default;
void pop();
T top() const;
private:
...
};

{
SECTION(“push”) 
SECTION(“pop”) 
SECTION(“depth”) 
SECTION(“top”) 
}

{
SECTION(“can be constructed”) 
SECTION(“can be pushed”) 
SECTION(“can be popped”) 
SECTION(“has depth”) 
SECTION(“has a top”) 
}

{
SECTION(“can be constructed”) 
SECTION(“can be pushed”) 
SECTION(“can sometimes be popped”) 
SECTION(“has depth”) 
SECTION(“sometimes has a top”) 
}

https://twitter.com/chrisoldwood/status/918139432447954944

Given
When
Then
an empty stack
an item is pushed
it should not be empty

Given
When
Then
an empty stack
an item is pushed
if it's OK with you, I
think that, perhaps, it
should probably not be
empty, don't you think?

Make definite assertions.
Avoid tame, colourless, hesitating, noncommittal
language.
When a sentence is made stronger, it usually becomes
shorter. Thus brevity is a by-product of vigour.
William Strunk and E B White
The Elements of Style

Given
When
Then
an empty stack
an item is pushed
it must not be empty

Given
When
Then
an empty stack
an item is pushed
it is not empty

Given
When
Then
And
an empty stack
an item is pushed
it has a depth of 1
the top item is the item
that was pushed

GivenAnEmptyStackWhenAnI
temIsPushedThenItHasADep
thOf1AndTheTopItemIsTheI
temThatWasPushed

Given_an_empty_stack_Whe
n_an_item_is_pushed_Then
_it_has_a_depth_of_1_And
_the_top_item_is_the_ite
m_that_was_pushed

Given an empty stack
When an item is pushed
Then it has a depth of 1
And the top item is the
item that was pushed

An empty stack acquires
depth by retaining a
pushed item as its top

Omit needless words.
William Strunk and E B White
The Elements of Style

TEST_CASE(“Stack specification”)
{

SECTION(“An empty stack acquires depth by retaining a pushed item as its top")
{
stack<std::string> stack;
stack.push("ACCU");
REQUIRE(stack.depth() == 1);
REQUIRE(stack.top() == "ACCU");
}

}

{
SECTION(“A new stack is empty”) 
SECTION(“An empty stack throws when queried for its top item”) 
SECTION(“An empty stack throws when popped”) 
SECTION(“An empty stack acquires depth by retaining a pushed item as its top”) 
SECTION(“A non-empty stack becomes deeper by retaining a pushed item as its top”) 
SECTION(“A non-empty stack on popping reveals tops in reverse order of pushing”) 
}

For each usage scenario, the test(s):
▪ Describe the context, starting point, or
preconditions that must be satisfied
▪ Illustrate how the software is invoked
▪ Describe the expected results or
postconditions to be verified
Gerard Meszaros

{

{
// Arrange:
// Act:
stack.push("ACCU");
// Assert:
}

}

{

{
// Given:
// When:
stack.push("ACCU");
// Then:
}

}

https://twitter.com/jasongorman/status/572829496342134784

Communications of the ACM, 12(10), October 1969

If the assertion P is true before
initiation of a program Q, then
the assertion R will be true on
its completion.

An interface is a contract to deliver a
certain amount of service.
Clients of the interface depend on the
contract, which is usually documented
in the interface specification.
Butler W Lampson
Hints for Computer System Design

class stack
{
public:
stack();
void pop();
const T & top() const;
private:
...
};
postcondition:
depth() == 0
where:
old_depth = depth()
postcondition:
depth() == old_depth + 1 && top() == new_top
where:
old_depth = depth()
precondition:
depth() > 0
postcondition:
depth() == old_depth – 1
where:
result = depth()
postcondition:
result >= 0
precondition:
depth() > 0

class stack
{
public:
stack();
void pop();
const T & top() const;
private:
...
};
[[expects: depth() == 0]]
[[ensures: depth() > 0]]
[[ensures: top() == new_top]]
[[expects: depth() > 0]]
[[ensures: depth() >= 0]]
[[ensures result: result >= 0]]
[[expects: depth() > 0]]

alphabet(Stack) =
{new, push, pop, depth, top}

trace(Stack) =
{⟨new⟩,
⟨new, push⟩,
⟨new, depth⟩,
⟨new, push, pop⟩,
⟨new, push, top⟩,
⟨new, push, depth⟩,
⟨new, push, push⟩,
⟨new, depth, push⟩,
⟨new, depth, depth⟩,
⟨new, push, push, pop⟩,
...}

Non-EmptyEmpty
depth depth
top
push
pop [depth > 1]
push
pop [depth = 1]
new

Non-EmptyEmpty
depth
top / error
pop / error
depth
top
push
pop [depth > 1]
push
pop [depth = 1]
new

TEST_CASE("Stack specification")
{
SECTION("A new stack")
{
SECTION("is empty") ...
}
SECTION("An empty stack")
{
SECTION("throws when queried for its top item") ...
SECTION("throws when popped") ...
SECTION("acquires depth by retaining a pushed item as its top") ...
}
SECTION("A non empty stack")
{
SECTION("becomes_deeper_by_retaining_a_pushed_item_as_its_top") ...
SECTION("on popping reveals tops in reverse order of pushing") ...
}
}

{
{
...
}
{
...
SECTION("acquires depth by retaining a pushed item as its top")
{
stack.push("ACCU");
}
}
{
...
}
}

{
{
...
}
{
...
{
stack.push("ACCU");
}
}
{
...
}
}

SCENARIO("Stack specification")
{
...
GIVEN("An empty stack")
{
...
WHEN("an item is pushed")
{
stack.push("ACCU");
THEN(“acquires depth by retaining the new item as its top”)
{
}
}
}
...
}

{
...
{
...
{
stack.push("ACCU");
THEN(“acquires depth”)
{
}
THEN(“retains the new item as its top”)
{
}
}
}
...
}

{
...
{
...
{
stack.push("ACCU");
THEN(“acquires depth”)
THEN(“retains the new item as its top”)
}
}
...
}

Given can be used to group
tests for operations with
respect to common
initial state

When can be used to group
tests by operation,
regardless of initial state
or outcome

Then can be used to group
tests by common
outcome, regardless of
operation or initial state

{
{
...
}
{
...
{
stack.push("ACCU");
REQUIRE(stack.items.size() == 1);
REQUIRE(stack.items[0] == "ACCU");
}
}
{
...
}
}

{
{
...
}
{
...
{
stack.push("ACCU");
REQUIRE(stack.get_items().size() == 1);
REQUIRE(stack.get_items()[0] == "ACCU");
}
}
{
...
}
}

{
{
...
}
{
...
{
stack.push("ACCU");
REQUIRE(stack.items().size() == 1);
REQUIRE(stack.items()[0] == "ACCU");
}
}
{
...
}
}

A programmer [...] is concerned
only with the behavior which
that object exhibits but not with
any details of how that behavior
is achieved by means of an
implementation.
Barbara Liskov
Programming with Abstract Data Types

class stack
{
public:
std::size_t depth() const
{
return items.size();
}
const T & top() const
{
if (depth() == 0)
throw std::logic_error("stack has no top");
return items.back();
}
void pop()
{
if (depth() == 0)
throw std::logic_error("cannot pop empty stack");
items.pop_back();
}
void push(const T & new_top)
{
items.push_back(new_top);
}
private:
std::vector<T> items;
};

class stack
{
public:
std::size_t depth() const
{
return size;
}
const T & top() const
{
if (depth() == 0)
throw std::logic_error("stack has no top");
return items.front();
}
void pop()
{
if (depth() == 0)
throw std::logic_error("cannot pop empty stack");
items.pop_front();
--size;
}
void push(const T & new_top)
{
items.push_front(new_top);
++size;
}
private:
std::size_t size = 0;
std::forward_list<T> items;
};

Queue
{new, length, capacity, enqueue, dequeue}

producer consumerspacetime decoupling

N
buffered
bounded
asynchronous

N =
buffered
unbounded
asynchronous
∞

N = 1
buffered
bounded
asynchronous
futurepromise

N = 0
unbuffered
bounded
synchronous

public class Queue<T>
{
...
public Queue(int capacity) ...
public int length() ...
public int capacity() ...
public boolean enqueue(T last) ...
public Optional<T> dequeue() ...
}

Non-EmptyEmpty
length
capacity
dequeue
length
capacity
enqueue
dequeue [length > 1]
enqueue
dequeue [length = 1]
new [capacity > 0]

Non-Full
Empty
length
capacity
dequeue
length
capacity
enqueue
dequeue [length > 1]
enqueue
dequeue [length = 1]
new [capacity > 0]
Full
Non-Empty
[length = capacity]
[length < capacity]

public class Queue_spec 
public class A_new_queue 
public void is_empty() 
public void preserves_positive_bounding_capacity(int capacity) 
public void cannot_be_created_with_non_positive_bounding_capacity(int capacity) 
public class An_empty_queue 
public void dequeues_an_empty_value() 
public void remains_empty_when_null_enqueued() 
public void becomes_non_empty_when_non_null_value_enqueued(String value) 
public class A_non_empty_queue 
public class that_is_not_full 
public void becomes_longer_when_non_null_value_enqueued(String value) 
public void becomes_full_when_enqueued_up_to_capacity() 
public class that_is_full 
public void ignores_further_enqueued_values() 
public void becomes_non_full_when_dequeued() 
public void dequeues_values_in_order_enqueued() 
public void remains_unchanged_when_null_enqueued() 

public class Queue<T>
{
private Deque<T> items = new LinkedList<>();
private final int capacity;
public Queue(int boundingCapacity)
{
if (boundingCapacity < 1)
throw new IllegalArgumentException();
capacity = boundingCapacity;
}
public int length()
{
return items.size();
}
public int capacity()
{
return capacity;
}
public boolean enqueue(T last)
{
boolean enqueuing = last != null && length() < capacity();
if (enqueuing)
items.addLast(last);
return enqueuing;
}
public Optional<T> dequeue()
{
return Optional.ofNullable(items.pollFirst());
}
}

Structure and Interpretation of Test Cases

More Related Content

What's hot (20)

Similar to Structure and Interpretation of Test Cases (20)

More from Kevlin Henney (20)

Recently uploaded (20)

Structure and Interpretation of Test Cases