Quickly testing legacy code

1
@ClareMacraeUK
Quickly Testing Legacy Code
Clare Macrae
5 February 2019

2
@ClareMacraeUK
@ClareMacraeUK

3
@ClareMacraeUK
Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

4
@ClareMacraeUK
Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

7
@ClareMacraeUK
A year of remote pairing later…

8
@ClareMacraeUK
Goal:
Share techniques for easier testing
in challenging scenarios

9
@ClareMacraeUK
Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

10
@ClareMacraeUK

12
@ClareMacraeUK
What does Legacy Code really mean?
• Michael Feathers
–“code without unit tests”
• J.B. Rainsberger
–“profitable code that we feel afraid to change.”
• Kate Gregory
–“any code that you want to change, but are afraid to”

13
@ClareMacraeUK
Typical Scenario
• I've inherited some
legacy code
• It's valuable
• I need to add feature
• Or fix bug
• How can I ever break
out of this loop?
Need to
change
the code
No tests
Not
designed
for testing
Needs
refactoring
to add
tests
Can’t
refactor
without
tests

14
@ClareMacraeUK
Typical Scenario
•
•
•
•
•
Need to
change
the code
No tests
Not
designed
for testing
Needs
refactoring
to add
tests
Can’t
refactor
without
tests
Topics of
this talk

15
@ClareMacraeUK
Assumptions
• Value of testing
• No worries about types of tests
– (unit, integration, regression)

16
@ClareMacraeUK
Any existing tests?

17
@ClareMacraeUK
What, no tests?
• If absolutely no tests…
• Stop now!
• Set one up!
• Existing framework

18
@ClareMacraeUK
Popular Test Frameworks
Google Test
• Google's C++ test framework
• https://github.com/google/googletest
Catch
• Phil Nash’s test framework
• https://github.com/catchorg/Catch2

20
@ClareMacraeUK
How good are your existing tests?

21
@ClareMacraeUK
First evaluate your tests
• If you do have tests….
• Test the tests!
• In area you are changing
• Reliable?

22
@ClareMacraeUK
Code Coverage
• Caution!
• Unexecuted code
• Techniques
• Debugger
• Code coverage tool
– What to measure?

23
@ClareMacraeUK
100% Test Coverage
Or is it?
OpenCppCoverage does not show
branch/condition coverage.

24
@ClareMacraeUK
BullseyeCoverage:
Unrolls If conditions
(line 10)
73% of conditions
covered

25
@ClareMacraeUK
Mutation testing: Sabotage the code!
• Test the tests
• Small changes
• Re-run tests
• Fail: 
• Pass: 

26
@ClareMacraeUK
Mutation testing approaches
• By hand,
– e.g. + to -
• By tool
– e.g. Mutate++ - https://github.com/nlohmann/mutate_cpp

27
@ClareMacraeUK
If tests need improving…
And unit tests not viable…

28
@ClareMacraeUK
Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

29
@ClareMacraeUK

30
@ClareMacraeUK
Key Phrase :
“Locking Down Current Behaviour”

31
@ClareMacraeUK
Writing Unit Tests for Legacy Code
• Time-consuming
• What’s intended behaviour?
• Is there another way?

32
@ClareMacraeUK
Alternative: Golden Master Testing

33
@ClareMacraeUK
Golden Master Test Setup
Input
Data
Existing
Code
Save
“Golden
Master”

34
@ClareMacraeUK
Golden Master Tests In Use
Input
Data
Updated
Code
Pass
Fail
Output
same?
Yes
No

35
@ClareMacraeUK
Thoughts on Golden Master Tests
• Good to start testing legacy systems
• Poor Person’s Integration Tests)
• Depends on ease of
– Capturing output
– Getting stable output
– Reviewing any differences
– Avoiding overwriting Master by mistake!
• Doesn’t matter that it’s not a Unit Test

36
@ClareMacraeUK
Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

37
@ClareMacraeUK

38
@ClareMacraeUK
One approach: “ApprovalTests”
• Llewellyn Falco’s convenient, powerful, flexible Golden Master implementation
• Supports many language
And now C++!
GO
Java
Lua
.NET
.Net.ASP
NodeJS
Objective-C
PHP
Perl
Python
Swift
TCL

39
@ClareMacraeUK
ApprovalTests.cpp
• New C++ library for applying Llewellyn Falco’s “Approval Tests” approach
• For testing cross-platform C++ code (Windows, Linux, Mac)
• For legacy and green-field systems
• It’s on github

40
@ClareMacraeUK
ApprovalTests.cpp
• Header-only
• Open source - Apache 2.0 licence

41
@ClareMacraeUK
ApprovalTests.cpp
• Works with a range of testing frameworks
• Currently supports Catch1, Catch2, Google Test and Okra

42
@ClareMacraeUK
Getting Started with Approvals

43
@ClareMacraeUK
StarterProject with Catch2
• https://github.com/approvals/ApprovalTests.cpp.StarterProject
• Download ZIP

44
@ClareMacraeUK
How to use it: Catch2 Boilerplate
• Your main.cpp
#define APPROVALS_CATCH
#include "ApprovalTests.hpp"

45
@ClareMacraeUK
Pure Catch2 Test
• A test file
#include "Catch.hpp"
// Catch-only test
TEST_CASE( "Sums are calculated" )
{
REQUIRE( 1 + 1 == 2 );
REQUIRE( 1 + 2 == 3 );
}

46
@ClareMacraeUK
Approvals Catch2 Test
• A test file (Test02.cpp)
#include "Catch.hpp"
// Approvals test - test static value, for demo purposes
TEST_CASE("TestFixedInput")
{
Approvals::verify("SomenMulti-linenoutput");
}

47
@ClareMacraeUK
First run
• 1.
• 2. Differencing tool pops up (Araxis Merge, in this case)

48
@ClareMacraeUK
Actual/Received Expected/Approved

49
@ClareMacraeUK

50
@ClareMacraeUK

51
@ClareMacraeUK
Second run
• I approved the output
• Then we commit the test, and the approved file(s) to version control
• The diffing tool only shows up if:
– there is not (yet) an Approved file or
– the Received differs from the Approved

52
@ClareMacraeUK
What’s going on here?
• It’s a very convenient form of Golden Master testing
• Objects being tested are stringified – that’s a requirement
• Captures snapshot of current behaviour
• Useful for locking down behaviour of existing code
– Not intended to replace Unit Tests

53
@ClareMacraeUK
Example test directory

54
@ClareMacraeUK
How to use it: GoogleTest Boilerplate
• Your main.cpp
#define APPROVALS_GOOGLETEST

55
@ClareMacraeUK
Pure Google Test
• A test file
#include <gtest/gtest.h>
// Google-only test
TEST( Test01, SumsAreCalculated )
{
EXPECT_EQ( 1 + 1, 2 );
EXPECT_EQ( 1 + 2, 3 );
}

56
@ClareMacraeUK
Approvals Google Test
• A test file
#include <gtest/gtest.h>
// googletest - test static value, for demo purposes
TEST( Test02, TestFixedInput )
{
Approvals::verify("SomenMulti-linenoutput");
}

57
@ClareMacraeUK
Reference: Add to existing GoogleTest
• Your main.cpp
#define APPROVALS_GOOGLETEST_EXISTING_MAIN
int main(int argc, char** argv)
{
::testing::InitGoogleTest(&argc, argv);
initializeApprovalTestsForGoogleTests();
return RUN_ALL_TESTS();
}

58
@ClareMacraeUK
Reference: Supporting a new test framework
• Any test framework that supplies the following:
– Current test’s name
– Current test’s source file (with correct case of filename)
– Report unexpected exceptions as a test failure
• That’s all that’s needed

59
@ClareMacraeUK
A note on Tools
• I’ll be using a variety of tools in the talk
• I’ll mention them as I go
• Slides of references at the end

60
@ClareMacraeUK
How does this help with legacy code?

61
@ClareMacraeUK
Applying this to legacy code
TEST_CASE("New test of legacy feature")
{
// Standard pattern:
// Wrap your legacy feature to test in a function call
// that returns some state that can be written to a text file
// for verification:
const LegacyThing result = doLegacyOperation();
Approvals::verify(result);
}

62
@ClareMacraeUK
Implementation
class LegacyThing;
std::ostream &operator<<(std::ostream &os, const LegacyThing &result) {
// write interesting info from result here:
os << result…;
return os;
}
LegacyThing doLegacyOperation() {
// your implementation here..
return LegacyThing();
}

63
@ClareMacraeUK
Feature: Consistency over machines
• Naming of output files
• Approved files version controlled

64
@ClareMacraeUK
Feature: Consistency over OSs
• Line-endings

65
@ClareMacraeUK
Feature: Quick to write tests
TEST_CASE("verifyAllWithHeaderBeginEndAndLambda")
{
std::list<int> numbers{ 0, 1, 2, 3};
// Multiple convenience overloads of Approvals::verifyAll()
Approvals::verifyAll(
"Test Squares", numbers.begin(), numbers.end(),
[](int v, std::ostream& s) { s << v << " => " << v*v << 'n' ; });
}

66
@ClareMacraeUK
Feature: Quick to get good coverage
TEST_CASE("verifyAllCombinationsWithLambda")
{
std::vector<std::string> strings{"hello", "world"};
std::vector<int> numbers{1, 2, 3};
CombinationApprovals::verifyAllCombinations<
std::vector<std::string>, // The type of element in our first input container
std::vector<int>, // The type of element in our second input container
std::string>( // The return type from testing one combination of inputs
// Lambda that acts on one combination of inputs, and returns the result to be approved:
[](std::string s, int i) { return s + " " + std::to_string(i); },
strings, // The first input container
numbers); // The second input container
}

67
@ClareMacraeUK
All values in single output file:
(hello, 1) => hello 1
(world, 1) => world 1

68
@ClareMacraeUK
Challenge: Golden Master is a log file
• Dates and times?
• Object addresses?
2019-01-29 15:27

69
@ClareMacraeUK
Options for unstable output
• Introduce date-time abstraction?
• Customised comparison function?
• Or: strip dates from the log file

70
@ClareMacraeUK
Tip: Rewrite output file
2019-01-29 15:27 [date-time-stamp]

71
@ClareMacraeUK
Customisability: ApprovalWriter interface
class ApprovalWriter
{
public:
virtual std::string getFileExtensionWithDot() = 0;
virtual void write(std::string path) = 0;
virtual void cleanUpReceived(std::string receivedPath) = 0;
};

72
@ClareMacraeUK
Feature: Run on build servers
TEST_CASE("UseQuietReporter")
{
// QuietReporter does nothing if a failure occurs.
// Failing tests will still fail, but nothing is launched.
Approvals::verify(
"SomenMulti-linenoutput",
QuietReporter());
}

73
@ClareMacraeUK
Customisability: Reporters
• Very simple but powerful abstraction
• Gives complete user control over how to inspect and act on a failure
• If you adopt Approvals, do review the supplied reporters for ideas

74
@ClareMacraeUK
Tip: currentReporter()
• Easy to switch reporter behaviour

75
@ClareMacraeUK
Feature: Convention over Configuration
• Fewer decisions for developers
• Users only specify unusual behaviours

76
@ClareMacraeUK
Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

77
@ClareMacraeUK
What I can do:

78
@ClareMacraeUK
What I need to do:

79
@ClareMacraeUK
Sugar - sucrose

80
@ClareMacraeUK
Sugar - sucrose

81
@ClareMacraeUK
Sugar - sucrose

82
@ClareMacraeUK
Sugar - sucrose

83
@ClareMacraeUK
Sugar - sucrose

85
@ClareMacraeUK
Approving Images

86
@ClareMacraeUK
What I’m aiming for
TEST(PolyhedralGraphicsStyleApprovals, CUVXAT)
{
// CUVXAT identifies a crystal structure in the database
const QImage crystal_picture = loadEntry("CUVXAT");
verifyQImage(crystal_picture, *currentReporter());
}

87
@ClareMacraeUK
Foundation of Approval tests
void FileApprover::verify(
ApprovalNamer& namer,
ApprovalWriter& writer,
const Reporter& reporter);

88
@ClareMacraeUK
Idiomatic verification of Qt image objects
void verifyQImage(
QImage image, const Reporter& reporter = DiffReporter())
{
ApprovalTestNamer namer;
QImageWriter writer(image);
FileApprover::verify(namer, writer, reporter);
}

89
@ClareMacraeUK
Customisability: ApprovalWriter interface
class ApprovalWriter
{
public:
virtual std::string getFileExtensionWithDot() = 0;
virtual void write(std::string path) = 0;
virtual void cleanUpReceived(std::string receivedPath) = 0;
};

90
@ClareMacraeUK
QImageWriter declaration
class QImageWriter : public ApprovalWriter
{
public:
explicit QImageWriter(QImage image, std::string fileExtensionWithDot = ".png");
std::string getFileExtensionWithDot() override;
void write(std::string path) override;
void cleanUpReceived(std::string receivedPath) override;
private:
QImage image_;
std::string ext;
};

91
@ClareMacraeUK
QImageWriter::write()
void QImageWriter::write(std::string path)
{
// Have to convert std::string to QString
image_.save( QString::fromStdString(path) );
}

92
@ClareMacraeUK
Useful feedback BeyondCompare 4

93
@ClareMacraeUK
Back In work…
• Previously, working from home via Remote Desktop
• The tests started failing when I ran them in work

94
@ClareMacraeUK
What? Re-running now gives random failures
Araxis Merge

95
@ClareMacraeUK
BeyondCompare 4

101
@ClareMacraeUK
Customisability: ApprovalComparator
class ApprovalComparator
{
public:
virtual ~ApprovalComparator() = default;
virtual bool contentsAreEquivalent(std::string receivedPath,
std::string approvedPath) const = 0;
};

102
@ClareMacraeUK
Create custom QImage comparison class
// Allow differences in up to 1/255 of RGB values at each pixel, as saved in 32-bit images
// sometimes have a few slightly different pixels, which are not visible to the human eye.
class QImageApprovalComparator : public ApprovalComparator
{
public:
bool contentsAreEquivalent(std::string receivedPath, std::string approvedPath) const override
{
const QImage receivedImage(QString::fromStdString(receivedPath));
const QImage approvedImage(QString::fromStdString(approvedPath));
return compareQImageIgnoringTinyDiffs(receivedImage, approvedImage);
}
};

103
@ClareMacraeUK
Register the custom comparison class
// Somewhere in main…
FileApprover::registerComparator(
".png",
std::make_shared<QImageApprovalComparator>());

104
@ClareMacraeUK
Does the difference matter?
• Legacy code is often brittle
• Testing makes changes visible
• Then decide if change matters
• Fast feedback cycle for efficient development

105
@ClareMacraeUK
Looking back
• User perspective
• Learned about own code
• Enabled unit tests for graphics problems
• Approvals useful even for temporary tests

106
@ClareMacraeUK
ApprovalTests Customisation Points
• Reporter
• ApprovalWriter
• ApprovalComparator
• ApprovalNamer

107
@ClareMacraeUK
Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

108
@ClareMacraeUK
References: Tools Used
• Diffing
– Araxis Merge: https://www.araxis.com/merge/
– Beyond Compare: https://www.scootersoftware.com/
• Code Coverage
– OpenCppCoverage and Visual Studio Plugin: https://github.com/OpenCppCoverage
– BullseyeCoverage: https://www.bullseye.com/

109
@ClareMacraeUK
Beautiful C++: Updating Legacy Code
• https://app.pluralsight.com/library/courses/cpp-updating-legacy-code/table-of-contents

110
@ClareMacraeUK
The Legacy Code Programmer’s Toolbox
• https://leanpub.com/legacycode
• The Legacy Code Programmer's Toolbox: Practical skills for software professionals working
with legacy code, by Jonathan Boccara

111
@ClareMacraeUK
Videos
https://youtu.be/aWiwDdx_rdo

112
@ClareMacraeUK
Adam Tornhill’s Books
• Mining actionable information from historical code bases (by Adam Tornhill)
– Your Code as a Crime Scene
– Software Design X-Rays: Fix Technical Debt with Behavioural Code Analysis

113
@ClareMacraeUK
Contents
• Introduction
• Legacy Code
• Golden Master
• Approval Tests
• Example
• Resources
• Summary

114
@ClareMacraeUK
Adopting legacy code
• You can do it!
• Evaluate current tests
• Quickly improve coverage with Golden Master
• ApprovalTests.cpp makes that really easy
• Even for non-text types

115
@ClareMacraeUK
ApprovalTests
• Powerful Golden Master tool
• verify(), verifyAll(), verifyAllCombinations()
• Adjust output files, after writing, to simplify comparison

116
@ClareMacraeUK
ApprovalTests.cpp
• Not (always) a replacement for Unit Tests!
Golden
Master
Refactor
code
Add unit
tests
Archive
Approval
Tests?

117
@ClareMacraeUK
Thank You: Any Questions?
• Example Code: https://github.com/claremacrae/cpponsea2019
• Slides: https://www.slideshare.net/ClareMacrae
– (later this week)
• ApprovalTests.cpp
– https://github.com/approvals/ApprovalTests.cpp
– https://github.com/approvals/ApprovalTests.cpp.StarterProject

Quickly testing legacy code

More Related Content

Similar to Quickly testing legacy code (20)

More from Clare Macrae (6)

Recently uploaded (20)

Quickly testing legacy code

Editor's Notes