A Replay Approach to Software Validation

A Replay Approach to Software Validation Using
Regular Expressions in C++11 and Python
James Pascoe
james@james-pascoe.com
www.james-pascoe.com

This is a story in two halves …
1. A ‘replay’ approach to software validation
• Replay in what sense?
• Why is this useful?
• Replay for The Android Hardware Composer
2. Why I love Regular Expressions and why you should too
• When (and when not) to use them
• Regular expressions in C++11 and Python
• Building a Replay Tool

Replay
• Most software produces some form of textual logging:
• Readable by humans (often for debug)
• Already in use by customers, validation engineers, triage teams
• Ideal for further processing …
• Instead of viewing logs purely as output, we can:
1. Use C++11 Regular Expressions to parse them
2. Interpret time-stamps to recreate event timing
3. Generate the original input stimuli
4. Call the underlying API at the ‘correct’ times

Replay Benefits
• Reduces the length of time for validation cycles
• Recreates bugs faster (and deterministically)
• Much easier to share experimental setups
• More effective bug triage
• Add automated test cases quickly
• Generate stimulus cases for benchmarking
• Create pathological test-cases
• Support TDD/BDD development activities

Surface Flinger
Android Surface Composition
Applications
Window
Manager
Hardware
Composer
Display
Driver
OpenGL ES
GPU
Overlay Planes
Encoder
1
23
4
5
6

The Hardware Composer
• Complex, highly dynamic, multi-threaded power optimisation engine
• Called by SurfaceFlinger. Determines which ‘layers’ should be:
1. Composed in software (GPU / OpenGL)
2. Mapped to hardware directly
• Next buffer has to be ready in 16ms (i.e. 60hz refresh):
• All HWC decision making, compositions, surface flinger, buffer writes …
Less power More power<- Static content … Dynamic content ->
GPU Composition and reuse … … Overlay planes… composition / hardware mix …

Hardware Composer API
/* (*prepare)() is called for each frame before composition
and is used by SurfaceFlinger to detect the compositions that
the HWC can handle. */
int (*prepare)(struct hwc_composer_device_1 *dev,
size_t numDisplays, hwc_display_contents_1_t** displays);
/* (*set)() is called to update the displays with the content
of their work lists. */
int (*set)(struct hwc_composer_device_1 *dev,
size_t numDisplays, hwc_display_contents_1_t** displays);
https://github.com/android/platform_hardware_libhardware/blob/master/include/hardware/hwcomposer.h

Hardware Composer Log File Extract
94257s 259ms 311115ns TID:16192 SF0 onPrepare Exit frame:6 Fd:-1 outBuf:0x0 outFd:-1 flags:0
0 OV 0x7f8d49e38e70:21:0 60 BL:1.00 RGBA:X 1920x1200 0.0, 0.0,1920.0,1200.0
0,0,1920,1200 -1 -1 V: 0, 0,1920,1200 U:00000b00 Hi:0 Fl:0 A BL
1 OV 0x7f8d49e38fb0:25:0 60 BL:1.00 RGBA:X 600x400 0.0, 0.0, 600.0, 400.0
300,200,900,600 -1 -1 V: 300, 200, 900, 600 U:00000b00 Hi:0 Fl:0 A BL
2 TG 0x7f8d49e38880:23:0 60 BL:1.00 RGBA:X 1920x1200 0.0, 0.0,1920.0,1200.0
0,0,1920,1200 -1 -1 V: 0, 0,1920,1200 U:00001a00 Hi:0 Fl:0 A BL
Timestamp
Thread Id
Individual layer fields

Parsing an HWC Log File – Incumbent Solution
bool Parse(std::string const& str)
{
int layer_num;
char layer_flag[3], layer_handle[13];
int num_parsed = std::sscanf(str.c_str(), " %d %s 0x%s:",
&layer_num, layer_flag, layer_handle);
int layer_handle_num = atoi(layer_handle);
if (num_parsed == 3)
{ …

Drawbacks
• No atomicity
• The code breaks (subtly) when the log format changes (and it does often)
• Code is always ‘slightly’ broken – no confidence in results
• Verbosity
• sscanf format string is not flexible enough to cover all variations
• Lots of duplication - required over 3 kloc just for parsing !
• Fragmentation
• Number of leaf functions is huge !
• All require unit-tests, causes code bloat, overall logic is obscured …

Regular Expression History
• Started in the 1940s with two Neurophysiologists:1
• Warren McCulloch and Walter Pitts
• Stephen Kleene described these models in algebra - ‘regular sets’
• Kleene devised a notation to express these sets – ‘regular expressions’
• In 1968, Ken Thompson wrote a regular expression compiler that
produced IBM 7094 object code:
• This led to work on qed - the editor which became ed on Unix
• ed had a command to display lines of a file that matched a regular expression
• g/Regular Expression/p was read ‘Global Regular Expression Print’
and became grep (which was later extended into egrep)
[1] Mastering Regular Expressions, 3rd Edition, Jeffrey E.F. Friedl, O'Reilly, 2006.

Regular Expressions in C++11
• Full match, any match and replacement:
• regex_match(), regex_search(), regex_replace()
• Iterate over matches / tokenize:
• regex_iterator(), regex_token_iterator()
• Constants (defined in std::regex_constants):
• Syntax options – case insensitivity, which regular expression grammar …
• Match options – defines whether the first character matches ^ …
• Error types – thrown when parsing badly formed regular expressions

C++11 ECMAScript Regex Refresher

Regular Expressions for Replay
94257s 166ms 673083ns TID:16192 SF0 onSet Entry frame:0 Fd:-1 outBuf:0x0 outFd:-1 flags:1
const std::string onset_string =
R"REGEX(^(d+)s (d+)ms (?:(d+)ns )?(?:TID:(d+) )?SF(d) onSet Entry )REGEX"
R"REGEX((?:frame:(d+) )?Fd:(-?d{1,2}) )REGEX"
R"REGEX(outBuf:0x(.{1,8}) outFd:(-?d{1,2}) [fF]lags:(d+)(:?.*)$)REGEX";

Regular Expressions for Replay
1 OV 0x7f8d49e38f60:22:0 60 BL:1.00 RGBA:X 600x400 0.0, 0.0, 600.0, 400.0
300, 200, 900, 600 -1 -1 V: 300, 200, 900, 600 U:00000b00 Hi:0 Fl:0 A BL
const std::string layer_string_hdr =
"^s*(d+) (w{2}) *0x(.{1,12}): ?(?:--|(d{1,3})): ?(d) ?(d+) "
"(w{2}): ?(.{1,4}) (w{1,5}) *:[XLY] *(d{1,4})x(d{1,4}) * "
" *(-?d+.?d*), *(-?d+.?d*), *(-?d+.?d*), *(-?d+.?d*)"
" *(-?d{1,4}), *(-?d{1,4}), *(-?d{1,4}), *(-?d{1,4}) "
"(-?d+) (-?d+) V: *(d{1,4}), *(d{1,4}), *(d{1,4}), *(d{1,4}) ";
const std::string layer_string_trl =
" *U:(.{1,8}) * Hi:(d+)(:?[[:alpha:]]*)? Fl:(d+)(:?[ [:alnum:]]*).*";

Matching and Searching
std::regex onset_regex(onset_string);
. . .
auto start = high_resolution_clock::now();
while (getline(infile, line))
{
++lines;
if (std::regex_match(line, onset_regex) ||
(std::regex_search(line, layer_hdr_regex) && std::regex_search(line, layer_trl_regex)))
++matches;
}
auto end = high_resolution_clock::now();
std::cout << "Took " << duration_cast<milliseconds>(end-start).count() << " ms for " << matches <<
" matches (" << lines << " lines processed)" << std::endl;
 Took 2906 ms for 2605 matches over 7269 lines
(Averaged over 10,000 runs) ~ 0.4 ms per line
Clang++-3.8 running on an I7 / 32 Gb (Ubuntu 14.04)

Capture Groups
Values are held in specialisations of std::match_results:
std::smatch onset_matches; // String specialization of std::match_results
if (std::regex_match(line, onset_matches, onset_regex))
{
std::cout << "Saw onSet Entry with timestamp " <<
onset_matches[1] << "s " <<
onset_matches[2] << "ms " <<
std::stoi(onset_matches[3]) << "ns " << std::endl;
// Note: onset_matches[0] is the whole match !
}

Advice
• Regular expressions are dense (with lots of different grammars)
• POSIX, ECMAScript, Python, Perl, JavaScript, awk, grep, egrep
• (C++11 regular expressions and Python re are compatible  ! )
• When writing Regular Expressions, start small and build
• Start with: ^.*$ and not ^(?:(?:(?:0?[13578]|1[02]) ...
• Use visualisation websites:
• regexpal.com and regexplained.co.uk
• Raw String Literals can be useful for Regexs
• Use counters to provide checks that are easy to verify with grep

How Do We Validate All of This?
• Question: how do you validate a Replay tool?
• Answer: compare the replayed log to the original log
• C++11 Regular Expressions and Python are compatible!
• We can develop validation / visualisation tools  !
• PyReplay:
• Compares two HWC log files. Identifies subtle mismatches
• ~250 lines of Python (with comments). Based on TkInter
• Python 3 regular expressions (re) are the same as C++11 !

Regular Expressions in Python
onset_string =
r'^(d+)s (d+)ms (?:(d+)ns )?(?:TID:(d+) )?SF(d) onSet Entry '
r'(?:frame:(d+) )?Fd:(-?d{1,2}) '
r'outBuf:0x(.{1,8}) outFd:(-?d{1,2}) [fF]lags:(d+)(:?.*)$'
layer_string_hdr =
"^s*(d+) (w{2}) *0x(.{1,12}): ?(?:--|(d{1,3})): ?(d) ?(d+) "
"(w{2}): ?(.{1,4}) (w{1,5}) *:[XLY] *(d{1,4})x(d{1,4}) *"
" *(-?d+.?d*), *(-?d+.?d*), *(-?d+.?d*), *(-?d+.?d*)"
" *(-?d{1,4}), *(-?d{1,4}), *(-?d{1,4}), *(-?d{1,4}) (-?d+) (-?d+) "
"V: *(d{1,4}), *(d{1,4}), *(d{1,4}), *(d{1,4}) ";
layer_string_trl =
" *U:(.{1,8}) * Hi:(d+)(:?[[:alpha:]]*)? Fl:(d+)(:?[ [:alnum:]]*).*";

A Replay Approach to Software Validation

Conclusions
• Replay:
• Found numerous real bugs and often very subtle
• Had some great cultural benefits and insights
• Particularly when combined with Jenkins / CI
• Low-cost to implement and maintain
• Compatibility of std::regex and Python re is very useful
• Limitations:
• Replay: multithreading can disrupt bug reproducibility
• Better to use fuzz-testing + coverage + sanitisers for that
• Regular expressions: not the right tool for sophisticated lexical analysis
• Probably want to build something like an Abstract Syntax Tree for that …

#include <iostream>
int main()
{
std::string questions;
while (1)
{
std::cout << "Questions?" << std::endl;
if (std::cin >> questions && questions == "Y")
std::cout << "Answers" << std::endl;
else
goto cornubia;
}
cornubia:
std::cout << "Thank you for coming !" << std::endl;
}

A Replay Approach to Software Validation

More Related Content

What's hot (20)

Similar to A Replay Approach to Software Validation (20)

A Replay Approach to Software Validation