![Coverage-guiding in action
if input[0] == '{' {
if input[1] == 'i' && input[2] == 'f' {
if input[3] == '(' {
input[input[4]] = input[5]; // potential OOB write
}
}
}
Requires "{if(" input to crash, ~2^32 guesses to crack when blind.
Coverage-guiding:
Guess "{" in ~2^8, add to corpus.
Guess "{i" in ~2^8, add to corpus.
Guess "{if" in ~2^8, add to corpus.
Guess "{if(" in ~2^8, add to corpus.
Total: ~2^10 guesses.
See: AFL: Pulling JPEGs out of thin air](https://image.slidesharecdn.com/fuzzing-thenewunittestingcrussia-170220114321/85/Fuzzing-The-New-Unit-Testing-10-320.jpg)
Fuzzing: The New Unit Testing
- 1. Fuzzing: The New Unit Testing C++ Russia 2017, Moscow, Feb 25 Dmitry Vyukov, dvyukov@, Google
- 2. Agenda ● What is fuzzing ● Coverage-guided fuzzing ● Small tutorial ● How to write effective fuzzers ● Fuzzing@Google
- 3. What is Fuzzing? wikipedia.org/wiki/Fuzz_testing: Fuzz testing or fuzzing is a software testing technique, often automated or semi-automated, that involves providing invalid, unexpected, or random data to the inputs of a computer program.
- 4. Who cares? - We are not testing/checking anything! - Random data will not trigger any bugs!
- 5. Fuzzing can find lots of bugs - With the help of sanitizers: - Use-after-free, buffer overflows - Uses of uninitialized memory - Memory leaks - Data races, deadlocks - Int/float overflows, bitwise shifts by invalid amount (other UB) - Plain crashes: - NULL dereferences, uncaught exceptions, div-by-zero - Resource usage bugs: - Memory exhaustion, hangs or infinite loops, infinite recursion (stack overflows) - Logical bugs (lots of, see below)
- 6. Data is not necessary "white noise" - There is number of tricks to generate "not so random" data - May or may not require some human help - If used correctly achieves very impressive code coverage
- 7. What can be fuzzed? Anything that consumes complex inputs: ● Parsers of any kind (xml, json, asn.1, pdf, truetype, ...) ● Media codecs (audio, video, raster & vector images, etc) ● Network protocols (HTTP, RPC, SMTP, MIME...) ● Crypto (boringssl, openssl) ● Compression (zip, gzip, bzip2, brotli, ...) ● Formatted output (sprintf, template engines) ● Compilers and interpreters (Javascript, PHP, Perl, Python, Go, Clang, ...) ● Regular expression matchers (PCRE, RE2, libc’s regcomp) ● Text/UTF processing (icu) ● Databases (SQLite) ● Browsers, text editors/processors (Chrome, vim, OpenOffice) ● OS Kernels (Linux), drivers, supervisors and VMs Must have for everything that consumes untrusted inputs, open to internet or otherwise security sensitive.
- 8. Types of Fuzzers - Grammar-based generation - Generate random inputs according to grammar rules - Peach, packetdrill, csmith, gosmith, syzkaller - Blind mutation - Requires a corpus of representative inputs, apply random mutations to them - ZZUF, Radamsa - Grammar reverse-engineering - Learn grammar from existing inputs using algorithmic approach of machine learning - Sequitur algorithm, go-fuzz - Symbolic execution + SAT solver - Synthesize inputs with maximum coverage using black magic - KLEE - Coverage-guided fuzzers - Genetic algorithm that strives to maximize code coverage - libFuzzer, AFL, honggfuzz, syzkaller - Hybrid
- 9. Coverage-guided fuzzing Build the program with code coverage instrumentation; Collect initial corpus of inputs (optional); while (true) { Choose a random input from corpus and mutate it; Run the target program on the input, collect code coverage; If the input gives new coverage, add mutation back to the corpus; }
- 10. Coverage-guiding in action if input[0] == '{' { if input[1] == 'i' && input[2] == 'f' { if input[3] == '(' { input[input[4]] = input[5]; // potential OOB write } } } Requires "{if(" input to crash, ~2^32 guesses to crack when blind. Coverage-guiding: Guess "{" in ~2^8, add to corpus. Guess "{i" in ~2^8, add to corpus. Guess "{if" in ~2^8, add to corpus. Guess "{if(" in ~2^8, add to corpus. Total: ~2^10 guesses. See: AFL: Pulling JPEGs out of thin air
- 11. Mutations ● erase/insert/change/shuffle bit/byte/bytes ● crossover/splice 2 inputs ● insert token from a dictionary ● insert magic numbers (2^10±1, 2^16±1, 2^31±1, 2^32±1) ● change an ASCII integer (e.g. "123" => "2465357635") ● ...
- 12. Coverage flavours Basic blocks: ... (A) if (...) { ... (B) } ... (C) -fsanitize-coverage=bb Edges: ... (A) if (...) { ... (B) } ... (C) -fsanitize-coverage=trace-pc-guard Gives better feedback signal. Counters: for (...) { ... (hit N times) } -fsanitize-coverage=8bit-counters Gives better feedback signal for loops and recursion.
- 13. Cracking hashes What about more complex cases? if (*(uint32_t*)input == crc32(input+4, size-4)) {...} if (*(uint64_t*)input == 0xBCEBC041BADBALL) {...}
- 14. Cracking hashes Intercept comparison operations: ● compiler intercepts int comparisons (-fsanitize-coverage=trace-cmp) ● runtime intercepts strcmp/memcmp and friends Several possibilities: ● extract int/string literals and insert them into inputs ● find one comparison operand in the input and replace with the other operand ● use PC^POPCNT(op1^op2) as "coverage" signal (Hamming distance)
- 15. Dictionaries ● User-provided ○ e.g. for HTTP: "HTTP/1.1", "Host", "Accept-Encoding" ● Automatically extracted from program ○ memcpy(input, "HTTP/1.1", 8)
- 16. Tutorial "...one of the most highly regarded and expertly designed C++ library projects in the world" boost.regex (latest version 1.63, in boost since 1.18)
- 17. Tutorial: fuzzing function As simple as: int LLVMFuzzerTestOneInput(const uint8_t * Data, size_t Size) { try { std::string str((char*)Data, Size); boost::regex e( str); boost::match_results<std::string::const_iterator> what; boost::regex_match(str, what, e, boost::match_default); } catch (const std::exception&) {} return 0; }
- 18. Tutorial: building (the hard part) 1. Build boost with coverage and AddressSanitizer: ./b2 cxxflags="-fsanitize-coverage=trace-pc-guard -fsanitize=address" toolset=clang install 2. Build fuzzer with coverage, AddressSanitizer and libFuzzer: clang++ fuzzer.cc -fsanitize-coverage=trace-pc-guard -fsanitize=address libFuzzer.a The rest is at tutorial.libfuzzer.info
- 19. Demo
- 20. 30 minutes, 13 bugs (ticket/12818): AddressSanitizer: heap-buffer-overflow perl_matcher.hpp:132 in re_skip_past_null AddressSanitizer: heap-buffer-overflow basic_regex_parser.hpp:2599 in parse_perl_extension AddressSanitizer: heap-buffer-overflow perl_matcher.hpp:221 in re_is_set_member AddressSanitizer: heap-buffer-overflow perl_matcher.hpp:166 in re_is_set_member AddressSanitizer: heap-buffer-overflow interceptors.inc:278 in strlen AddressSanitizer: stack-overflow basic_regex_creator.hpp:1054 in create_startmap AddressSanitizer: SEGV on unknown address 0x0000000016e0 MemorySanitizer: use-of-uninitialized-value perl_matcher.hpp:166 in re_is_set_member basic_regex_parser.hpp:904: runtime error: shift exponent 325804978 is too large for 32-bit type 'unsigned int' basic_regex_parser.hpp:2599: runtime error: load of value 56794092, which is not a valid value for type 'syntax_element_type' a.out: perl_matcher_common.hpp:606: Assertion `r.first != r.second' failed Direct leak of 4096 byte(s) in 1 object(s) allocated in get_mem_block regex.cpp:204 ALARM: working on the last Unit for 17 seconds Will find more when these are fixed! Results
- 21. Finding logical bugs Not only security/stability - But we don't know the right result! - Use your imagination!
- 22. Finding logical bugs ● sanity checks on results ○ uncompressesed image decoder: 100 byte input -> 100 MB output? ○ function returns both error and object, or no error and no object ○ know that some substring must present in output, but it is not ○ encrypt, check that decryption with wrong key fails ● sometimes we do know the right result ○ any sorting: check that each element is present, check that it's not descending ○ building a trie: check size, all elements are present ● asserts ○ assert(a == b)
- 23. Finding logical bugs Round-trip: ● encode-decode ● serialize-deserialize ● compress-decompress ● encrypt-decrypt ● assemble-disassemble Checks: ● decode-encode: check that encode don't fail ● decode-encode-decode: check that second decode don't fail ● decode-encode-decode: check that decode results are equal ● encode-decode-encode: check that encode results are equal Very powerful technique.
- 24. Finding logical bugs Comparing two (or more) implementations gives phenomenal results: ● check that output is equal ● or at least check that ok/fail result is the same ○ e.g. gcc and clang both accept or reject the code But I don't want to write the second impl! ● there can be several libraries implementing the same (libxmlFoo vs libxmlBar) ● implementation in a different language (re2 vs Go's regexp) ● compare "fast but complex" with "slow but dumb" (sometimes easy to write) ● compare different functions (marshalBinary vs marshalText)
- 25. Quick Quiz: how to fuzz clang-format? clang-format: shuffles whitespaces in a source file. Let's imaging destiny of mankind depends on correctness of clang-format! How would you fuzz test it?
- 26. Quick Quiz: how to fuzz clang-format? ● run with asan/msan/ubsan ● format twice, compare results (e.g. relies on unordered_map order) ● format, then format result (must be idempotent) ● strip all whitespaces, compare before/after ● check violations of max line length ● compile before/after (formatting breaks/unbreaks code)
- 27. Regression testing Normally you run fuzzer for a long time. But any guided fuzzer accumulates corpus of inputs with max coverage. And that's perfect for regression testing! Just run it once on every change!
- 28. Fuzzing@Google Why? - faster and faster development - more and more code - correctness is important - stability is still important - security is super important - want to move fast, but keep development costs under control Traditional testing is not enough anymore!
- 29. Fuzzing@Google How? - Developers can write "fuzz tests" - picked up by automatic large-scale fuzzing system - but also work as regression unit tests - OSS-Fuzz: continuous fuzzing for OSS - 50+ projects, 190 fuzzers - libFuzzer, radamsa, AFL (coming) - 5000 cores - ClusterFuzz: automated fuzzing for Chromium - 350 fuzzers - libFuzzer, radamsa, AFL, custom fuzzers - 12000 cores - Automatically files bugs and verifies fixes - syzkaller: continuous fuzzing of Linux kernel - several upstream branches + android/chromeos - 100+ VMs + physical devices
- 30. Fuzzing@Google ~15'000bugs GLIBC MUSL LIBC pugixml PCRE ICU Freetype ffmpeg Harfbuzz SQLite Python OpenSSL BoringSSL libxml2 BPF Capstone file Radare2 gRPC WOFF2 LLVM Tensorflow libav FreeType2 Foxit libtiff Go Linux libexif libFLAC Little CMS Adobe Reader Adobe Flash Player Adobe DNG SDK ESET NOD32 ClamAV BitDefender poppler ghostscript dcraw qcms libwebp libwebm libvpx gipfeli libots Snapseed Dart VM IJG libjpeg-turbo libpng mozjpeg PHP Firefox Internet Explorer Apple Safari LibreOffice GnuTLS GnuPG OpenSSH bash tcpdump JavaScriptCore pdfium libmatroska libarchive wireshark ImageMagick lcms libbpg lame libsndfile less lesspipe strings dpkg rcs systemd-resolved libyaml Info-Zip unzip libtasn pfctl mandoc IDA Pro clamav nasm ctags mutt procmail fontconfig pdksh wavpack redis cmsgpack taglib privoxy perl libxmpradare2 SleuthKit X.Org exifprobe jhead Xerces-C metacam exiv btrfs Knot DNS curl wpa_supplicant dnsmasq imlib2 libraw libwmf uudecode MuPDF libbson libsass boost
- 31. Sales pitch ● Fuzzing is complimentary to any other testing technique ● Fuzzing is mandatory for anything security-related ● Fuzzing finds LOTS of bugs ● Fuzzing is easy to use Call to action: ● choose 1 library that uses complex inputs (important or you suspect for bugs) ● write a fuzzer ● run locally with ASAN