Testing web application firewalls (waf) accuracy
Download as PPTX, PDF1 like2,472 views
The document discusses the importance of accurate Web Application Firewall (WAF) testing, emphasizing metrics such as precision, recall, and accuracy in assessing WAF effectiveness. It introduces the Akamai WAF Testing (AWT) framework, which provides tools and capabilities for testing WAFs by sending both valid traffic and real attacks. The document also outlines requirements for the testing framework and the methodology for collecting test cases.
Testing web application firewalls (waf) accuracy
- 1. WAF Accuracy Testing Done Properly Introducing AWT framework Ory Segal, Director of Threat Research
- 2. ©2015 AKAMAI | FASTER FORWARDTM WAF Accuracy Lingo • Imagine a WAF that protects against 100% of all possible attack vectors …by blocking 100% of all HTTP requests • Accurate WAF testing requires you to measure: • How many real attacks got blocked (TP) • How much valid requests were allowed through (TN) • How much valid traffic was inappropriately blocked (FP) • How many attacks were allowed through (FN) • Lets talk about Precision, Recall, Accuracy, MCC…
- 3. ©2015 AKAMAI | FASTER FORWARDTM Precision, Recall, Accuracy, MCC % of blocked requests that were actual attacks % of attacks that were actually blocked % of decisions that were good decisions * MCC: http://en.wikipedia.org/wiki/Matthews_correlation_coefficient Correlation between WAF decisions and actual nature of requests Precision = tp tp+ fp Recall = tp tp+ fn Accuracy = tp+tn tp+tn+ fp+ fn MCC = tp×tn (tp+ fp)(tp+ fn)(tn+ fp)(tn+ fn)
- 4. ©2015 AKAMAI | FASTER FORWARDTM Lets Look at Some Examples A WAF’s accuracy needs to be measured both in its ability to block attacks, as well as it’s ability to allow good traffic through… WAF Type Requests Valid Attacks Blocked TP TN FP FN P R A MCC Real 1000 990 10 11 8 987 3 2 0.73 0.8 0.995 0.76 Off 1000 990 10 0 0 990 0 10 N/A 0 0.99 0 Always Block 1000 990 10 1000 10 0 990 0 0.01 1 0.01 0 Noisy 1000 990 10 31 8 967 23 2 0.26 0.8 0.975 0.45 Conservative 1000 990 10 2 2 990 0 8 1.00 0.2 0.992 0.45
- 5. ©2015 AKAMAI | FASTER FORWARDTM WAF Testing Framework Requirements • A tool that will send both valid traffic and real attacks • Easy addition of test cases (both valid & attacks) • Accuracy statistics gathering – FP, FN, TP, TN, P, R, A, MCC • Rich info about each test that was sent – full request, response, expected behavior, request nature • Reporting capabilities
- 6. ©2015 AKAMAI | FASTER FORWARDTM Introducing: Akamai WAF Testing Framework
- 7. ©2015 AKAMAI | FASTER FORWARDTM Akamai WAF Testing (AWT) Framework • Written in Python • Test cases are represented as textual files (.awt) • Options to create or add new test cases: • Write text files • Use a “Burp Extender” to record web interaction (meaningful requests only) • Transform Wireshark .pcap files (only ports HTTP traffic) • Multithreaded – can be very fast, or very “considerate” • Configurable and can work with any WAF • Intuitive XML & HTML reports • Easy debugging of FP/FN
- 8. ©2015 AKAMAI | FASTER FORWARDTM AWT Built-In Test Cases In order to accurately assess WAF, we collected test cases from the following sources: Retrieved valid traffic from Akamai’s Cloud Security Intelligence big data platform Recorded manual interaction with top “problematic” web sites Ported known “false positive” test cases from other tools Commercial web scanners Popular SQLi tools Exploits from the internet (fuzzers, exploit-db, … Traffic database is divided to 95% / 5% Automatic crawling of Alexa Top 100 internet sites Malicious traffic from Akamai’s Cloud Security Intelligence big data platform
- 9. ©2015 AKAMAI | FASTER FORWARDTM AWT Reports - Example
Editor's Notes
- #5: High recall is important if you are looking for a secure product High Precision is important for those looking for a system that doesn’t block valid users by mistake