Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Black-box Web Application Vulnerability Scanners
Download as PPTX, PDF1 like759 views
The document presents a comparative analysis of six open-source web application vulnerability scanners by evaluating their performance in both point-and-shoot (pas) and trained configurations. It discusses the pervasive vulnerabilities in web applications, finds that all scanners missed significant vulnerabilities, and highlights their varying strengths and weaknesses in crawling and detecting specific vulnerabilities. The analysis concludes that while open-source scanners can perform comparably to commercial ones, they still face challenges in effectively securing web applications.
Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Black-box Web Application Vulnerability Scanners
- 1. Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Black-box Web Application Vulnerability Scanners @rana__khalil Rana Khalil, University of Ottawa
- 2. Who am I? • Student at the University of Ottawa • B.S. in Mathematics and Computer Science (2016) • M.S. in Computer Science (2018) • Supervisor: Dr. Carlisle Adams • OSCP Certification (current) • Previous work experience include: software development, testing, ransomware research, teaching and penetration testing 2
- 3. Agenda 1. Introduction 2. Methodology 3. Results 4. Conclusion 3
- 4. Introduction
- 5. Web Applications • We use web applications for everything: • Over 3.9 billion users world wide • Over 1.8 billion websites online 5 Banking Education Shopping Communication • How much personal data do you have online? • Name, SIN, addresses, phone numbers, emails • Financial information • Heath information
- 6. Web Security • State of web security today • Trustwave’s 2018 Global Security Report: • 100% of web applications displayed at least one vulnerability • Median number of 11 vulnerabilities per application 6
- 8. How to Secure a Web Application? • A combination of techniques are used to secure web applications: 8 • Static code analysis • Web application firewalls • Secure coding practices • Web application vulnerability scanners
- 9. How to Secure a Web Application? • A combination of techniques are used to secure web applications. 9 • Static code analysis • Web application firewalls • Secure coding practices • Web application vulnerability scanners
- 10. WAVS Web Application Vulnerability Scanners have three modules: 10 Crawler Attacker Analysis *XSS found* *SQLi found* *LFI found* *RFI found*
- 11. WAVS Web application vulnerability scanners are largely used in two ways: 1. Point-and-Shoot (PaS) / Default • Scanner is given root URL of the application • Default configuration remains unchanged • Minimal human interference 11
- 12. WAVS Web application vulnerability scanners are used in two ways: 2. Trained / Configured • Change configuration (ex. crawl depth) • Manually visit every page of the application while scanner is in proxy mode. 12 Browser Scanner Proxy Web Application
- 13. Previous Work 13 • Suto’s case studies: • 2007 paper evaluated scanners in PaS mode • 2010 paper evaluated scanners in PaS and Trained modes • Benchmark applications: • Web Input Vector Extractor Teaser (WIVET) created in 2009 by Tatli et al. • Web Application Vulnerability Scanner Evaluation Project (WAVSEP) created in 2010 by Chen • Doupé et al.’s 2010 work on evaluating WAVS on the WackoPicko application • Several other more recent studies evaluate scanners in PaS mode only
- 14. Methodology
- 15. Research Goal • Goal: Performing a comprehensive comparative analysis of the performance of six chosen scanners in two modes: • PaS / Default • Trained / Configured 15 Tool Selection Benchmark Selection Environment Setup Feature & Metric Selection Result Analysis
- 16. Tool Selection • Chen’s evaluation • Consultation with professional ethical hackers 16 Name Version License Price Last Update* Arachni 1.5.1-0.5.12 Arachni Public Source v1.0 N/A 2017-03-29 Burp Pro 1.7.35 Commercial $349/year 2018-08-29 Skipfish 2.10b Apache v2.0 N/A 2012-12-04 Vega 1.0 MIT N/A 2016-06-29 Wapiti 3.0.1 GNU GPL v2 N/A 2018-05-11 ZAP 2.7.0 Apache v2.0 N/A 2017-11-28 *Checked on August 2018
- 17. Benchmark Selection • Benchmark applications: • WIVET – crawling challenges • WAVSEP – vulnerability classes • Intentionally vulnerable realistic web application • Type of vulnerabilities included in the application • Architecture of the application and the web technologies used • Ability of the application to withstand aggressive automated scans • OWASP Vulnerable Web Applications Directory (VWAD) project • WackoPicko 17
- 18. Benchmark Selection - WIVET • Contains 56 test cases that utilize both Web 1.0 and Web 2.0 technologies • Test cases include: • Standard anchor links • Links created dynamically using JavaScript • Multi-page forms • Links in comments • Links embedded in Flash objects • Links within AJAX requests 18
- 19. Benchmark Selection - WAVSEP • Consists of a total of 1220 true positive (TP) test cases and 40 false positive (FP) test cases 19 Vulnerability Category # of TP test cases # of FP test cases SQL Injection 138 10 Reflected XSS 89 7 Path Traversal / LFI 816 8 RFI 108 6 Unvalidated Redirect 60 9 DOM XSS 4 0 Passive 5 0
- 20. Benchmark Selection - WackoPicko 20 • Open-source intentionally vulnerable realistic web application • Photo sharing and purchasing site • Contains 16 vulnerabilities covering several of the OWASP Top 10 • Contains crawling challenges: • HTML parsing • Multi-step process • Infinite website • Authentication • Client-side code
- 21. Environment Setup 1/2 21 Tools Applications * * VM restored to initial state before every test run
- 22. Environment Setup 2/2 22 • Each scanner was run in two modes: • PaS / Default - default configuration setting • Trained / Configured 1. Maximize crawling coverage – changing configuration 2. Maximize crawling coverage – use of proxy 3. Maximize attack strength • WackoPicko test scans were further divided into two subcategories: • INITIAL – without authentication / publicly accessible • CONFIG - valid username/password combination • In total, each scanner was run eight times
- 23. Feature and Metric Selection • Crawling coverage • % of passed test cases on the WIVET application • Crawling challenges in the WackoPicko application • Vulnerability detection accuracy • TP, FN and FP on the WAVSEP and WackoPicko applications • Speed • Scan time on the WAVSEP and WackoPicko appliations • Reporting • Vulnerability detected • Vulnerability location • Exploit performed • Usability • Efficiency • Product documentation • Community support 23 Crawling Coverage Detection Accuracy Speed WIVET WackoPicko WAVSEP Features Applications
- 24. Feature and Metric Selection • Crawling coverage • % of passed test cases on the WIVET application • Crawling challenges in the WackoPicko application • Vulnerability detection accuracy • TP, FN and FP on the WAVSEP and WackoPicko applications • Speed • Scan time on the WAVSEP and WackoPicko applications • Reporting • Vulnerability detected • Vulnerability location • Exploit performed • Usability • Efficiency • Product documentation • Community support 24 Crawling Coverage Detection Accuracy Speed WIVET WackoPicko WAVSEP Features Applications
- 25. Results
- 26. Vulnerability Detection Accuracy – FNs 1/4 Vulnerabilities in WackoPicko that were not detected by any scanners: 1. Weak authentication credentials • admin/admin • Reasons: • Scanners did not attempt to guess username/password • Scanners did attempt to guess username/password but failed 26
- 27. Vulnerability Detection Accuracy – FNs 2/4 Vulnerabilities in WackoPicko that were not detected by any scanners: 2. Parameter Manipulation • Sample user: WackoPicko/users/sample.php?userid=1 Real user: WackoPicko/users/sample.php?userid=2 • Reasons: • Most scanners did not attempt to manipulate the userid field • Arachni manipulated the userid field but failed to enter a valid number • Skipfish successfully manipulated the userid field but did not report it as a vulnerability 27 userid=2
- 28. Vulnerability Detection Accuracy – FNs 3/4 Vulnerabilities in WackoPicko that were not detected by any scanners: 3. Sored SQL Injection 4. Directory Traversal 5. Stored XSS Reasons: • Crawling challenges – discussed later • Lack of detection for these types of vulnerabilities 28
- 29. Vulnerability Detection Accuracy – FNs 4/4 Vulnerabilities in WackoPicko that were not detected by any scanners: 6. Forceful Browsing • Access to a link that contains a high quality version of a picture without authentication • /WackoPicko/pictures/high_quality.php?key=hig hquality&picid=11 7. Logic Flaw • Coupon management functionality Reasons: • Require understanding business logic of the application • Application specific vulnerabilities 29
- 30. Vulnerability Detection Accuracy – TPs 1/4 30 WackoPicko Overall Scan Detection Results Arachni Burp Skipfish Vega Wapiti ZAP PaS 37.5 37.5 31.25 18.75 25 37.5 Trained 37.5 50 31.25 25 25 43.75 0 10 20 30 40 50 60 70 80 90 100 %ofDetectedVulnerabilities Key Observations: • All scanners missed at least 50% of the vulnerabilities • In PaS mode Burp, ZAP and Arachni achieved the same score • Running the scanners in trained mode increased the overall detection • Vega – increase in attack vector • ZAP & Burp – Manually visiting the pages in proxy mode for Flash and dynamic JS technologies
- 31. 31 WackoPicko Detection Results. The simplest configuration that detected a vulnerability is listed. Name RXSS XSS Stored SQLi Reflected Command line injection File Inclusion File Exposure RXSS behind JS RXSS behind Flash Arachni INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL Burp Pro INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL CONFIG Skipfish INITIAL INITIAL INITIAL INITIAL INITIAL Vega INITIAL INITIAL INITIAL INITIAL Wapiti INITIAL INITIAL INITIAL INITIAL ZAP INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL CONFIG PaS Trained • Reminder: INITIAL means w/o authentication credentials and CONFIG means w/ authentication • Running the scanners in trained mode increased the overall detection Vulnerability Detection Accuracy – TPs 2/4
- 32. Vulnerability Detection Accuracy – TPs 3/4 32 WAVSEP Overall TP Detection Key Observations: • WAVSEP results were better than WackoPicko. • Vulnerability categories in the application • Integrating WAVSEP in the SDLC of the scanner • ZAP achieved highest score, followed by Vega and Skipfish Arachni Burp Skipfish Wapiti Vega ZAP PaS 60.2 27.9 4.0 25.4 71.3 60.7 Trained 60.2 42.5 62.6 24.4 71.3 79.3 0 10 20 30 40 50 60 70 80 90 100 %ofWAVSEPTestsDetected
- 33. Vulnerability Detection Accuracy – TPs 4/4 33 • Vulnerability category detection varied with scanner Scanner SQLi RXSS LFI RFI Unvalidated Redirect DOM XSS Passive Arachni 100% 73% 44% 97% 100% 100% 40% Burp 97% 83% 24% 60% 76% 0% 0% Skipfish 78% 80% 64% 33% 36% 0% 40% Wapiti 50% 60% 15% 44% 0% 100% 0% Vega 100% 85% 70% 100% 0% 0% 40% ZAP 86% 74% 74% 100% 100% 0% 40%
- 34. Crawling Challenges 1/6 Features that scanners found difficult to crawl in WackoPicko: 1. Uploading a file • All scanners were not able to upload a picture in PaS mode • Burp and ZAP were able to in Trained mode 34
- 35. Crawling Challenges 2/6 Features that scanners found difficult to crawl in WackoPicko: 2. Authentication • All scanners except for Wapiti successfully created accounts • None of the scanners used the created accounts to authenticate 35 Scanner # of Accounts Arachni 202 Burp 113 Skipfish 364 Vega 117 Wapiti 0 ZAP 111
- 36. Crawling Challenges 3/6 36 Features that scanners found difficult to crawl in WackoPicko: 3. Multi-step processes • All scanners were not able to complete the process in PaS mode • Burp and ZAP were able to in Trained mode
- 37. Crawling Challenges 4/6 Features that scanners found difficult to crawl in WackoPicko: 4. Infinite websites • All scanners recognized the infinite loop except Arachni 37 ….. /calendar.php?date=1541454543 /calendar.php?date=1541540943 /calendar.php?date=1541627343
- 38. Crawling Challenges 5/6 Features that scanners found difficult to crawl in WackoPicko: 5. Client-side code • Flash applications • Dynamic JavaScript • Ajax Requests 38 Arachni Burp Skipfish Wapiti Vega ZAP PaS 94 50 50 50 16 42 Trained 94 50 50 50 16 78 0 10 20 30 40 50 60 70 80 90 100 %ofWIVETTestsPassed WIVET Results
- 39. Crawling Challenges 6/6 Features that scanners found difficult to crawl in WackoPicko: 6. State - awareness • All the scanners exploited SQL injection vulnerability in login form, however didn’t discover any of the vulnerabilities that require authentication • Vulnerabilities that require authentication were only discovered in Trained mode • Credentials given • Logout link excluded 39 Scanner Web Application
- 40. Crawling Challenges 6/6 Features that scanners found difficult to crawl in WackoPicko: 6. State - awareness • All the scanners exploited SQL injection vulnerability in login form, however didn’t discover any of the vulnerabilities that require authentication • Vulnerabilities that require authentication were only discovered in Trained mode • Credentials given • Logout link excluded 40 Scanner Web Application
- 41. Conclusion
- 42. Conclusion • Scanners are far from being used as PaS tools only • Several classes of vulnerabilities were not detected • Scanners had difficulty crawling through common web architectures and web technologies • Different scanners have different strengths/weaknesses • Open-source scanner performance is comparable to commercial scanner performance and in several cases better 42
- 43. Last Words… To secure a web application you need to find and stop ALL attack vectors, whereas to break a web application you just need to exploit ONE attack vector. 43 Web application vulnerability scanners are trying to solve a VERY hard problem!