Fault Models and Fuzzing

2. About...Shmuel Gershon

3. Testing Engineer

4. http://testing.gershon.info

5. Creator of Rapid Reporter

6. Twitter: @sgershon, Skype: sgershon, shmuel@gershon.infoDisclaimer:Names and Brands referenced herein may be claimed as property of third parties

7. Views expressed in this presentation are solely my own, and do not in any manner represent the views of my employer

9. Fault Models and FuzzTalk ObjectivesUnderstand the principles of the techniquesMeet tools available for both techniquesLearn to apply the methods (in different app types)Perceive drawbacks and difficultiesLay down a foundation for further researchHave at least one new idea for your tests :)Testing is questioning a product in order to evaluate its value to a person that mattersAdapted from Jerry Weinberg + James Bach + Michael BoltonCopyright © CC:BY-NC-SA 2007-11

13. Fault Models Overview – DefinitionDefinitions:Introducing faults in order to test (error handling) code paths, that might otherwise rarely be followedAssess the robustness of software by checking it's reaction to adverse eventsPurposeValidation of: Robustness, Dependability (Availability) and SecurityAlso known as: Recovery Code test, Fault Injection, Negative test, Error Handling, Stress test...Copyright © CC:BY-NC-SA 2007-11

14. Fault Models Overview – ApproachesRandom Runtime Fault Injections:A tool control the type, time and location to ‘attack’. May modify the coverage of a set of tests (issues found may be harder to reproduce or debug).Initiated Runtime Fault Injection:Specific tests in which the faults are controlled at specific point, aiming a clear error handling flow.Bugs found in this approach may be easier to reproduce, accept and fix.(This is the approach we'll focus on).Copyright © CC:BY-NC-SA 2007-11

15. Fault Models Overview – ReactionsReactions against Runtime Fault BugsIt will never happen in real lifeA user will not do thatIt should fail in such conditionsIt is an unsupported scenarioWe can’t fix thisIt is a third-party problemHow would you reply?Whose decision it ultimately is?Copyright © CC:BY-NC-SA 2007-11

16. Fault Injection – Example #1Memory Starvation:Refuses to allocate a memory upon request (Insufficient Memory).Pinball:Silently skips the action,no harm done.WordPad:Disappears with your most valuable workCopyright © CC:BY-NC-SA 2007-11

17. Fault Injection – Example #2Network Errors:Simulates an error responses for common network requests and resources.Browser Wars:One browser survives

19. Fault Injection – Example #3Low Resources / Slow or Clogged CPUComputer behaves as a busy or old computer(CPU is busy on other tasks / time scheduler starvation...)Playing to win:Can we escape from death by simply slowing down the rest?Copyright © CC:BY-NC-SA 2007-11

20. Fault Models – NotesFinding bugs can take a long time (but you record important data during the process)Abstract knowledge of the internal flows of the software is imperative for good resultsBugs are received with the comments seen earlierIt is difficult to assess the exposure of such bugsThese points can make it harder to adopt the practices in the company......so be sure to address themCopyright © CC:BY-NC-SA 2007-11

21. Fault Injection Tools ExamplesCanned Heat is free & easy, but buggyLimitations:Does not support .NET apps or ServicesBug: In some systems it will not load or needs to be reloaded between applications.Others:Verifier, AppVerifierSlowProc, HeavyLoadHolodeck - not free.PIN and (soon) the random malloc blockerIn-House or Brute-Force toolsCopyright © CC:BY-NC-SA 2007-11

23. Fuzz Testing – DefinitionDefinitions:Providing random data which is free of preconceptions to the inputs of a program, in order to reach unexpected statesFuzz explores the points that programmers and testers leave out due to assumption(Data that does not necessarily map to harmful inputs, or to valid inputs)PurposeBuilt-in code assertions, Coding and state assumptionsValidation of: Robustness, Dependability (Availability) and Security (many times it finds buffer overflows)Copyright © CC:BY-NC-SA 2007-11

24. Be only as smart as you have to Fuzz Testing – ApproachesSmart Fuzzers:Organize the input data so it would pass initial filters on the software (or the environments)Dumb Fuzzers:Data is close to random and chaos, in order to avoid assumption pitfallsTest approach:Simple, automated, no assumptions or objectiveCopyright © CC:BY-NC-SA 2007-11

25. Fuzz Testing – ReactionsContrary Reactions are similar, but usually in less extentIt will never happen in real lifeA user will not do thatIt should fail in such conditionsWe can not fix itWhen fuzz finally finds a bug it can be very severe, (crashes? overflows? data loss?)Can take little active time from testers, adoption mostly requires initial investment.Copyright © CC:BY-NC-SA 2007-11

27. Fuzz Testing – Example #1GUI Fuzzer:Sends random keyboard and mouse events all over the application.Solitaire

29. Fuzz Testing – Example #2File Fuzzer:Manipulates a file in order to create a set of randomly modified files.File Fuzzerdemo

31. Fuzz Testing – Examples #3CD-Rom driver:Interface is easy, the Peach Frameworkmakes it automaticWeb HTTP:

32. POST Fuzzing

34. Comments on Fuzz TestingFinding bugs can take a long timeKnowledge of the internal flows of the software is no necessary, and it can even hurtDoes not necessarily find software weaknesses in the shortest amount of timeNot every issue found is exploitableDeveloping a smart framework is important in order to achieve efficiency.Copyright © CC:BY-NC-SA 2007-11

35. Fuzz Testing Tools ExamplesFree tools:GUI and Command line:Fuzzer by the University of WisconsinFile manipulation:FileFuzzFuzz Framework:Peach FuzzerList of fuzzers:http://www.infosecinstitute.com/blog/2005/12/fuzzers-ultimate-list.htmlhttp://www.computerdefense.org/2007/01/15/fuzzing-tools/Commercial tools:Network Protocols:Codenomicon’sDefensicsIxiaCopyright © CC:BY-NC-SA 2007-11

36. Fault Models and Fuzz SummaryWe can reach hard-to-reach-in-lab scenarios with these techniques

37. These tests are a good way to prevent(some) surprises

38. Impact of such bugs can be very high!

39. There are tools available, many of them free

40. You can build your own

42. Fault Models and Fuzz Techniques - Learn MoreWikipedia:Fuzz: http://en.wikipedia.org/wiki/Fuzz_testingFault Injection: http://en.wikipedia.org/wiki/Fault_injectionUniversity of Wisconsin-Madison Fuzz Articles and Software:http://www.cs.wisc.edu/~bart/fuzz/Look out! It’s the fuzz:http://iac.dtic.mil/iatac/download/Vol10_No1.pdfFault Injection:“How to Break Software” book by James WhitakerA study on fuzzing effectiveness:http://www.docstoc.com/docs/53958850/Fuzz-By-NumberFuzzing Examples at the Open Wen App Security Project:http://www.owasp.org/index.php/FuzzingAttack your programs before someone else does:http://www.whitestar.linuxbox.org/pipermail/fuzzing/2006-November/000168.htmlCopyright © CC:BY-NC-SA 2007-11

Fault Models and Fuzzing

More Related Content

What's hot (20)

Viewers also liked (12)

Similar to Fault Models and Fuzzing (20)

Recently uploaded (20)

Fault Models and Fuzzing