Advances in File Carving
Download as PPTX, PDF2 likes6,683 views
This document discusses advances in file carving techniques for data recovery from disks and unallocated space. It describes various carving methods like header/footer carving, statistical carving, and fragment recovery carving. It also outlines limitations of current file carving tools and proposes ideas for future tools that combine methods and support more file types and fragmented files.
Advances in File Carving
- 1. Advances in File CarvingRob Zirnstein, PresidentForensic Innovations, Inc.www.ForensicInnovations.com7/14/2011
- 2. Our Data is GONE!All of your servers have Crashed!Your customers’ Data is Lost!You backed up last week, but important business transactions have taken place since.70% of companies with devastating data loss go out of business.All it took was one employee writing a simple SQL database script after you fired them.
- 3. We Didn’t Find The Evidence!What do you do when you’ve searched through all of the evidence and came up empty?When you know a suspect is hiding something, where do you look first?TrueCrypt Volumes & Unallocated SpaceEven good people shred data when faced with an investigation.The tools are easy to find.www.TrueCrypt.org
- 4. How They Hide the EvidenceDeleting a fileSends the file to the Windows Recycle BinEmpty or bypass the Recycle BinUndelete tools depend on the deleted directory entryThat can be deleted or overwritten tooThen there’s no undeleting possibleStore files in a TrueCrypt VolumeUndetectable as a file (except for my tools)Looks like random data in unallocated space (except for my tool)
- 5. How To Get The Files BackFile CarvingDefinition: “General term for extracting data (files) out of undifferentiated blocks (raw data), like "carving" a sculpture out of soap stone.” http://www.forensicswiki.org/wiki/File_CarvingThe sectors containing the files are orphanedSome of them may get overwrittenThey are like many jigsaw puzzles thrown into a trash bag, if they were fragmented.If some sectors were stored consecutively, then it’s like puzzle pieces that weren’t pulled apart before getting trashed.
- 6. File Carving AssumptionsNo Files are Fragmented!?!All Files are stored in consecutive sectorsSector Size = 512 bytesMay be detected through disk structureCluster Size = 512 to 16,384 bytesMay be detected through disk structureFile Slack may be ignoredRAM slack is ignoredOr incorrectly bundled in with File SlackIsn’t it always zeroed out?
- 7. File Carving TechniquesBlock Based CarvingStatistical CarvingHeader/Footer CarvingHeader/Maximum File Size CarvingHeader/Embedded Length CarvingFile Structure Based CarvingSemantic CarvingCarving with ValidationFragment Recovery CarvingRepackaging CarvingSmartCarvingHash CarvingFuzzy Hash Carvinghttp://www.forensicswiki.org/wiki/File_Carving
- 8. Block Based CarvingAnalyze each sector on a block-by-block basis to determine if they belong together in the same file.Assuming that each sector can only be part of a single file
- 9. Statistical CarvingUse statistics or content characteristics to identify each sector.Entropy measurementFilter out blocks that clearly aren’t part of a desired file type.
- 10. Header/Footer CarvingSearch for file header signature(s).Search for the matching file footer signatures.Capture the sectors in between.
- 11. Header/Maximum File Size CarvingSearch for file header signature(s).Consult a list of maximum file lengths for each header type.Capture the sectors in between.Many file types do not detect the additional unrelated data that may get appended to the recovered file.
- 12. Header/Embedded Length CarvingSearch for file header signature(s).Read the file length from one of the fields in the header.
- 13. File Structure Based CarvingOnce a sector’s file type is identifiedMatch to other sectors that contain similar data structures.Use knowledge of the file type’s data structures to search for structure parts expected to exist in later sectors.
- 14. Semantic CarvingIdentify the language used in a sector.Identify the language used in each of the following sectorsCollect the sectors that are written in the same language
- 15. Carving with ValidationUse a file interpreter or viewer to load each recovered file.If the interpreter encounters invalid data, assume that is the point where the carving method failed.Use on completed files.Use on each added sector.
- 16. Fragment Recovery CarvingFind two or more fragments that belong to the same file.Filter out the sectors between the fragments that don’t belong.
- 17. Repackaging CarvingUsed on partially recovered files.Rebuild the parts of the file that were not able to be recovered.The result should be a file that can be opened with it’s native application or a standard viewer.
- 18. SmartCarvingUse knowledge of the file system’s typical fragmentation effects.Preprocess the source sectors.Decompress, decrypt or translate the dataCollate the identified blocks.Sort by file typeReassemble the blocks in sequences that match their file type.
- 19. Hash CarvingCalculate a hash value for each sectorMD5, SHA-1Compare the hash value to a list of known sector hash valuesThis list can be of known Good and/or known Bad files.Filter out known Good files. (ex: Installed applications)Recover known Bad files. (ex: known illicit material)
- 20. Fuzzy Hash CarvingCalculate a fuzzy hash value for each sector.Compare the fuzzy hash values of sectors to determine which sectors are similar in content.Combine similar sectors into recovered files.Match raw data sectors together for object types that have no identifiable signatures or that extend beyond a single sector.Recover file types not previously encountered.
- 21. Tools Today (1)Adroit Photo Recovery/Forensicscombination of SmartCarving, header carving, structure based validation and validation of the entire file to determine if each new sector belongs; Repackaging Carving is also available; http://www.forensicswiki.org/wiki/File_Carving:SmartCarvingSupports JPEG, RAW camera images, PNG, BMP and GIF filesDataLifterheader-footer carving; Supports 25 file typesEncaseheader-footer carving; Supports ~250 file typesForemostfile structure based carving for avi, bmp, doc, gif, hmlt, jpg, mov, pdf, png, rar, wav and zip files.header-footer carving for art, asf, chm, cookie, cpp, dat, dbx, fws, idx, java, lnk, mail, mbx, mp3, mpg, ost, pgd, pgp, ppt, pst, ra, rdp, rpm, tif, txt, wma, wmv, wpc and xls files.Forensic Toolkit (FTK)internal techniques unknown; Supports abl, aol, asd, bmp, doc, dot, emf, gif, html, jpg, mpp, one, pdf, png, ppt, pub, puz, vsd, vss, vst, xla, xls and xlt files.http://www.forensicswiki.org/w/images/b/b9/Kloet_2007.pdf
- 22. Tools Today (2)HstEx / Netanalysisinternal techniques unknown; Supports browser history formatsNFI DefraserFragment recovery carving & carving with validation; Supports MPEG, 3GPP, Quicktime & AVI filesPhotoReccombination of file structure based carving and header-footer carving of 80 file formatsPyFlagappears to use a simple text search method, ignoring sector boundaries; Supports server log file formatsRecover My Filesinternal techniques unknown; Supports 200 file typesRevitSmartCarving; Supported file types list not availablehttp://www.forensicswiki.org/w/images/b/b9/Kloet_2007.pdf
- 23. Tools Today (3)Scalpelcombination of header-footer and header-maximum file size carving; Supports art, avi, dat, dbx, doc, fws, gif, htm, idx, java, jpg, mail, max, mbx, mov, mpg, ost, pdf, pgd, pgp, pins, png, pst, ra, rpm, tif, txt, wav, wpc and zip files.X-Waysheader-footer carving; unknown support listhttp://www.forensicswiki.org/wiki/Tools:Data_Recovery#Carving
- 24. Tool ProblemsFew tools handle file fragmentationThe tools that handle fragmentation support very few file typesMost tools can not detect false positivesMost tools hard code file type supportOnly 1 tool claims to rebuild partial filesIt only supports 5 file types (image files)Performance is a problemmost tools utilize inefficient databases and scripting languages
- 25. Future ToolsCarver 2.0Open Source, in the early specification stagesFile HarvesterCombination of multiple methods:Block Based CarvingStatistical CarvingHeader/Footer CarvingHeader/Embedded Length CarvingFile Structure Based CarvingFragment Recovery CarvingRepackaging Carving (Phase 3)SmartCarvingFuzzy Hash Carving(secret sauce)
Editor's Notes
- #24: Simple Carver Suite - www.simplecarver.comCarvFs - www.forensicswiki.org/wiki/CarvFsmidi-carver - http://greg-kennedy.com/nwserver/?p=10PhotoRescue - www.datarescue.com/photorescueMagic Rescue - http://jbj.rapanden.dk/magicrescue/X-Ways