Automated static deobfuscation in the context of Reverse Engineering
0 likes417 views
This document discusses automated static deobfuscation in the context of reverse engineering. It describes challenges with deobfuscation like side effects, code blowup, and address mapping. It summarizes the authors' approach of translating obfuscated assembly to REIL and running algorithms on it. However, it notes limitations like inability to debug and side effects. It considers alternatives like effect comments, formula extraction, and a full compiler to better address problems but notes the significant work required.
Automated static deobfuscation in the context of Reverse Engineering
- 1. Automated static deobfuscation in the context of Reverse Engineering Sebastian Porst (sebastian.porst@zynamics.com) Christian Ketterer (cketti@gmail.com)
- 2. Sebastian Christian • zynamics GmbH • Student • Lead Developer • University of Karlsruhe – BinNavi • Deobfuscation – REIL/MonoREIL
- 3. This talk Obfuscated Code Readable Code (mysterious things happen here) 20% 40% 40%
- 4. Motivation • Combat common obfuscation techniques • Can it be done? • Will it produce useful results? • Can it be integrated into our technology stack?
- 5. Examples of Obfuscation Simple • Jump chains • Splitting calculations • Garbage code insertion • Predictable branches • Self-modifying code • Control-flow flattening • Opaque predicates • Code parallelization • Virtual Machines • ... Tricky
- 6. Our Deobfuscation Approach I. Copy ancient algorithms from compiler theory books II. Translate obfuscated assembly code to REIL III. Run algorithms on REIL code IV. Profit (?)
- 7. We‘re late in the game ... 199X 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 U of Auckland U of Wisc + zynamics F. Perriot TU Munich Mathur U of Ghent M. Mohammed Mathur Christodorescu (see end of this presentation for proper source references) Bruschi
- 8. ... but 199X 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Malware Research Defensive Reverse Engineering Offensive Reverse Engineering
- 9. REIL • Reverse Engineering Intermediate Language • Specifically designed for Reverse Engineering • Design Goal: As simple as possible, but not simpler • In use since 2007
- 10. Uses of REIL Register Tracking: Helps Reverse Engineers follow data flow through code (Never officially presented) Index Underflow Detection: Automatically find negative array accesses (CanSecWest 2009, Vancouver) Automated Deobfuscation: Make obfuscated code more readable (SOURCE Barcelona 2009, Barcelona) ROP Gadget Generator: Automatically generates return-oriented shellcode (Work in progress; scheduled for Q1/2010)
- 11. The REIL Instruction Set Arithmetical Bitwise Data Transfer Logical Other ADD AND STR BISZ NOP SUB OR LDM JCC UNDEF MUL XOR STM UNKN DIV MOD BSH
- 13. Why REIL? • Simplifies input code • Makes effects obvious • Makes algorithms platform-independent
- 14. MonoREIL • Monotone Framework for REIL • Based on Abstract Interpretation • Used to write static code analysis algorithms http://www.flickr.com/photos/wedrrc/3586908193/
- 15. Why MonoREIL? • In General: Makes complicated algorithms simple (trade brain effort for runtime) • Deobfuscator: Wrong choice really, but we wanted more real-life test cases for MonoREIL
- 16. Building the Deobfuscator • Java • BinNavi Plugin • REIL + MonoREIL http://www.flickr.com/photos/mattimattila/3602654187/
- 17. Block Merging • Long chains of basic blocks ending with unconditional jumps • Confusing to follow in text-based disassemblers • Advantage of higher abstraction level in BinNavi – Block merging is purely cosmetic
- 18. Block Merging Before After
- 19. Constant Propagation and Folding • Two different concepts • One algorithm in our implementation • Partial evaluation of the input code
- 20. Constant Propagation and Folding Before After
- 21. Dead Branch Elimination • Removes branches that are never executed – Turns conditional jumps into unconditional jumps – Removes code from unreachable branch • Requires constant propagation/folding
- 22. Dead Branch Elimination Before After
- 23. Dead Code Elimination • Removes code that computes unused values • Gets rid of inserted garbage code • Cleans up after constant propagation/folding
- 24. Dead Code Elimination Before After
- 25. Dead Store Elimination • Comparable to dead code elimination • Removes useless memory write accesses • Limited to stack access in our implementation • Only platform-specific part of our optimizer
- 26. Dead Store Elimination Before After
- 27. Suddenly it dawned us: Deobfuscation for RE brings new problems which do not exist in other areas
- 28. Let‘s get some help
- 29. Problem: Side effects push 10 mov eax, 10 pop eax Removed code was used • in a CRC32 integrity check • as key of a decryption routine • as part of an anti-debug check • ...
- 30. Problem: Code Blowup mov eax, 10 mov eax, 20 add eax, 10 clc ... Good luck setting • AF • CF • OF • PF • ZF
- 31. Problem: Moving addresses 0000: jmp ecx 0000: jmp ecx 0002: push 10 0002: mov eax, 10 0003: pop eax ecx is 0003 but we just missed the static analysis pop instruction can not know this
- 32. Problem: Inability to debug mov eax, 10 Executable Input File Deobfuscated list of Instructions but no executable file
- 33. The only way to solve all* problems: A full-blown native code compiler with an integrated optimizer Too much work, maybe we can approximate ... * except for the side-effects issue
- 34. Only generate optimized REIL code Before After
- 35. Only generate optimized REIL code • Produces excellent input for • Side effects problem remains other analysis algorithms • Pretty much unreadable for • Code blow-up solved human reverse engineers • Keeps address/instruction mapping • Code can not be debugged natively but interpreted
- 36. Effect comments Before After
- 37. Effect comments • Results can easily be used by • Side effects problem remains human reverse engineers • Address mapping problem • Code blow-up solved • Code can not be debugged • Comments have semantic meaning
- 38. Extract formulas from code Before After
- 39. Extract formulas from code • Results can easily be used by • Not really deobfuscation (but human reverse engineers produces similar result?) • No code generation necessary, only extraction of semantic information • Solves all problems because original program remains unchanged
- 40. Implement a small pseudo-compiler Before After
- 41. Implement a small pseudo-compiler • This is what we did • Side effects problem remains • Closest thing to the real deal • Address mapping problem • Code blow-up is solved remains • Partially • Why not go for a complete • Natively debug the output compiler? • not in our case • pseudo x86 instructions
- 42. Economic value in creating a complete optimizing compiler for RE? Not for us • Small company • Limited market • Wrong approach?
- 43. Alternative Approaches • Deobfuscator built into disassembler • REIL-based formula extraction • Hex-Rays Decompiler • Code optimization and generation based on LLVM • Emulation / Dynamic deobfuscation
- 44. Conclusion • The concept of static deobfuscation is sound – Except for things like side-effects, SMC, ... • A lot of work • Expression reconstruction might be much easier and still produce comparable results
- 45. Related work • A taxonomy of obfuscating transformations • Defeating polymorphism through code optimization • Code Normalization for Self-Mutating Malware • Software transformations to improve malware detection • Zeroing in on Metamorphic Computer Viruses • ...