Libor Mořkovský - Recognizing Malware
0 likes417 views
This document discusses techniques for recognizing malware files. It describes how each file is represented as a feature vector that captures static and dynamic attributes. A distance function is used to measure the similarity between vectors and identify nearest neighbors for classification. Files are classified using an instance-based classifier and optimized with techniques like a VP-tree and distance-bounded search. Classifications are deployed in a system that collects file fingerprints from users and shares threats and updates between components. A rule generator also aims to detect malware variants by learning rules based on conditions in files.
Libor Mořkovský - Recognizing Malware
- 2. Computer virus bacterial cell based on work by Anderson Brito
- 3. Computer virus executable file entry point
- 4. Computer virus Inserting code into files is never “good”. executable file entry point
- 5. image courtesy of Looking Glass Studios Malware How do you recognize a thief?
- 6. image courtesy of Looking Glass Studios Twentieth Century Fox Malware How do you recognize a thief?
- 7. Malware How do you recognize a thief? image courtesy of Looking Glass Studios Twentieth Century Fox Paramount Pictures
- 8. Malware completely different behaviors are considered “bad” we need a judge to decide who crossed the line • •
- 9. Malware | Many faces unlike real thieves, malware can be duplicated not only duplicated, but also modified all this is done by machines too much work to judge each one manually • • • •
- 11. Finding similar files need a file representation need a distance function • •
- 12. Finding similar files | File vector each executable file is represented by a feature vector the PE format is complex, so we keep exactly one version of the extractor code (C++) the vector comprises static and dynamic features, the exact content is proprietary • • • Database record • One record = constant vector of over 100 attributes • the “file fingerprint” • Each attribute has a data type and semantic Attribute Data Type Semantic sha256 32 byte array CHECKSUM pe_sect_cnt uint16_t VALUE pe_sect_rawoff_entry uint32_t OFFSET • The complete contents of the vector are kept secret • static and dynamic features of PE executables
- 13. Finding similar files | Distance sum of partial distances each distance operator assigned manually weights assigned manually to equalize contribution • • • Nearest neighbor query • Compound distance function • Data type and semantic determine partial dist. func. Data Type Semantic Partial distance function 32 byte array CHECKSUM RETURN_ZERO uint16_t VALUE EQUAL_RET32 uint32_t OFFSET LOG • Each partial distance function = one kernel function • Over 100 kernels for every NN query • Intermediate results kept in the “Scratchpad”
- 14. Finding similar files | Data ~60 M data points sparse and well separated (in many cases) • •
- 15. Finding similar files | Implementation we started with GPUs their high memory throughput allows “naive” implementation and rapid prototyping column-oriented database • • •
- 16. Classification | Requirements find easily what is responsible for a mistake – transparency fix the problem quickly – tractability • •
- 17. Classification | Algorithm Instance based classifier.
- 18. Classification | Optimizations scaling and HW problems with GPUs we invested in algorithmic optimizations: VP-tree, distance bounded search hand optimized distance function (assembly) CPU version is ~100x faster • • • •
- 19. Classification | Deployment → FileSHAandu ser id → ←Fileprevale nce ← ← Fileclass ification ← → Filefinger print → ← Generic detections ← ↑ File classifications and Evo-gen detections → Threats → Set updates ↓ Medusa Scavenger Avast users FileRep
- 20. Classification | Deployment → FileSHAandu ser id → ←Fileprevale nce ← ← Fileclass ification ← → Filefinger print → ← Generic detections ← ↑ File classifications and Evo-gen detections → Threats → Set updates ↓ Medusa Scavenger Avast users FileRep
- 21. Classification | Deployment → FileSHAandu ser id → ←Fileprevale nce ← ← Fileclass ification ← → Filefinger print → ← Generic detections ← ↑ File classifications and Evo-gen detections → Threats → Set updates ↓ Medusa Scavenger Avast users FileRep
- 22. Rule generator detect more variants in the wild (our) rule is a conjunction of several conditions known as Win32:Evo-Gen completely different optimization problem than classification - still uses the GPU • • • •
- 24. Q&A