VB2015 Malware Classification meets crowd-sourcing

Editor's Notes

• #2: Thanks for coming. It’s late in the afternoon, so I will try to make this presentation interesting and worthwhile for your time. Let’s jump right in. In this session, I would like to talk about “malware classification and Crowdsorucing”.
• #3: what is malware classification We will talk abnout the logistics of crowd sourcing.  And, what is crousing coursinc g good for., And, the REAL focus of this talk is “what comes after” crowdsourcing. [it will starts in 20000 ft up in the air, then zoom down to at the ground level. To the bit level I don’t like people who just drops “machine learning” as “something smart”. So, I like to get to the nitty gritty detail. In a way, it is Trojan horse. I will come out easy. and set that expection, that this is a fluff talk. Still useful fluff talk. ]
• #5: I love circular definition.  Concept = actual process
• #8: Shady = Legal. Because Out user has freedom of choice. But, it is in a bad taste. (Apple ecosystem has the notion of “good taste”. Windows + Android ecosystem, is starting to have the notion of “good taste”) <CLICK> So, how does Malware Analyst actually make that decision?
• #9: And, we have this thing called “Virus Total” On the surface, it is just multivendor platform. But, it is a mean of communicating with the rest of the community. It is like asking this room, “have you seen this file? If so, I don’t want to duplicate the work” There are already more files to look at.
• #10: We start to put VT in the pipeline. To detect, and to prioritize. (multiple vendor detect, work on it first) Then, we make 3 vendor rules. <CLICK> Inside circle is what is becoming into the “uber-system” So, how do you make this “uber system”
• #11: Subliminal HPE teal box messaging. “The 4 Component” of Malware Classificaiton “system” Largely, there are 4 main component, in this picture. next slide will describe in details.
• #12: Malware Analysts,= malwre classifciaiton is not simple, as shown before. so we already need a human, to make the final call. a human, is always the final assurance. There are always edge cases, and human is the central. Data Sharing is the “threat intel sharing”. VirusTotal is one way. ThreatCentral is another way. There are many platforms. Automation IT infrasturucte. To process large amount of files, we need a huge server farms. Vanja’s Friday talk. Algorithm is missing piece. And this is where Data Science comes in. Also, cross-field. Algorithms are universal.
• #13: What is good about “crowd-sourcing” or “competition”, is that, You can tap into a large number of smart people, from the diverse field.
• #14: Netflix was the first “internet scale” competition. for the internet scale competition, it has to be “REALLYU SIMPLE” to explain. These are people, who are not familiar with the industry. So, it would be best, if it can be explain within 3 bullet points. .. It’s Simple, but not Easy. it took the “internet” 3 years to achieve this goal. And, the winning team came up with some crazy shit algorithm.
• #15: How you approach to solve this problem is that, something called “Collaborative Filtering”, The simplest visualization is that, every rating is a line. Then you construct this “2 row structure”. Once that is done, imagine you are pouring water to User1. the edge will acts as “conduit”, and which ever Movie that collects the most “water”, is the matching movie. It works surprisingly well!
• #16: 7 min mark
• #17: Even though this was the first competition from out industry, actually many company already does this. On Kaggle. Netflix Competition was dragged out too long. 3 yrs was too long. We don’t have that long attention span. And, also they found that, it is not the prize that draw the crowd. It is the ego, that drives. And the job offer. With shorter cycle and smaller winning, they went with a fast iteration. improved the competition format. It is like having 200 research team, all working different method. it is a tremendous “brain collecting” ZDI is one method of crowd-sourcing research
• #18: Now. It has large users base. It is its own world, like security people has their own world. Depending on the interestingness of the topic, or easiness of the competition. it draws on average 200 teams to 1000 teams. Most of the registered are non-plyaers, or just want to see. But, the top 10% are very good.
• #19: Month 3 leap frogging. Once you put 200 hrs on it, and other people do better, you get “angry” and do everything. The best thing about these competitions is “insights sharing” But, what is the best thing about these crowdsourcing is that. The insight from these diverse, short cycle competitions is “nurturing” the next competition. You learn some method works well for one data set. Then, you apply the similar method to a different set. .
• #20: Every competition is little different. It is similar to “malware analysis”. And you need to quickly understand the core of problem set. Security has been popular topic for a while. On average, it draws 2x the average. And this is good for the “security industry” One of the largest datasets. Most datasets are 5MB to 200MB. If not image files. Cant do it in MacBook Air. If this large, First pass would need to be Single Pass. For most of the crowd, PE files structure is foreign. (it tooks us a while to get familiar with it)
• #21: Data Sciencist like to view things as in “matrix” This is the standard table format. IF you give data in this format, they are happy. If the data is some other format, they cant do anything, until the data can be transformed into this format. N entries. M features, Answers. You are given 2 sets of data. Training set is the set with the answer Test set is the set, missing the answer. And, your objective is to …. <CLICK> The objective is to guess this value. Now, most people stuck at what to do with data, at this point. They will try to learn new frameworks, that comes out every month. Blindly pluck in the data. So, let me offer you an easy framework of thought.
• #22: I was looking into old works from Shannon, Turing, Weaver, Wiener. Whom I consider “foundind fathers” of computers. And, how they visulaied data, back then, is stll the best way to explain. I like “communication theory” or “information theory”, because it is technical, as well as, philosophical. The best way to communicate is “to minimize the loss of signal” between each steps.
• #23: Let’s rearange the “communication” in a slightly different way.
• #24: It’s like magic. Now every can see the angle. This is how you should think. You are sitting in the “answer” region of training set. And you want to pass message to the “answer” in the test set. So, the best strategy is to “reduce noise, while pushing as much signal to the next step” Find any link that is losing too much signal, and fix that.
• #25: First rule of the competition is, that “there is no rule”. It gets fierce. 1. There are always some form of data leak. That Path is data-leak path. And what we are doing is “data leak collecting” But, sometimes, too much data is leaked in a single variable. (sometimes, machine measured data had least significant digits acting as hash value” ) 2. LeaderBoards is on the test set. Cross-Validation is within Trainingset. If the training/test set split is not even, Trusting on the wrong board doesn’t get you. 3. If bad error metic is chosen, then one weird outliar mess the whole game. And it is about hunting out that one “error”. In this case, it is more about debugging. 4. Sometimes, If you understand well about how error metrics works, you can score high, even without seeing the data. 5. Most of algorithm are geared toward compact, matrix format. So, if it not typical “toy dataset”, it does not perform well. Narrow means there are only few features. When not enough information is given about an object, you use strategy to maximize on feature creationg, such as hotcoding/pairing Wide means there are lots of features. But don’t know which is important. Sparse means there are lots of 0 in the data. Dense means there are lots of information in a small space.
• #26: In my days of Malware Analyst, I used have this chart, similar to this. Black Box and WhiteBox Analysis mainly to extract existence of these functionalities. You extract all these “Functionality” features. And use “Logics” and “Commen sense” to determine if a files is bad or not. A skilled malware analyst can identify a malware, within about 10 min. and, in that 10 min, we look for “red flag”. There are probably about 30 to 50 things you look for. At one point, I try to make a system, classify malware using this method.
• #27: And, when a new files comes in, We use this logic to find what the unknown files is.
• #28: I used to make a classification tool, using the previous slide’s method. But, along the way, I tried something different. In order to maximize on “Computer’s ability”, you need to think little bit differently. The main strategy is “lots of tiny things” Human brain can’t hold 20+ features in the head. So, we simplify things into “Chunks”. Each functionality, such as network access, has “meaning.”. It means it has already been “chunked” Computer works before “chunking” Machines don’t have logic. Only statistics. In that world, the best way is “lots of meaningless bits”, but in very large numbers. Like filesizes. Who remembers that stuff… Like the number of occurance of ASCII 61. But, computers can do that, easily. What they excel is doing lots arithemrics, really really fast.
• #29: To make an AI, human is a good bench mark. But, it is not exactly. A good analogy is Birds and Airplanes. “Flying” is what is important. You are trying to make an “flying machine”. It does not have to be “feathered light-weight folding wings”.
• #30: This is the solution given by the first place. This is the way, how data scientist communicate to each other. With this single picture, everyone knows how to replicate it. -Feature Engineering: Golden features. These are the features, if missing, you cannot win the competition. or, if running out of time/computing power, just delving into these features, are better than adding other noisy features. -Opcode n-gram is important. -Segment/Section count is important. In the real practice, I don’t think this would be useful. Or it could be easily fooled. But for this competition, it worked great. -asm pixel intensity: is something that security people would not think of. And, those from “vision” would use it. We could use entropy, or something similar. But, we could not see program as pixels. This is great about “crowdsourcing” Then, modeling – XGBoost. Gradient boosting. It is basically building a tree, and on build additional depth, to minimize the error. Then repeat. (and some other clever method) Ensemble. - basically random forest. Separate training dataset into pieces, and let them vote later.
• #31: But, to all the kagglers, to win the competition, it is about feature engineering. feature extraction. what does it mean. features, think of them like sense. missing a sense. missing a smell, in food.  There are frameworks to try every algorithm at the same time. So, it comes down to “conceptualizing the data”. Or the “feature engineering” that decide the winner. Lets say you are collecting color. You don’t have to know “apple” is “red”. But you need to know “color” is important.
• #32: 22min Feature extraction is like “taking a snapshot” Missing a critical feature, is like For example, food. It looks good. But, how does it smell? Is it hot or cold? Is it spicy or sweet.
• #33: Many people think “data science” == “very difficult uber smart algorithm” For most of the competition, The winning team, has done, using one of these 3 algorithm families. And, rest was about “feature engineering” You don’t have to understand everything. But, What is important is that, you SHOULD KNOW how it work. Similar to driving a car. You don’t need exactly how the every parts of the car works. But, you do need to understand how Internal Combustion Engine works. How the outside temperatue affects. How the physics works, such large mass has more momentum than smaller mass. These days, there are frameworks that applys every algorithm. So, picking an algorithm matters less. These are 3 good algorithms. XGBoost is GBM (gradient boosted machine). Boosting is a way of building a tree, to minimize the misclassification. In the “kaggle” world, the global top 10 has “god status”, and the first place winner has been saygin “when in doubt, use XGBoost”. It is flexible. Deep Learning is “kinda new hot method”. And for “computer vision” challenges, DL has been sweeping the wins. For smaller data, or scienfitic data, such as weather data, where there is no “fine grain” manipulation. SVR has been working well. If you are sure that there is some “magic mathermatical formula”, such as physics, then SVR works well.
• #35: 23 min mark Read the quote. Netflix acatually didn’t use the winning solution into the production system. For crowssourcing, and for everything else in the life, It is about, taking that knowledge, and making it better.
• #36: This might be “backward reasoning” Opcode sequence works well. But this must be the reason. Data science has no such thing like that. It works, and we are trying to conceptualize “why” it works. Then, you use the same conceptuliaztion for other problem sets. “imagine you are a CPU”. I don’t think this is that difficult for this conference crowd.
• #37: I like communication theory. I like linguistics. 30 sec intro to NLP. For “human Language”, this is the very basic structure. In a sentence, the central piece of information is about “Verb”. The action. What is happening. Thus, if you are looking for “golden feature” in NLP, you look for “Verbs”. And, Computers are no different. “We are still under the “mental frameworks” of communication theory”. Human and computers are both “communicating organism”.
• #38: 30 sec intro to NLP. For “human Language”, this is the very basic structure. In a sentence, the central piece of information is about “Verb”. The action. What is happening. Thus, if you are looking for “golden feature” in NLP, you look for “Verbs”. And, Computers are no different. “We are still under the “mental frameworks” of communication theory”. Human and computers are both “communicating organism”.
• #39: XGBoost is awesome. But, it is not “solve it all”. I don’t like it that much, because a complicated tree is hard to interpret. Remember the 3 “usually winning” algorithms. XGBoost, DeepLearning, and SVR. SVR is parametric. Don’t use SVR for malware. It just does not work. XGBoost is non-parametic. Deep Learning is hard to say what it is. It is stacking of parametric component in non-parametric way. It worked well for this competition, because -The problem set was easy. It didn’t contain junks. -it is classifly inbween malwares. No “junks” and “blah” and “shady”. It is from 30000 files? In the real world, XGBoost might not be enough…..
• #40: … If you are familiar with “machine learning” scene, “deep learning” has been the hottest topic.
• #41: Then, I came across Nilsimsa hash. I didn’t make it. It is from a mysterious origin. and it looks remarkable similar to DNN.
• #42: It is a “BEAST of an algorithm”. – mix of modulus arithematic, and frequency analysis. efficient storage. You can view that in 4 large stages. I prefer this “intermediate storage”, because it is easier to trace back.
• #43: Sliding Window. N-gram. Could do byte-grams. Feature Hashing
• #44: Bloom Filter is already an awesome data structure. In engineering, We usually talk about Space and Time trade-offs. BloomFilter is a mix of Space + Time + Allowable Error Tradeoffs. If it is there, it is there. If it is not there, it could say it is there. It is awesome. Look it up.
• #45: Counting Bloom Filter is a modification of Bloom Filter. It is a bloom filter, that can count more than 1. Instead of Yes or No (binary), we could store quantity. N-gram + CBF is just a way of storing a “LARGE Feature Space” into “the most tight space”. In Data Science, it is called “Feature Hashing.”. But, not using bloomfilter. Bloomfilter can reduce the memory space significantly.
• #46: We are turning back to “binary” format here. This is the step, where each “n-gram” is compared against the average. Each small snippet is compared against the whole file. Mostly, to prep it for the final stage of “Hamming Distance”
• #47: And, all you need to do, is to find the another known file with the lowest hamming distance away. Hamming Distance is “XOR bitwise, then count 1 bits.” It is much faster than “edit distance”.
• #48: 26 min mark We don’t want to stop at this. Just read, and go over each element. In the next slides. (this is such a generic slide, that it could fit into any slide…)
• #49: We are not the only company, that has researching into improving Nilsimsa. Trend Micro did this. It’s a very good improvement. Is there any other vendors, who are looking into this? Please raise your hand?? <then, explain what Trend did> Why this works well is… < next slide>
• #51: You can view this as “minimizing the collision” It would double the size of the memory. But, at this point, it is already tiny memory.
• #52: If TLSH is improving the “height” of the CBF, we could also look at the “width”. Earlier than Later. since once info is lost, you can’t get it back. If 2 different element falls into the same bucket in CBF, that is where info-collision, or info-loss is occurring. Basically, the CBF need to be large enough space to contain “information/entropy/ability to store information” So, how to measure a good size.
• #53: Hyper Log Log is a cheap and fast way to estimate how much space you need. This is how BitCoin/BlockChain decide the “hash difficulty”.
• #54: Competition only cares about the accuracy. But, in the real world, Our industry has to deal with “deliberate attack”. Bloom Filter is where “hash collision” is a norm.
• #55: If dual hashed , the memory requirement doubles. in security, if preventing attack is imporatant, we can use some spaces. Feeling paranoid? Do a triple hash. (It is Dual Hash. Not Double-hash == (hash(hash(val)), as in BlockChain.)
• #56: There are more ways to improve. WE can make our problem, more difficult. Why? Because it is fun.
• #57: PE file has been transformed into ASM file, using disassembler. This is assuming, we know the PE file structure well.
• #58: Let’s say “ Some country, or some organization made a totally different computer architecture. You know it is compiled code. Or some sort of language. there is a compiler, with totally different opcode structure” We don’t have Disassembler, What to do?
• #59: Because what is importance is the “frequence of the sequecnce of the frequent elements.” We make our own ad-hoc DisASM. “Select-gram” is some word I made up. I don’t know what it is called in the industry. Basically, take the most frequently occurring bytes. These will be the “popular Opcodes”. Any “efficient language” cannot escape this pattern. If escaped, it start to lose the notion of a language.
• #60: This is the last slide. If you have been dozing off, this is a single slide, to take home with. Data Science, will get big, in the near future. And if you are stuck, you should come back to this framework. All you need to do, is to To pass as much information to the next step. Without introducing too much noice, in the link. And, the rest will work out by itself.