Using Deep Learning And NLP To Predict Performance From Resumes

Editor's Notes

• #2: My name is Ben Taylor, I’m the Chief Data Scientist for a great startup called HireVue. Today I will be talking about NLP as well as deep learning. This talk is meant to be an introductory to those who are less familiar with NLP
• #4: HR has seen a great cross pollination from other industries, I am an example of that. I studied chemical engineering through both undergrad and graduate programs. I then went and worked as a quant for a Manhattan hedge fund manager on a 600 GPU cluster. And… now I’m in HR. Oh, and I LOVE love love backcountry snowboarding. I took this photo last week and I go 2-3 times a week before work. I will never work anywhere besides Utah because of this.
• #5: What is HireVue? We are a digital interviewing & interaction company We are backed by Sequoia Capital In 2014 we were # 10 for Forebes most promising companies Global, supporting digital interviews in 189 countries
• #7: Building predictive models from competencies or other numeric features is straight forward. You take the columns or features of interest on the left, and the performance labels on the right and you pass them through a type of regression. Excel will do this, many programs will do this just fine. If you are MORE advanced you can use programs like R, python to do more advanced regressions like random forest, gradient boosting regression, or other… Raise your hand if you know how to build a model from this data?
• #8: Now to throw a wrench in your process, I have decided to inject open ended essay response into my feature set. Raise your hand again if you know how/what to build a predictive model with this? Most classical statistician/mathematicians/analysis are justifiabilty confused by this is
• #9: Like most data science or machine learning tricks, once they are explained at a 5th grade level, we tend to be underwhelmed. The computer can’t understand the raw text in its native format, it must convert them to numbers. One way to accomplish this is to map the text to numeric replacements. Good, better, best, can become 1,2, and 3. What would you do if you had something like male or female? You can map these, because if you made the male 2 and the female 1 are you being sexist? The are completely different, they can’t be directly compared. Therefore they must be tokenized where each column now represents the variable, so columns are created
• #10: In the case of text you can have a LOT of columns. In some cases you may exceed 10,000, 100,000, or even 10M columns. Imagine attempting to open a dataset like this in excel, with over 1M columns. You have to use special software in R or python that can handle these types of data objects in a compressed sparse format. Can anyone see what the problems are with this approach? There is a major drawback. [sequence loss[
• #11: Bag of words! This is called bag of words because you can visualize the words as if they are picked up by a paper bag. All sequence and ordering is lost. Is that a problem? Maybe.
• #12: I analyzed some twitter data for Skullcandy a few years ago. When we presented our results to the engineering team we asked them “If someone says the F word in a tweet and tags your company… is that a bad thing?”. Think about it, for most of us in the room, with the companies we work for and represent does that give you anxiety thinking about that? The reason that gives us anxiety is because we know that would be a terrible thing and it would be really bad. Skullcandy knew their customer base well enough, they said they were sure. And sure enough the data showed that half of the people saying the F word on twitter and tagging Skullcandy said nice things, and the other have said mean things. So a word that is typically polarizing had no impact. Bad.... Bad is a bad word Ass.... Ass is a bad word But... If I say “bad ass” my bag of words method is going to see that as a very very bad thing, when in fact is is a very nice thing. How do we fix that?
• #13: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #15: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #16: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #17: Excel 2002: 256 columns Excel 2003: 256 columns Excel 2016: 1,048,576 rows by 16,384 columns
• #18: Excel 2002: 256 columns Excel 2003: 256 columns Excel 2016: 1,048,576 rows by 16,384 columns
• #20: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #21: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #22: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #23: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #25: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #26: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #27: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #28: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #29: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #30: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #31: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #32: One point to bring up is proper model validation. I used to be confused when someone said train on 70, test on 30. Or train on 80 test on 20. Who was right? The answer I have settled on now is their of them are right. Explain the conflict.
• #33: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #34: Introduction to n-gram tuples. Not only can we create unique placeholders for words, but we can also do it for word pairs. We can do it for single word, two word, three word, as many pairs as we want. But as we do that the columns count explodes exponentially and… the reoccurance of that observation goes down... Both of these are bad so you have rapidly diminensing returns. Also, if you throw a single adjective or word inbetween your expected bi-gram it won’t be found.
• #45: Now that we have some basic NLP background we will change gears to RESUME modeling. Who hates looking through stacks of resumes? Not sure, sometimes it can be fun, but depending on the stack size you might be spending 30 seconds on a resume, 7 seconds? How quickly can you screen a resume? Think about what you are doing when you screen a resume? Where you are your eyes looking? School GPA Skills Work history Name? Hopefully you don’t look at name.
• #46: To review possible flow using NLP first we have an unstructured resume, we are forced to structure it somehow, then tokenize or munge the data into numeric
• #47: Sometimes we can predict things without opening up the resume. Checkout these file extensions. It is hard to see, but statistically someone who uploads a DOC resume is be more likely to interview well than someone who does RTF. Likewise DOCX beats DOC And PDF beats DOCX
• #48: What do we do with ALL of these formats? DOCX, txt, pdf? This is actually a big problem, we can’t do anything cool until we standardize the formats. Luckily there is a free open source office platform that can do the conversion for us. I recommend converting it to either txt or html.
• #49: Now that we have text we can write specific feature grabbers like GPA. For the resumes we analyzed we noticed that GPAs were only included 1/5 resumes. Also this is where the distributions fell, not very many below 3.00 GPA are reporting. What do you do if someone does not include a GPA? When a feature is missing you MUST replace. Do you replace the GPA with a 0? That’s harsh, a 2.0? 4.0? Average? It depends
• #50: Testing prediction quality we found that optimal prediction quality comes when we replace the GPA with a 3.6 What does that mean? That means if you have less than a 3.6 GPA, as far as the computer is concerned, including it doesn’t help you.
• #51: There are so many features to create in the case of a resume model, you can save yourself a lot of time using a resume parsing service. The majority of the value comes from BOW. You quickly begin approaching incremental returns where a LOT of effort results in marginal gain. Malicious resume
• #53: The biggest value that deep learning offers is automatic feature value discovery. This has been incredibly valuable with image, hitting new high points. It can also be valuable for text, allowing you to forget the concept of a tuple or n-gram.
• #54: In the end the computer always needs a number, but in this case it is looking at very large sequences of numbers (100-300 word windows). Run it on entire resume: What is the prediction?
• #55: In the end the computer always needs a number, but in this case it is looking at very large sequences of numbers (100-300 word windows). Run it on entire resume: What is the prediction?
• #56: In the end the computer always needs a number, but in this case it is looking at very large sequences of numbers (100-300 word windows). Run it on entire resume: What is the prediction?
• #57: In the end the computer always needs a number, but in this case it is looking at very large sequences of numbers (100-300 word windows). Run it on entire resume: What is the prediction?
• #58: Fun tangent, does resume formatting matter? Margins? Font size, layout?
• #60: Would you ever hire from just a resume? Why not?
• #61: For interview modeling we use spoken text, which is more difficult because of the transcription accuracies. Raw audio (utterence, repetition) Video, micro expression (Lie to me)