Predict oscars (5:11)
- 1. Predicting the Oscars with Data Science http://bit.ly/tf-predict-oscars
- 2. About me • Jasjit Singh • Self-taught developer • Worked in finance & tech • Co-Founder Hotspot • Thinkful General Manager
- 3. About us Thinkful prepares students for web development & data science jobs with 1-on-1 mentorship programs
- 4. What’s your background? • I have a software background • I have a math or stats background • None of the above
- 5. Learning goals • “Data Science Process” • Python’s data science toolkit and methods • Basic machine learning concepts
- 6. Data Science Process • Frame the question. • Collect the raw data. • Process the data. • Explore the data. • Communicate results.
- 7. Frame the question • Who will win the Oscar for Best Picture?
- 8. Collect the Data • What kind of data do we need?
- 9. Collect the Data • Financial data (Budget, box office…) • Reviews, ratings and scores. • Awards and nominations.
- 10. Process the data • How’s the data “dirty” and how can we fix it?
- 11. Process the data • User input, redundancies, missing data… • Formatting: adapt the data to meet certain specifications. • Cleaning: detecting and correcting corrupt or inaccurate records.
- 12. Explore the data • What are the meaningful patterns in the data? • How meaningful is each data point for our predictions?
- 14. Jupyter Notebooks • One of data scientist’s everyday tools. • Lets us show our work: can read the story, follow the process, and run the code • Find the links in our classroom tool.
- 15. NumPy • The fundamental package for scientific computing with Python. • Lets us store our data into special multi- dimensional array objects. • Many methods for fast operations on arrays.
- 16. Pandas • Fundamental high-level building block for doing practical, real world data analysis in Python. • Built on top of NumPy.
- 17. Scikit-learn • Python module for machine learning.
- 18. Working through the code • The goal is to have you follow along. Towards the end of the class, you can play around with the code yourself • Go to Jupyter notebook (http://bit.ly/tf- jupyter) • Open “Stage One.ipynb”
- 19. Initial imports and loading data with Pandas
- 20. Understanding your data • .head(n) method: Returns first n rows. • .value_counts() method: Returns the counts of unique values in that column
- 22. Processing our data: Formatting • Ratings are in a non-numeric format. We need to assign each rating a unique integer so that Python can handle the information. • We’ll do that with the .ix method
- 24. Processing our Data: Cleaning
- 25. Classification vs Regression • Regression — Predict values • Classification — Predict categories.
- 26. Classification
- 27. Classification ?
- 28. For our model we’ll use a decision tree • It breaks down a dataset into smaller and smaller subsets. • The final result is a model with a tree structure that has: • Decision nodes: ask a question and have two or more branches. • Leaf nodes: represent a classification or decision.
- 30. Why a decision tree? • Decision tree is just one of many algorithms you can use (e.g. Naive Bayes, SVM, Least Squares, Logistic, etc) • We’re using decision trees because it mirrors human decision making so it’s simpler to understand and interpret
- 31. Creating your first Decision Tree You will use the scikit-learn and numpy libraries to build your first decision tree. We will need the following to build a decision tree • target: A one-dimensional numpy array containing the target from the train data. • features: A multidimensional numpy array containing the features/predictors from the train data.
- 32. Creating your first Decision Tree
- 34. Problem #1: Poor Predictions • We only got 1 out of 4 movies correct (Amadeus, 1984) • Open Stage 2 and lets try and improve our predictions!
- 35. Solution: Modify the feature list
- 36. Run the prediction again
- 37. Success!? • We now got 3 out of 4 movies correct • Our model predicted Inglorious Bastards would win. Hurt Locker won instead
- 38. Problem #2: Overfitting • Resulting model too tied to the training set. • It doesn’t generalize to new data, which is the point of prediction.
- 39. Overfitting, a visualization In overfitting, a statistical model describes random error or noise instead of the underlying relationship.
- 40. Solution: Random Forest Classifier • Random Forest Classifiers use many Decision Trees to build a classifier. • We introduce a bit of randomness. • Each Tree can give a different answer (a vote). The final classification is the most common amongst the Trees.
- 42. Creating your first Decision Tree
- 44. Results
- 45. 1976 Rocky
- 46. 1984 Amadeus
- 49. And our prediction for the 2016 Oscar is…
- 51. We can predict the Oscars
- 52. Except for 2017 ¯_(ツ)_/¯
- 53. Next steps for learning • Google (self-taught) • Coursera • Bootcamps
- 54. 1-on-1 mentorship enables flexibility
- 55. Next steps for learning
- 56. Graduate outcomes Job Titles after GraduationMonths until Employed
- 57. Special Introductory Offer • Prep course for 50% off — $250 instead of $500 • Covers math, stats, Python, and data science toolkit • Option to continue into full data science program • If you’re interested talk to me after or email me at jasjit@thinkful.com