Creativity and Curiosity - The Trial and Error of Data Science
- 1. Creativity and Curiosity THE TRIAL AND ERROR OF DATA SCIENCE
- 2. I love data, everything comes easy to me…
- 3. There are so many things to try and explore on a given problem, where to start? • Language (Julia, Python,R, C++,etc) • Visualization (ggplot, Tableau, D3,etc) • Pre-process (standardize, variance scaling, feature encoding, etc) • Classifier (GLM, SVM, SGD, Knn, Random Forest, etc) • Post-process (Rule-truncatation, post-pruning, etc) • Ensemble (weighted average, min, max, probabilities, etc)
- 4. Where Many Individual Come To Die… (Model Tuning Hell)
- 5. Structured Process Allows you to remove uncertainty and ensure outcomes in a methodical way. Gives you an idea of what activities to do and when. Details for each project varies, however the structure should stay the same. The process is almost never linear, you should revisit each step again and again. Knowledge Discovery Process 1. Define the goal 2. Explore the data 3. Prepare the data 4. Choosing and evaluating models 5. Ensemble
- 6. Define the Goal • Why do the sponsors want the project in the first place? What do they lack, and what do they need? • What are they doing to solve the problem now, and why isn’t that good enough? • What resources will you need: what kind of data? Do you have domain experts to collaborate with, and what are the computational resources? • How do the project sponsors plan to deploy your results? What are the constraints that have to be met for successful deployment? • Is the data quality good enough?
- 7. Define the Goal Modeling: • Classification • Scoring • Ranking • Clustering • Finding relations • Characterization Model Evaluation and critique • Is it accurate enough for your needs? Does it generalize well? • Does it perform better than “the obvious guess”? Better than whatever is currently in use? • Do the results of the model (coefficients, clusters, rules) make sense in the context of the problem domain?
- 8. Explore the Data Use summary statistics to spot problems • Missingness • Data ranges (too wide/too narrow) • Invalid values • Outliers • Units
- 9. Explore the Data Use graphics and visualization to spot problems Single-Variable First • Peak of distribution? • How many peaks? • How normal (or lognormal is the data? • How much data variation is there? Is it concentrated in a certain interval or category? • Use histograms, density plots, bar charts, scatter plots with smoothing curve.
- 10. Prepare the Data Cleaning Data • Treating missing values (NAs) • Data Transformations Sampling for Modeling and Validation • Test and training splits • Creating sample group column • Record grouping
- 11. Choosing and Evaluating Models Mapping problems to machine learning tasks (use a problem-to-method mapping) • Solving classification problems • Naïve Bayes • Decision Trees • Logistic Regression • Solving scoring problems • Linear Regression • Logistic Regression • Working without known targets • K-means clustering • Apriori algo to find association rules • Nearest neighbor
- 12. Choosing and Evaluating Models Evaluating models • Evaluating classification models • Confusion matrix • Precision • Recall • Sensitivity • Specificity • Evaluating scoring models • Root Mean Square Error • R-squared • Correlation • Absolute Error
- 13. Choosing and Evaluating Models Evaluating models • Evaluating probability models • Area Under the Curve • Log Likelihood • Deviance • Akaike Information Criterion (AIC) • Entropy • Evaluating ranking models • Intra-cluster distances • Cross-cluster distances
- 14. Choosing and Evaluating Models Validating models • Identify common model problems • Bias – systematic error • Variance – oversensitivity of the model • Overfit – doesn’t generalize well • Nonsignficance – relation may not hold • Ensuring model quality • Testing on Held-Out Data • K-Fold Cross Validation • Significance Testing • Confidence Intervals
- 15. Ensemble How do I bring all my work together? • Weighted average • Min • Max • Voting • Stacking • Neural network
- 16. More Ideas Learn about ensemble methods, regularization, and principled dimension reduction • Hastie, Tibshirani, and Friedman’s The Elements of Statistical Learning, Second Edition • If you want to understand the consequences of a method, has a math bent Keep your saw sharp Plug-in
- 17. Using your creativity and curiosity you can slay mighty data science problems.
Editor's Notes
- #5: Number of iterations Interaction depth Number of trees Learning rate