Kaggle and data science

© DataRobot, Inc. All rights reserved.
Kaggle
and
Data Science
Japan, 2018

Sergey Yurgenson
Director, Advanced Data Science Services
Kaggle Grandmaster

Kaggle
● Kaggle is a platform for data science competitions
● It was created by Anthony Goldbloom in 2010 in Australia and then moved to San
Francisco
● In March of 2017 it was acquired by Google
● Right now many other start-up are trying to replicate the same idea, but Kaggle is still the
most known in data science community name
● As of now Kaggle hosted more than 280 competitions and has more than 1 million
members from more than 190 countries

Kaggle competitions
● Most of Kaggle competitions are predictive modeling competition
● Participants are provided with training data to train their models and test data with
unknown targets
● Participants need to calculate predictions for test data and submit those
predictions to Kaggle platform.
● Accuracy of predictions is evaluated using predefined objective metric and that
result is provided back to participants.
● Model performance of all participants is publicly available and participants can
compare quality of their models with models of other participants
● Many competitions have monetary prizes for top finishers

Kaggle competitions

Kaggle ranking
● Based on competitions performance Kaggle ranks members using points and
awards titles for top finishing in competitions
● For example to get title of master member needs to earn one gold medal and
two silver medal. For competitions with 1000 participants it means to finish
once in top 10 places and twice in top 50.

Kaggle ranking

Kaggle and Data Science

Why do you dislike Kaggle ?
● Kaggle competition does not have much in common with real Data Science
○ The problems are already well formulated with metrics predefined. In an industry setting there is
ambiguity, and knowing what to solve is one of the key steps towards a solution.
○ Data is most cases is already provided and is relatively clean.
○ The goal is more leaderboard driven rather than understanding driven. Winning a competition
versus why an approach works is a top priority. Results may not be trustworthy.
○ There are chances of overfitting to test data with repeated submissions.
○ In most cases the solution is an ensemble of algorithms and not “productionizable”.
https://www.quora.com/Why-do-you-dislike-Kaggle

True or False ?
● “The problems are already well formulated with metrics predefined. In an
industry setting there is ambiguity, and knowing what to solve is one of the
key steps towards a solution.”

Problem is well formulated
Mostly True , however...
● Need for criteria is inherited property of any competition.
● In real world not all data scientists are free to select and reformulate the problem. Many problems
are already defined with assigned specific success criteria.
● We learn many subjects and skills by solving provided predefined problems, doing predefined
exercises. We learn math by solving problems from textbooks, we learn physics by solving
problems from textbooks. Problems already formulated. By solving problems we also learn how
to formulate problems, what is suitable approach in particular data science situation.
● We also have to admit that evaluating business value of solving the problem is completely out of
scope of Kaggle competitions. While business value analysis and problem prioritization is
important part of many real life data science projects.

True or False ?
● “Data is most cases is already provided and is relatively clean.”

Data is clean
Half true
● In many competitions datasets are
○ Very big
○ Have multiple tables
○ Some records are duplicated and mislabeled
○ Contain combination of structured data and unstructured data
● Some competitions encourage search for additional sources of data
● Many data leaks
● Often features names and meaning are not provided making problem even more difficult than in real
world
● Data may be intentionally distorted to conform to data privacy laws

Data is clean
● Complex data structure ● Big datasets
● No meaningful feature names

Data is clean
● Kaggle competitions teach unique data manipulation skills:
○ Dealing with data with hardware limitations : efficient code, smart sampling, clever encoding…
○ Using EDA to uncover meaning of data without relying on labels or other provided information
○ Data leaks discovering based on the data analysis

True or False ?
● The goal is more leaderboard driven rather than understanding driven. Winning
a competition versus why an approach works is a top priority. Results may not
be trustworthy.

No understanding
True but maybe not that important
● Assumes that model we can not understand is less valuable than model we can understand
○ Model is not necessarily used for knowledge discovery
○ In real life we often use something and rely on something we do not completely understand
○ If something that we do not understand can not be trustworthy then how we ever trust other
people?
○ Even complex machine learning model may provide simplification of even more complex real
system

No understanding
● Ignores all new research of model interpretability
○ Feature importance
○ Reason codes
○ Partial dependence plots
○ Surrogate models
○ Neuron activation visualization
○ ...
● Those methods allow us to analyze and understand behaviour of models as complicated as GBM and
Neural Networks

No understanding ?

True or False ?
● There are chances of overfitting to test data with repeated submissions.

Overfitting
False
● Complete misunderstanding of how Kaggle works
○ Test data in Kaggle competition is split into two parts - public and private
○ During competition models are evaluated only on public part of the test set
○ Final results are based only on private part of the test dataset
○ Thus final model evaluation is based on completely new data
● One of first lessons all competitions participants learn very fast
○ Do not overfit leaderboard.
○ Create training/validation partition which reflect as much as possible test data including
seasonality effects and data drift

True or False ?
● In most cases the solution is an ensemble of algorithms and not
“productionizable”.

Difficult to put in production
Half True, half false
● Yes, in most cases top models are complicated ensembles
● Difficult to put in production if one does it one-by-one for each model separately
● Easy if one uses appropriately developed platform that can handle many models and blenders

True or False ?
● Sometimes, a 0.01 difference in AUC can be the difference between 1st place
and 294th place (out of 626) . Those marginal gains take significant time and
effort that may not be worthwhile in the face of other projects and priorities
https://www.quora.com/How-similar-are-Kaggle-competitions-to-what-data-scientists-do

Marginal gain is not valuable
Not always true
● Often we ourselves advise clients on balance between time spent and model performance
● However in investment world 0.01 AUC difference means difference in millions of dollars of gain or
loss
● Competition aspect of the data science problem with small margins drives innovation
○ New preprocessing steps
○ New feature engineering ideas
○ Continues testing of new algorithms and implementations (GBM - XGboost - LightGBM -
CatBoost)

● “Kaggle competitions cover a decent amount of what a data scientist does.
The two big missing pieces are:
○ 1. taking a business problem and specifying it as a data science problem
(which includes pulling the data and structuring it so that it addresses that
business problem).
○ 2. putting models into production.”
Anthony Goldbloom

● Kaggle is a competition
● “Real” Data Science is ...
also competition

Kaggle to “real life” Data Science
● DataRobot - created by top Kagglers
Owen Zhang
Product Advisor
Highest: #1
Xavier Conort
Chief Data Scientist
Highest: 1st
Sergey Yurgenson
Director- AI Services
Highest: 1st
Jeremy Achin
CEO & Co-Founder
Highest: 20th
Tom de Godoy
CTO & Co-Founder
Highest: 20th
Amanda Schierz
Data Scientist
Highest: 24
DataRobot automatically replicates the steps seasoned data scientists take. This allows
non-technical business users to create accurate predictive models and data scientists to add
to their existing tool set.

Kaggle and data science

More Related Content

What's hot (20)

Similar to Kaggle and data science (20)

More from Akira Shibata (20)

Recently uploaded (20)

Kaggle and data science