What is wrong with data challenges
4 likes1,953 views
This document summarizes discussions from a workshop on data science challenges. It notes both benefits and limitations of crowdsourced data challenges, including lack of direct access to solutions and limited incentive for collaboration. It then describes the development of a new format called Rapid Analytics and Model Prototyping (RAMP) that aims to address these limitations by facilitating direct access to code and data, and incentivizing diversity and collaboration. RAMP was created based on experiences with the Higgs Boson Machine Learning Challenge and drawing from literature on open innovation.
What is wrong with data challenges
- 1. Center for Data Science Paris-Saclay1 CNRS & University Paris Saclay Center for Data Science BALÁZS KÉGL WHAT IS WRONG WITH DATA CHALLENGES THE HIGGSML STORY: THE GOOD, THE BAD AND THE UGLY
- 2. 2 Why am I so critical? ! Why do I mitigate our own success with the HiggsML?
- 3. 3 Because I believe that there is enormous potential in open innovation/crowdsourcing in science. ! The current data challenge format is a single point in the landscape.
- 4. 4 Olga Kokshagina 2015 INTERMEDIARIES: THE GROWING INTEREST FOR « CROWDS » - > EXPLOSION OF TOOLS ! Crowdsourcing ! is a model leveraging on novel technologies (web 2.0, mobile apps, social networks) ! To build content and a structured set of information by gathering contributions from large groups of individuals 5
- 5. Center for Data Science Paris-Saclay CROWDSOURCING ANNOTATION 5
- 6. Center for Data Science Paris-Saclay CROWDSOURCING COLLECTION AND ANNOTATION 6
- 7. Center for Data Science Paris-Saclay CROWDSOURCING MATH 7
- 8. Center for Data Science Paris-Saclay CROWDSOURCING ANALYTICS 8
- 9. Center for Data Science Paris-Saclay OPEN SOURCE 9
- 10. Center for Data Science Paris-Saclay NEW PUBLICATION MODELS 10
- 11. Center for Data Science Paris-Saclay THE BOOK TO READ 11
- 12. Center for Data Science Paris-Saclay • Summary of our conclusions after the HiggsML challenge • The good, the bad and the ugly • Elaborating on some of the points • Rapid Analytics and Model Prototyping • an experimental format we have been developing 12 OUTLINE
- 13. Center for Data Science Paris-Saclay13 CIML WORKSHOP TOMORROW
- 14. Center for Data Science Paris-Saclay • Publicity, awareness • both in physics (about the technology) and in ML (about the problem) • Triggering open data • http://opendata.cern.ch/collection/ATLAS-Higgs-Challenge-2014 • Learning a lot from Gábor on how to win a challenge • Gábor getting hired by Google Deep Mind • Benchmarking • Tool dissemination (xgboost, keras) 14 THE GOOD
- 15. Center for Data Science Paris-Saclay • No direct access to code • No direct access to data scientists • No fundamentally new ideas • No incentive to collaborate 15 THE BAD
- 16. Center for Data Science Paris-Saclay • 18 months to prepare • legal issues, access to data • problem formulation: intellectually way more interesting than the challenge itself, but difficult to “market” or to crowdsource • once a problem is formalized/formatted to challenge, the problem is solved (“learning is easy” - GaelVaroquaux) 16 THE UGLY
- 17. Center for Data Science Paris-Saclay • We asked the wrong question, on purpose! • because the right questions are complex and don’t fit the challenge setup • would have led to way less participation • would have led to bitterness among the participants, bad (?) for marketing 17 THE UGLY
- 18. Center for Data Science Paris-Saclay • The HiggsML challenge on Kaggle • https://www.kaggle.com/c/higgs-boson 18 PUBLICITY, AWARENESS
- 19. Center for Data Science Paris-Saclay PUBLICITY, AWARENESS 19 B. Kégl / AppStat@LAL Learning to discover CLASSIFICATION FOR DISCOVERY 14
- 20. Center for Data Science Paris-Saclay AWARENESS DYNAMICS 20 • HEPML workshop @NIPS14 • JMLR WS proceedings: http://jmlr.csail.mit.edu/proceedings/papers/v42 • CERN Open Data • http://opendata.cern.ch/collection/ATLAS-Higgs-Challenge-2014 • DataScience@LHC • http://indico.cern.ch/event/395374/ • Flavors of physics challenge • https://www.kaggle.com/c/flavours-of-physics
- 21. Center for Data Science Paris-Saclay LEARNING FROM THE WINNER 21 https://indico.lal.in2p3.fr/event/2692/contribution/1/material/slides/0.pdf
- 22. Center for Data Science Paris-Saclay LEARNING FROM THE WINNER 22 • Sophisticated cross validation, CV bagging • Sophisticated calibration and model averaging • The first step: pro participants check if the effort is worthy, risk assessment • variance estimate of the score • Don’t use the public leaderboard score for model selection • None of Gábor’s 200 out-of-the-ordinary ideas worked https://indico.lal.in2p3.fr/event/2692/contribution/1/material/slides/0.pdf
- 23. Center for Data Science Paris-Saclay BENCHMARKING 23 CLASSIFICATION FOR DISCOVERY 15
- 24. Center for Data Science Paris-Saclay BENCHMARKING 24 But what score did we optimize? ! And why?
- 25. Center for Data Science Paris-Saclay count (per year) background signal probability background signal CLASSIFICATION FOR DISCOVERY 25 Goal: optimize the expected discovery significance flux × time selection expected background say, b = 100 events total count, say, 150 events excess is s = 50 events AMS = = 5 sigma ground expectation µb. When optimizing the design of gion G = {x : g(x) = s}, we do not know n and µb. As we estimate the expectation µb by its empirical counter- + b to obtain the approximate median significance ⇣ (s + b) ln ⇣ 1 + s b ⌘ s ⌘ . (14) x + 1) = x + x2/2 + O(x3), AMS2 can be rewritten as MS3 ⇥ s 1 + O ✓⇣ s b ⌘3 ◆ , AMS3 = s p b . (15) tically indistinguishable when b s. This approxima- nding on the chosen search region, be a valid surrogate selection thresholdselection threshold
- 26. Center for Data Science Paris-Saclay How to handle systematic (model) uncertainties? • OK, so let’s design an objective function that can take background systematics into consideration • Likelihood with unknown background b ⇠ N(µb, b) L(µs, µb) = P(n, b|µs, µb, b) = (µs + µb)n n! e (µs+µb) 1 p 2⇡ b e (b µb)2 /2 b 2 • Profile likelihood ratio (0) = L(0, ˆˆµb) L(ˆµs, ˆµb) • The new Approximate Median Significance (by Glen Cowan) AMS = s 2 ✓ (s + b) ln s + b b0 s b + b0 ◆ + (b b0)2 b 2 where b0 = 1 2 ⇣ b b 2 + p (b b 2)2 + 4(s + b) b 2 ⌘ 1 / 1 26
- 27. Center for Data Science Paris-Saclay HOW TO HANDLE SYSTEMATIC UNCERTAINTIES 27 Why didn’t we use it?
- 28. Center for Data Science Paris-Saclay28 How to handle systematic (model) uncertainties? • The new Approximate Median Significance AMS = s 2 ✓ (s + b) ln s + b b0 s b + b0 ◆ + (b b0)2 b 2 where b0 = 1 2 ⇣ b b 2 + p (b b 2)2 + 4(s + b) b 2 ⌘ 1 / 1 New AMS ATLAS Old AMS
- 29. Center for Data Science Paris-Saclay LEARNING FROM THE WINNER 29 • Sophisticated cross validation, CV bagging • Sophisticated calibration and model averaging • The first step: pro participants check if the effort is worthy, risk assessment • variance estimate of the score • Don’t use the public leaderboard score for model selection • None of Gábor’s 200 out-of-the-ordinary ideas worked
- 30. Center for Data Science Paris-Saclay THE TWO MOST COMMON DATA CHALLENGE KILLERS 30 Leakage Variance of the test score
- 31. Center for Data Science Paris-Saclay VARIANCE OF THE TEST SCORE 31
- 32. Center for Data Science Paris-Saclay • Challenges are useful for • generating visibility in the data science community about novel application domains • benchmarking in a fair way state-of-the-art techniques on well-defined problems • finding talented data scientists • Limitations • not necessary adapted to solving complex and open-ended data science problems in realistic environments • no direct access to solutions and data scientist • no incentive to collaboration 32 DATA CHALLENGES
- 33. 33 We decided to design something better
- 34. Center for Data Science Paris-Saclay • Direct access to code, prototyping • Incentivizing diversity • Incentivizing collaboration • Training • Networking 34 RAPID ANALYTICS AND MODEL PROTOTYPING (RAMP)
- 35. Center for Data Science Paris-Saclay • Our experience with the HiggsML challenge • Need to connect data scientist to domain scientists and problems at the Paris-Saclay Center for Data Science • Collaboration with management scientists specializing in managing innovation • Michel Nielsen’s book: Reinventing Discovery • 5+ iterations so far 35 WHERE DOES IT COME FROM?
- 36. Center for Data Science Paris-Saclay UNIVERSITÉ PARIS-SACLAY 36 + horizontal multi-disciplinary and multi-partner initiatives to create cohesion
- 37. Center for Data Science Paris-Saclay37 Center for Data Science Paris-Saclay A multi-disciplinary initiative to define, structure, and manage the data science ecosystem at the Université Paris-Saclay http://www.datascience-paris-saclay.fr/ Biology & bioinformatics IBISC/UEvry LRI/UPSud Hepatinov CESP/UPSud-UVSQ-Inserm IGM-I2BC/UPSud MIA/Agro MIAj-MIG/INRA LMAS/Centrale Chemistry EA4041/UPSud Earth sciences LATMOS/UVSQ GEOPS/UPSud IPSL/UVSQ LSCE/UVSQ LMD/Polytechnique Economy LM/ENSAE RITM/UPSud LFA/ENSAE Neuroscience UNICOG/Inserm U1000/Inserm NeuroSpin/CEA Particle physics astrophysics & cosmology LPP/Polytechnique DMPH/ONERA CosmoStat/CEA IAS/UPSud AIM/CEA LAL/UPSud 250researchers in 35laboratories Machine learning LRI/UPSud LTCI/Telecom CMLA/Cachan LS/ENSAE LIX/Polytechnique MIA/Agro CMA/Polytechnique LSS/Supélec CVN/Centrale LMAS/Centrale DTIM/ONERA IBISC/UEvry Visualization INRIA LIMSI Signal processing LTCI/Telecom CMA/Polytechnique CVN/Centrale LSS/Supélec CMLA/Cachan LIMSI DTIM/ONERA Statistics LMO/UPSud LS/ENSAE LSS/Supélec CMA/Polytechnique LMAS/Centrale MIA/AgroParisTech machine learning information retrieval signal processing data visualization databases Domain science human society life brain earth universe Tool building software engineering clouds/grids high-performance computing optimization Domain scientistSoftware engineer datascience-paris-saclay.fr LIST/CEA
- 38. 38 THE DATA SCIENCE LANDSCAPE Domain science energy and physical sciences health and life sciences Earth and environment economy and society brain Data scientist Data trainer Applied scientist Domain scientistSoftware engineer Data engineer Data science statistics machine learning information retrieval signal processing data visualization databases Tool building software engineering clouds/grids high-performance computing optimization
- 39. Center for Data Science Paris-Saclay39 https://medium.com/@balazskegl
- 40. Center for Data Science Paris-Saclay TOOLS: LANDSCAPE TO ECOSYSTEM 40 Data scientist Data trainer Applied scientist Domain expertSoftware engineer Data engineer Tool building Data domains Data science statistics machine learning information retrieval signal processing data visualization databases • interdisciplinary projects • matchmaking tool • design and innovation strategy workshops • data challenges • coding sprints • Open Software Initiative • code consolidator and engineering projects software engineering clouds/grids high-performance computing optimization energy and physical sciences health and life sciences Earth and environment economy and society brain • data science RAMPs and TSs • IT platform for linked data • annotation tools • SaaS data science platform
- 41. Center for Data Science Paris-Saclay • Modularizing the collaboration • independent subtasks • reduces barriers • broadens the range of available expertise • Encouraging small contributions • Rich and well-structured information commons • so people can build on earlier work 41 NIELSEN’S CROWDSOURCING PRINCIPLES
- 42. Center for Data Science Paris-Saclay42 RAMPS • Single-day coding sessions • 20-40 participants • preparation is similar to challenges • Goals • focusing and motivating top talents • promoting collaboration, speed, and efficiency • solving (prototyping) real problems
- 43. 43 TRAINING SPRINTS • Single-day training sessions • 20-40 participants • focusing on a single subject (deep learning, model tuning, functional data, etc.) • preparing RAMPs
- 44. 44 ANALYTICS TOOLS TO PROMOTE COLLABORATION AND CODE REUSE
- 45. Center for Data Science Paris-Saclay45 ANALYTICS TOOL TO PROMOTE COLLABORATION AND CODE REUSE
- 46. Center for Data Science Paris-Saclay ANALYTICS TOOLS TO MONITOR PROGRESS 46
- 47. Center for Data Science Paris-Saclay RAPID ANALYTICS AND MODEL PROTOTYPING 2015 Jan 15 The HiggsML challenge 47
- 48. Center for Data Science Paris-Saclay RAPID ANALYTICS AND MODEL PROTOTYPING 2015 Apr 10 Classifying variable stars 48
- 49. Center for Data Science Paris-Saclay VARIABLE STARS 49
- 50. Learning to discoverB. Kégl / CNRS - Saclay VARIABLE STARS 50 accuracy improvement: 89% to 96%
- 51. Center for Data Science Paris-Saclay RAPID ANALYTICS AND MODEL PROTOTYPING 2015 June 16 and Sept 26 Predicting El Nino 51
- 52. 52 RAPID ANALYTICS AND MODEL PROTOTYPING RMSE improvement: 0.9˚C to 0.4˚C
- 53. 53 2015 October 8 Insect classification RAPID ANALYTICS AND MODEL PROTOTYPING
- 54. 54 RAPID ANALYTICS AND MODEL PROTOTYPING accuracy improvement: 30% to 70%
- 55. 55 CONCLUSIONS • Explore the open innovation space • read Nielsen’s book • Drop me a mail (balazs.kegl@gmail.com) if you are interested in beta-testing the RAMP tool • Come to our CIML WS tomorrow
- 56. Center for Data Science Paris-Saclay56 THANK YOU!