A data science observatory based on RAMP - rapid analytics and model prototyping
2 likes704 views
The document discusses the challenges in data science and design, highlighting issues of in-breeding and the need for innovative approaches. It introduces 'RAMP,' a collaborative development platform aimed at enhancing data science modeling and evaluation through shared submissions and performance metrics. Additionally, it explores applications across various fields like climatology, astrophysics, and ecology, emphasizing improvements in prediction accuracy through collaborative efforts.
A data science observatory based on RAMP - rapid analytics and model prototyping
- 1. A data science observatory Akin Kazakci, Mines ParisTech Balazs Kégl, CNRS
- 2. Team Balázs Kégl CNRS Alexandre Gramfort Télécom ParisTech Akın Kazakçı Mines ParisTech Camille Marini Télécom ParisTech Mehdi Cherti UP Saclay Yohann Sitruk Mines ParisTech Djalel Benbouzid UPMC
- 3. 1 The research objective & questions
- 4. Enough with the chairs • Design research is falling behind in dealing with contemporary challenges • Claim 1a: Too much in-breeding and repetition • Claim 1b: Huge amount of work is based on ideas from 80s’ • Design is not about objects, but about reasoning
- 5. - Physics (Particle physics, Plasma physics, astrophysics…) - Biology (Genetics, Epidemiology…) - Chemistry - Economics, Finance, Banking - Manufacturing, Industrial Internet - Internet of things, Connected Devices - Social media - Transport & Mobility - … There is not enough data scientists to handle this much data Revealing the potential of data: what role for design?
- 6. Last year: Crowdsourcing data challenges 1785 teams Kazakci, A., Data science as a new frontier for design, ICED’15, Milan Reasonable doubts about the effectiveness of data science contests
- 7. Crowdsourcing /?/ Design crowd /kraʊd/ noun 1.a large number of people gathered together in a disorganized or unruly way 1. How to study the design process of a crowd? 2. How to manage the design process of a crowd?
- 8. SeekerSolvers ? Crowdsourcing: C-K dynamics Crowdsourced contests Crowdsourced collaboration
- 9. Analysis of design strategies Achieve 5σ Dicovery condition: A discovery is claimed when we find a ‘region’ of the space where there is significant excess of ‘signal’ events. (rejecting background-only hypothesis with a p value less than 2,9 x 10-7, corresponding to 5 Sigma). Problem formulation: Traditional classification setting: « the task of the participants is to train a classifier g based on the training data D with the goal of maximizing the AMS (7) on a held-out (test) data set » (HiggsML documentation) With 2 tweaks: - Training set events are « weighted » - Maximize « Approximate Median Significance »: Select a classification method Pre-processing Choose hyper-params Train Optimize for X SVM Decision Trees NN…..….. Performance metrics: During the overall learning process performance metrics are used to supervise the quality and convergence of a learned model. A traditional metric is accuracy: where Note that for HiggsML AMS, TP (s) and FP (b) are of particular importance. Boosting Bagging others Ensemble Methods (Extended) Dominant Design Traditional workflow = Dominant design C space K Space Fixating others… Achieve 5σ Select a classification method Pre-processing Choose hyper-params Train Optimize for accuracy SVM Decision Trees NN…..….. Integrate AMS directly in training during Gradient Boosting (John) Dicovery condition: A discovery is claimed when we … Problem formulation: Traditional classification setting… Cross-Validation: Techniques for evaluating how a … Ensemble Methods during node split in random forest (John) Weighted Classification Cascades ? ? ? ? ? Optimization of AMS Design for statistical efficiency The biggest challenge is the unstability of AMS. Competition results clearly show that only participants who dealt effectively with this issue have had higher ranks. 1st 2nd 3rd Ensembles + CV monitoring + cutoff threshold seem to be a winning strategy monitoring progress with CV + ensembles + selecting a cutoff threshold that optimise (or stabilise AMS) Public guide to AMS 3.6 « moves » many participants to the given path Fixation vs. Creative Authority (Agogué et al, 2014) Generating new design strategies Data science as a new frontier for design A. Kazakci, ICED’15 (submitted) • Available data for HiggsML - Forums ➔ 136 topics, 1400+ posts - Documentation - Participants’ blog entries - GitHub codes • Qualitative interpretation combined with C-K modelling of participants’ strategies Data challenges are hard to analyze
- 10. How do you put a crowd under a microscope?
- 12. RAMP - Rapid Analytics and Model Prototyping A Collaborative Development Platform for Data Science Instant access to all submitted code - for participants & organizers
- 13. RAMP allows us to collect data on the data science model development process 1 A Collaborative Development Platform for Data Science 2 3 We prepare a ‘starting kit’ Continuous access to code: Organizers can follow real-time what’s happening - and react Participants can analyse and build on every submission Submissions are trained and performances are displayed 4 Users actions and interactions are recorded 5 Main Output: Dozens of predictive models and performance benchmark RAMP - Rapid Analytics and Model Prototyping
- 14. Collecting data with RAMP - Number of submissions - Frequency of submissions - Timing of submissions - User interactions - Performance of submissions - Submitted code - … We are interested in - the variety (code space + prediction space) - the mutual influences and inheritance (code space) - score and delta score (impact) - …
- 15. 3 Some applications, preliminary observations & findings
- 16. Climatology Time Series Based Event Prediction on Geo-tagged data Two workshops: Improvement in RMSE score: from 0.90 to 0.43 El Nino Prediction - Temperature data - Geo-tagged time series - Prediction: 6 months ahead George Washington University George Mason University
- 17. Astrophysics Classification of variable stars One day workshop: Accuracy improvement: %89 to %96 Light curves (luminosity vs time profiles) - Static features - Functional data Marc Monier (LAL), Gilles Faÿ (Centrale-Supelec)
- 18. Ecology Finding & Classifying Insects One day event: Improvement in prediction accuracy: from 0.31 to 0.70 from Image Data Pollenating Insects - Image data (20K images) - 18 types of insects - Deep neural net models Paris Museum of Natural History, SPIPOLL.org, NVDIA, Université de Champagne- Ardenne ROMEO HPC Center
- 19. Cancer Research
- 20. A graph of model similarities - Steady progression. They have built systematically on a submission they previously created, without being influenced by the others. Their performance may go either up (constantly) or down (constantly). - Breakdowns or jumps. There are other groups, where the performance increased or decreased strongly from one modification submission to the next). There may be some robustness/vulnerability issue with their approach - to be further investigated. - Successful expansions. An important “break” has happened at 12:00. This corresponds to “cropping” idea. Strangely, two very similar submissions (small distance) have been submitted at the same time - one of them did not improve the score at all (around 0.35, while the leader was around 0.55), whereas the other improved considerably (0.65). - Currently, we see no dependency between this break and the winning solution. This might be related to the way we have measured the code similarity. Some observations
- 21. “Note that, following the RAMP approach, this model is the result of a succession of small improvements made on top of other participants’ contributions. We did not reach a prediction score of 0.71 in one shot, but after applying several tricks and manually tuning some parameters.” Heuritech, Winner of Insect Challenge Blog entry
- 22. How to compare design concepts - represented as code? ~ ?
- 24. Comparing performance profiles: Promoting novelty search - Greyness: Model’s raw score - Size: model’s contribution - Position: similarity/dissimilarity in predictions 2D projection (MDS) of model’s prediction profiles
- 25. • Monitoring & Modelling “contribution” (Pierre Fleckinger, Economic Agents & Incentive Theory) • Pushing towards “Novelty Search” (Jean-Bastiste Mouret, Novelty Search) • Controlled experiments We Found (to be validated by further studies): In progress: • Gravitation: following a given submission, others are hovering around the same coordinates, by incremental adjustments • Repulsion: new submission using out-of-the-box code to explore the white space (no previous close-by submissions exist) • Hybridation: opportunistic integration of previous submissions, involving/inspired by at least two different source of code. RAMP platform • RAMP platform is meant to be a free tool for researchers and students; this opens up new perspectives (pedagogy & research) and hopefully brings closer different communities
- 26. Akin Kazakci, Mines ParisTech akin.kazakci@mines-paristech.fr Thank you