OpenML Reproducibility in Machine Learning ICML2017
2 likes254 views
The document discusses OpenML, a platform designed to enhance reproducibility and collaboration in machine learning. It highlights challenges in data accessibility, code usage, and experimentation, proposing solutions such as easy online sharing of datasets and integration with various tools. OpenML aims to facilitate real-time collaborative analysis, streamline experimentation, and provide a structured reward system for contributions, fostering a community-driven approach to machine learning.
OpenML Reproducibility in Machine Learning ICML2017
- 1. OpenML M A K I N G M A C H I N E L E A R N I N G M O R E R E P R O D U C I B L E ( A N D E A S I E R ) B Y B R I N G I N G I T O N L I N E J O A Q U I N VA N S C H O R E N R E P R O D U C I B I L I T Y I N M A C H I N E L E A R N I N G , I C M L 2 0 1 7 @open_ml
- 2. Research different. Polymaths: Solve math problems by massive online collaboration Broadcast question, combine many minds to solve it
- 3. Networked Science Designed serendipity: what’s hard for one person is easy for another Collaboration only scales if all friction is eliminated
- 4. Friction in machine learning • Data hard to find, in different formats • Code hard to find, difficult to use • Published experiments are often not reproducible • Reproducing my own results can be difficult • No easy overview of state-of-the-art • Benchmarks are often incomparable • Experimentation is mostly solitary, offline • Only afterwards, we share aggregated results • Limited interaction, visibility in early stage • Fear or getting scooped
- 5. Make machine learning as simple as possible (but not simpler) Use any (open-source) tool to try many algorithms, Share code and experiments online Find richly annotated datasets, import easily. Or share your own. Clear problem descriptions, evaluation protocols, overview of state of the art. Reproducible, transparent, reusable results Organized for easy analysis and reuse Frictionless online collaboration, track contributions Easy sharing of data and results
- 6. W H AT I F W E C A N A N A LY S E D ATA C O L L A B O R AT I V E LY
- 7. W H AT I F W E C A N A N A LY S E D ATA C O L L A B O R AT I V E LY O N W E B S C A L E
- 8. W H AT I F W E C A N A N A LY S E D ATA C O L L A B O R AT I V E LY O N W E B S C A L E I N R E A L T I M E
- 9. C O L L A B O R AT I V E M A C H I N E L E A R N I N G Easy to use: Integrated in many ML tools/environments Modular contribution: Share data, code, models, evaluations Organized data: Meta-data, reproducible models, link to people Reward structure: Track your impact, build reputation Self-learning: Learn from many experiments to help people OpenML
- 10. It starts with data
- 11. Data (ARFF) uploaded or referenced (URL) auto-versioned, analysed, meta-data extracted, organised online It starts with data
- 12. auto-versioned, analysed, meta-data extracted, organised online
- 13. Tasks contain data, goals, procedures. Readable by tools, automates experimentation Train-test splits Classify target X All results organized online: realtime overview Collaborate online
- 14. Train-test splits All results organized online: realtime overview Classify target X
- 15. Flows (workflows, scripts) can run anywhere (locally) Tool integrations + APIs (REST, R, Python, Java,…) Use any ML library
- 16. Integrations + APIs (REST, R, Python, Java,…)
- 17. Integrations + APIs (REST, R, Python, Java,…) Run wherever you want
- 18. reproducible, linked to data, flows, authors and all other experiments Experiments auto-uploaded, evaluated online Analyse results objectively
- 19. Experiments auto-uploaded, evaluated online
- 20. Publish, and track your impact
- 21. Demo
- 22. OpenML Community Nov-Dec 2016 3000+ registered, 2500 30-day active users
- 23. Studies Online collaboration environment Link to GitHub, Jupyter notebooks, wiki, Overleaf,… Sharing settings Share datasets, flows, studies with certain people. Easily publish them later. Code submissions, GitHub integration Sharing versioned code, docker images, archiving In progress… Benchmarking suites Curated datasets + all reproducible results
- 24. Meta-learning, learning to learn Learn from all prior experiments -> bots Learn across datasets, use OpenML as ‘memory’ Deep learning integration Native integration in deep learning libraries Store/analyse models (architecture, parameters) In progress (all help welcome!) Daily research Support other machine learning tasks, tools Support ML in other scientific domains
- 26. Join us! Open initiative, contribute via GitHub/Slack Regular week-long hackathons Bring your own data, tools Resources, ideas welcome
- 27. Thank You @open_ml OpenML Now try it yourself :) www.openml.org