Reproducibility in machine learning
Download as PPTX, PDF2 likes291 views
This document discusses reproducibility in machine learning experiments and provides a checklist to improve reproducibility. It contains the following key points in 3 sentences: The document introduces the topic of reproducibility in machine learning and discusses the importance of making machine learning experiment results more reproducible. It then provides and explains in detail the "Machine Learning Reproducibility Checklist" created by Joelle Pineau, which contains steps researchers should take to clearly describe their models, algorithms, data, hyperparameters and results to enable other researchers to understand and replicate their work. The checklist aims to improve reproducibility by ensuring researchers provide all necessary information and details to allow other to understand, evaluate and build upon their findings.
Reproducibility in machine learning
- 1. Reproducibility in machine learning 1 Payam Barnaghi Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey Care Technology & Research Centre, The UK Dementia Research Institute
- 2. 2
- 3. 3
- 4. 4 Source: Odd Erik Gundersen, the Norwegian University of Science and Technology in Trondheim, Science Magazine
- 5. 5
- 6. 6
- 7. What is machine learning? How can we make the results of machine learning experiments more reproducible? 7
- 8. 8 Leia: Yaté. Yaté. Yotó. (SUBTITLE: “I have come for the bounty on this Wookiee.”) C-3PO relays this message and Jabba says he’ll offer 25,000 for Chewie. Leia: Yotó. Yotó. (SUBTITLE: “50,000, no less.”) C-3PO relays this message and Jabba asks why he should pay so much. Leia: Eí yóto. The above isn’t subtitled, but Leia pulls out a bomb and activates it. c-3po: Because he’s holding a thermal detonator! Jabba is impressed by this and offers 35,000. Leia: Yató cha. The above isn’t subtitled, but Leia deactivates the bomb and puts it away. c-3po: He agrees. Order is restored. Adapted from: The Art of Language Invention: From Horse-Lords to Dark Elves, The Words Behind World-Building, by
- 9. 9 Leia: Yaté. Yaté. Yotó. (SUBTITLE: “I have come for the bounty on this Wookiee.”) C-3PO relays this message and Jabba says he’ll offer 25,000 for Chewie. Leia: Yotó. Yotó. (SUBTITLE: “50,000, no less.”) C-3PO relays this message and Jabba asks why he should pay so much. Leia: Eí yóto. The above isn’t subtitled, but Leia pulls out a bomb and activates it. c-3po: Because he’s holding a thermal detonator! Jabba is impressed by this and offers 35,000. Leia: Yató cha. The above isn’t subtitled, but Leia deactivates the bomb and puts it away. c-3po: He agrees. Order is restored. Adapted from: The Art of Language Invention: From Horse-Lords to Dark Elves, The Words Behind World-Building, by
- 10. 10
- 12. However, don’t underestimate the human brain − For example, in German: −you have to distinguish between “the” and “a” articles −and then each one has four case forms −three genders −and singular and plural forms −and then the adjectives have to agree −and then there are verbs! −or, take this as an example: 12 Adapted from: The Art of Language Invention: From Horse-Lords to Dark Elves, The Words Behind World-Building, by (biáng)
- 13. Peterson 13 / (ˈpiːtəsən) / Pear Peter/ Peter’s Son Bear Meter Ear Greta … …
- 14. Payam 14 / (ˈpæːjʌm) / Pay + am or Pa + yam پیام
- 15. 15 Source: https://www.originofalphabet.com/meow-is-just-another-name-for- cat-2/
- 16. 16 / (ˈpiːtəsən) / / (ˈpiːtəsən) / P ˈp e i … … n n
- 17. 17 “It is a capital mistake to theorise before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” Arthur Conan Doyle, Sherlock Holmes
- 18. 18
- 19. 19
- 20. 20
- 21. 21
- 22. 22
- 23. The Machine Learning Reproducibility Checklist (Version 1.2, Mar.27 2019)- Joelle Pineau −For all models and algorithms presented, check if you include: −A clear description of the mathematical setting, algorithm, and/or model. −An analysis of the complexity (time, space, sample size) of any algorithm. −A link to a downloadable source code, with specification of all dependencies, including external libraries. 23 Reproduced from: www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf
- 24. The Machine Learning Reproducibility Checklist − For any theoretical claim, check if you include: −A statement of the result. −A clear explanation of any assumptions. −A complete proof of the claim. 24 Reproduced from: www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf
- 25. The Machine Learning Reproducibility Checklist − For all figures and tables that present empirical results, check if you include: − A complete description of the data collection process, including sample size. − A link to a downloadable version of the dataset or simulation environment. − An explanation of any data that were excluded, description of any pre-processing step. − An explanation of how samples were allocated for training/ validation/ testing. 25 Reproduced from: www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf
- 26. The Machine Learning Reproducibility Checklist − The range of hyper-parameters considered, method to select the best hyper-parameter configuration, and specification of all hyper- parameters used to generate results. − The exact number of evaluation runs. − A description of how experiments were run. − A clear definition of the specific measure or statistics used to report results. − Clearly defined error bars. − A description of results with central tendency (e.g. mean) & variation(e.g. stddev). − A description of the computing infrastructure used. 26 Reproduced from: www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf
- 28. Reproducibility in ML ”The point of reproducibility isn’t to replicate the results exactly. That would be nearly impossible given the natural randomness in neural networks and variations in hardware and code. Instead, the idea is to offer a road map to reach the same conclusions as the original research, especially when that involves deciding which machine-learning system is best for a particular task.” Jesse Dodge, Carnegie Mellon University 28 Reproduced from: https://www.wired.com/story/artificial-intelligence-confronts- reproducibility-crisis/
- 29. 29
- 30. Thank you. 30