Human-Centric Machine Learning
1 like548 views
This document summarizes Matthias Gallé's presentation on human-centric machine learning at the 2017 Rakuten Technology Conference. It discusses supervised learning where the goal is formalized and large labeled datasets are available, unsupervised learning where patterns are discovered in unlabeled data, and interactive learning which allows users to provide feedback to guide the machine learning model. Specific applications mentioned include machine translation, exploratory search, active learning, and removing the burden of labeling from humans.
![Example: Active Learning
Give initiative to the algorithm
allow action of type: “please, label instance x”
Cognitive effort of labeling a document 3-5x higher than labelling a word [1]
Feature labelling:
• type(feedback) ≠ type(label)
• information load of a word label is small
• word sense disambiguation
[1] Raghavan, Hema, Omid Madani, and Rosie Jones. "Active learning with feedback on features and instances." Journal of
Machine Learning Research7.Aug (2006): 1655-1686.](https://image.slidesharecdn.com/matthiasgallertcfinal-171114074028/85/Human-Centric-Machine-Learning-12-320.jpg)
![Conclusion
If you really want to solve a problem, don’t be prisoner of your
performance indicator
Ask yourself:
1. Does it really capture success?
does it align with human judgment?
2. What does the [machine | human] best?
3. Can you remove the burden from humans by smarter algorithms?](https://image.slidesharecdn.com/matthiasgallertcfinal-171114074028/85/Human-Centric-Machine-Learning-13-320.jpg)
Human-Centric Machine Learning
- 1. © 2017 NAVER LABS. All rights reserved. Matthias Gallé Naver Labs Europe @mgalle Human-Centric Machine Learning Rakuten Technology Conference 2017
- 3. Supervised Learning Where f typically such that 𝑓 = argmin 𝑓∈𝐹 1 𝑁 𝑖=1 𝐿 𝑓 𝑥𝑖 , 𝑦𝑖 + 𝜆𝑅 𝑓 I know what I want (and can formalize it) I have time & money to label lots of data X,Y f(x)
- 4. Example: Machine Translation Given a text s and its proposed translation p, how to measure its distance with respect to a reference translation t ? BLEU: n-gram overlap between t and p typically: 1 ≤ 𝑛 ≤ 4, precision only, brevity penalty METEOR bonus points for matching stems and synonyms use paraphrases
- 5. Statistical Machine Translation P Koehn (www.statmt.org/book/slides/08- evaluation.pdf)
- 6. Consequences of not formalizing correctly Users do not use your model Computer-Assisted Translation used rule-based systems for years Ad-hoc solutions Quality Prediction Automatic Post Edition
- 7. Unsupervised Learning Where Z(X) capture some prior: • Compression • Clustering • Coverage • …. I am not sure what I want I have a (big) corpus with assumed patterns X Z(X)
- 8. Example: Exploratory Search Whenever your task is: • Ill-defined: – Broad / under-specified – Multi-faceted • Dynamic: – Searcher’s understanding inadequate at the beginning – Searcher’s understanding evolves as results are gradually retrieved. The answer to what you search is “I know it when I see it”
- 11. Exploratory Search: examples E-Discovery Sensitivity Review • Vo, Ngoc Phuoc An, et al. "DISCO: A System Leveraging Semantic Search in Document Review." COLING (Demos). 2016. • Privault, Caroline, et al. "A new tangible user interface for machine learning document review." Artificial Intelligence and Law 18.4 (2010): 459-479. • Ferrero, Germán, Audi Primadhanty, and Ariadna Quattoni. "InToEventS: An Interactive Toolkit for Discovering and Building Event Schemas." EACL 2017 (2017): 104.
- 12. Example: Active Learning Give initiative to the algorithm allow action of type: “please, label instance x” Cognitive effort of labeling a document 3-5x higher than labelling a word [1] Feature labelling: • type(feedback) ≠ type(label) • information load of a word label is small • word sense disambiguation [1] Raghavan, Hema, Omid Madani, and Rosie Jones. "Active learning with feedback on features and instances." Journal of Machine Learning Research7.Aug (2006): 1655-1686.
- 13. Conclusion If you really want to solve a problem, don’t be prisoner of your performance indicator Ask yourself: 1. Does it really capture success? does it align with human judgment? 2. What does the [machine | human] best? 3. Can you remove the burden from humans by smarter algorithms?
- 14. Further reading & Acknowledgments Jean-Michel RendersMarc Dymetman Ariadna Quattoni http://www.europe.naverlabs.com/Blog
- 15. Q&A © 2017 NAVER LABS. All rights reserved.
- 16. Appendix © 2017 NAVER LABS. All rights reserved.
- 17. Statistical Machine Translation P Koehn (www.statmt.org/book/slides/08- evaluation.pdf)