Error analysis of Word Sense Disambiguation
Download as PPTX, PDF2 likes916 views
The document discusses the challenges of word sense disambiguation (WSD), identifying that it is not a single problem but multiple issues related to context modeling and reliance on the most frequent sense. Through error analysis of various WSD evaluations, the authors highlight common errors, including those from monosemous words and part-of-speech tagging biases. The findings indicate that the difficulties in word disambiguation are largely influenced by polysemy and context exploitation.
Error analysis of Word Sense Disambiguation
- 1. Error analysis ofWord Sense Disambiguation Ruben Izquierdo Marten Postma PiekVossen Izquierdo,PostmaandVossen VUAmsterdam
- 2. Motivation Word Sense Disambiguation is still an unsolved problem 2 Izquierdo, Postma and Vossen VU Amsterdam
- 3. Error Analysis Perform error analysis on previousWSD evaluations to prove our hypothesis Senseval-2: all-words task Senseval-3: all-words task Semeval2007: all-words task (#17) Semeval2010: all-words on specific domain (#17) Semeval2013: multilingual all-wordsWSD and entity linking (#12) 3 Izquierdo, Postma and Vossen VU Amsterdam
- 4. Motivation Some “propagated” errors Errors on monosemous Errors because pos-tags Multiwords and phrasal verbs Little attention has been paid to the real problem WSD is not 1 problem but N problems Our hypothesis Context is not modeled properly in general System rely too much on the most frequent sense 4 Izquierdo, Postma and Vossen VU Amsterdam
- 5. Monosemous errors 5 Izquierdo, Postma and Vossen VU Amsterdam
- 6. Monosemous errors 6 Izquierdo, Postma and Vossen VU Amsterdam Competition Monosemous Wrong Examples Senseval2 499 (20.9%) 37.5% gene.n (suppressor_gene.n), chance.a (chance.n) next.r (next.a) Senseval3 334 (16.6%) 44.1% Datum.n (data.n) making.n (make.v) out_of_sight (sight) Semeval2007 25 (5.5%) 11.1% get_stuck.v, lack.v, write_about.v Semeval2010 31 (2.2%) 97.9% Tidal_zone.n pine_marten.n roe_deer.n cordgrass.n Semeval2013 (lemmas) 348 (21.1%) 1.9% Private_enterprise, developing_country, narrow_margin
- 7. Most Frequent Sense 7 Izquierdo, Postma and Vossen VU Amsterdam
- 8. Most Frequent Sense When the correct sense is NOT the most frequent sense Systems still assign mostly the MFS Senseval2 799 tokens are not MFS 84% systems still assign the MFS Most “failed” words due to MFS bias Senseval2, senseval3 Say.v find.v take.v have.v cell.n church.n Semeval2010 Area.n nature.n connection.n water.n population.n 8 Izquierdo, Postma and Vossen VU Amsterdam
- 9. Analysis per PoS-tag 9 Izquierdo, Postma and Vossen VU Amsterdam
- 10. Analysis per polysemy class 10 Izquierdo, Postma and Vossen VU Amsterdam 2Senses Poly. C. 6 15 Low Medium High
- 11. Analysis per frequency class 11 Izquierdo, Postma and Vossen VU Amsterdam
- 12. Most difficult words 12 Izquierdo, Postma and Vossen VU Amsterdam
- 13. Expected vs. Observed difficulties Calculate per sentence The “expected” difficulty Average polysemy, sentence length, average word length 13 Izquierdo, Postma and Vossen VU Amsterdam
- 14. Calculate per sentence The “expected” difficulty Average polysemy, sentence length, average word length 14 Izquierdo, Postma and Vossen VU Amsterdam Expected vs. Observed difficulties
- 15. Calculate per sentence The “expected” difficulty Average polysemy, sentence length, average wor length The “observed” difficulty From the real participant outputs, average error rate We should expect: harder sentences higher error rate easier sentences lower error rate 15 Izquierdo, Postma and Vossen VU Amsterdam Expected vs. Observed difficulties
- 16. 16 Izquierdo, Postma and Vossen VU Amsterdam Expected vs. Observed difficulties
- 17. 17 Izquierdo, Postma and Vossen VU Amsterdam Expected vs. Observed difficulties
- 18. • The context is not (probably) exploited properly • Expected “easy” sentences SHOULD show low error rates • Occurrences of the same word in different contexts have similar error rate • The difficulty of a word depends more on its polysemy than on the context where it appears 18 Izquierdo, Postma and Vossen VU Amsterdam Expected vs. Observed difficulties
- 19. WSD Corpora http://github.com/rubenIzquierdo/wsd_corpora 19 Izquierdo, Postma and Vossen VU Amsterdam
- 20. WSD Corpora 20 Izquierdo, Postma and Vossen VU Amsterdam
- 21. System Outputs https://github.com/rubenIzquierdo/sval_systems 21 Izquierdo, Postma and Vossen VU Amsterdam
- 22. System Outputs 22 Izquierdo, Postma and Vossen VU Amsterdam
- 23. Error analysis of Word Sense Disambiguation Ruben Izquierdo Marten Postma PiekVossen ruben.izquierdobevia@vu.nl http://github.com/rubenIzquierdo/wsd_corpora http://github.com/rubenIzquierdo/sval_systems 23
- 24. Analysis per PoS-tag 24 Izquierdo, Postma and Vossen VU Amsterdam
Editor's Notes
- #12: Relative freq (norvig method) <0.01 low 0.01 -= 0.05 medium > 0.05 high