Lung-Hao Lee - 2015 - Overview of the NLP-TEA 2015 Shared Task for Chinese Grammatical Error Diagnosis
0 likes71 views
This document provides an overview of the NLP-TEA 2015 Shared Task for Chinese Grammatical Error Diagnosis. The shared task aimed to develop tools for identifying grammatical errors in Chinese sentences. Participants were provided a training set of annotated sentences and asked to classify errors in a test set. Errors involved misordering, redundancy, omission, and misselection. The overview describes the data, task, evaluation metrics, participants, and results, concluding that while submissions were helpful, developing an effective grammatical diagnosis tool remains challenging.
Lung-Hao Lee - 2015 - Overview of the NLP-TEA 2015 Shared Task for Chinese Grammatical Error Diagnosis
- 1. Overview of the NLP-‐TEA 2015 Shared Task for Chinese Gramma@cal Error Diagnosis Lung-‐Hao Lee, Na%onal Taiwan Normal University Liang-‐Chih Yu, Yuan Ze University Li-‐Ping Chang, Na%onal Taiwan Normal University
- 2. Introduc%on • The NLP-‐TEA 2015 shared task features a Chinese Gramma%cal Error Diagnosis (CGED) task, providing an evalua%on plaMorm for the development and implementa%on of NLP tools for computer-‐assisted Chinese learning NLP-‐TEA 2015 @ Beijing, China 2
- 3. Shared Task Descrip%on • The developed tool is expected to iden%fy the error types and its posi%on at which it occurs in the sentence • Four PADS error types are included in the target modifica%on taxonomy – Mis-‐Ordering (Permuta%on) – Redundancy (Addi%on) – Omission (Dele%on) – Mis-‐Selec%on (Subs%tu%on) • For the sake of simplicity, the input sentence is selected to contain one defined error types NLP-‐TEA 2015 @ Beijing, China 3
- 4. Tes%ng Examples • Example 1 Input: (sid=B2-‐0080) 他是我的以前的室友 Output: B2-‐0080, 4, 4, Redundant • Example 2 Input: (sid=A2-‐0017) 那電影是機器人的故事 Output: A2-‐0017, 2, 2, Missing • Example 3 Input: (sid=A2-‐0017) 那部電影是機器人的故事 Output: A2-‐0017, Correct • Example 4 Input: (sid=B1-‐1193) 吳先生是修理腳踏車的拿手 Output: B1-‐1193, 11, 12, Selec%on • Example 5 Input: (sid=B2-‐2292) 所以我不會讓失望她 Output: B2-‐2292, 7, 9, Disorder NLP-‐TEA 2015 @ Beijing, China 4
- 5. Data Prepara%on • The essay sec%on of the computer-‐based Test of Chinese as a Foreign Language (TOCFL) • Na%ve Chinese speakers were trained to manually annotate gramma%cal errors and provide correc%ons corresponding to each error NLP-‐TEA 2015 @ Beijing, China 5
- 6. Training Set • This set included 2,205 selected sentences • Error types were categorized as redundant (430 instances), missing (620), selec%on (849), and disorder (306) • Each sentence is represented in SGML format NLP-‐TEA 2015 @ Beijing, China 6
- 7. Dryrun Set • A total of 55 sentences were given to par%cipants to familiarize themselves with the final tes%ng process. • The purpose is output format valida%on only • No macer which performance can be achieved that will not be included in our official evalua%on. NLP-‐TEA 2015 @ Beijing, China 7
- 8. Test Set • This set consists of 1,000 tes%ng sentences • Half of these sentences contained no gramma%cal errors, while the other half included a single defined gramma%cal error: redundant (132 instances), missing (126), selec%on (110), and disorder (132) NLP-‐TEA 2015 @ Beijing, China 8
- 9. Performance Metrics • Correctness is determined at three levels – Detec%on-‐level – Iden%fica%on-‐level – Posi%on-‐level • Metrics – False posi%ve rate (FPR) = FP / (FP+TP) – Accuracy = (TP+TN) / (TP+FP+TN+FN) – Precision = TP / (TP+FP) – Recall = TP / (TP+FN) – F1 = 2 * Precision * Recall / (Precision+Recall) NLP-‐TEA 2015 @ Beijing, China 9
- 10. Evalua%on Examples • FPR = 0.5 • Detec%on-‐level Acc. = 0.75, Pre.=0.67, Rec.=1, F1=0.8 • Correc%on-‐level Acc. = 0.625, Pre.=0.5, Rec.=0.75, F1=0.6 • Posi%on-‐level Acc. = 0.5, Pre.=0.33, Rec.=0.5, F1=0.4 • System Results “B1-‐1138, 7, 8, Disorder”, “A2-‐0087, 12, 13, Missing”, “A2-‐0904, 5, 6, Missing”, “B1-‐0990, correct”, “A2-‐0789, 2, 5, Disorder”, “B1-‐0295, correct”, “B2-‐0591, 3, 3, Redundant” and “A2-‐0920, 4, 5, Selec%on” • Gold Standard “B1-‐1138, 7, 10, Disorder”, “A2-‐0087, 12, 13, Missing”, “A2-‐0904, correct”, “B1-‐0990, correct”, “A2-‐0789, 2, 3, Selec%on”, “B1-‐0295, correct”, “B2-‐0591, 3, 3, Redundant” and “A2-‐0920, correct” NLP-‐TEA 2015 @ Beijing, China 10
- 11. 13 Par%cipants and 18 Submiced Runs NLP-‐TEA 2015 @ Beijing, China 11
- 12. Tes%ng Results NLP-‐TEA 2015 @ Beijing, China 12
- 13. Summary • It is a really difficult task to develop the computer-‐assisted Chinese learning tool, since there are only target sentences without the help of their context • None of the submiced systems provided superior performance. In general, this research problem s%ll has long way to go. NLP-‐TEA 2015 @ Beijing, China 13
- 14. Conclusions • All submissions contribute to the common effort to produce an effec%ve Chinese gramma%cal diagnosis tool • The individual reports in the shared task proceedings provide useful insight into Chinese language processing NLP-‐TEA 2015 @ Beijing, China 14
- 15. Future Work • NLP-‐TEA-‐3 Workshop in COLING 2016 – To be bided – Osaka, Japan • The Shared Task – Chinese Gramma%cal Error Diagnosis NLP-‐TEA 2015 @ Beijing, China 15
- 16. Acknowledgments • Ministry of Educa%on, Taiwan – Aim for the Top University Project – Center of Learning Technology for Chinese, NTNU – Innova%ve Center for Big Data and Digital Convergence, YZU • Ministry of Science and Technology, Taiwan – Interna%onal Research-‐Intensive Center of Excellence Program – Grant no.: MOST 104-‐2911-‐I-‐003-‐301 SIGHAN 2015 @ Beijing, China 16
- 17. THANK YOU • All data sets with gold standards and evalua%on tool are publicly available for research purposes at hcp://ir.itc.ntnu.edu.tw/lre/nlptea15cged.htm NLP-‐TEA 2015 @ Beijing, China 17