Problem of majority voting
0 likes105 views
The document discusses agreement levels among raters when evaluating comments. It shows that with 3 raters, 100% agreement is achieved, but with 5 raters only 40.1% of comments reach at least 66.7% agreement, and increasing the number of raters to 7 and 9 only modestly increases the percentage of comments with over 66.7% agreement to 43.5% and 44.9%, respectively.
1 of 13
Download to read offline
Problem of majority voting
- 9. According to the majority rule, with three raters, 100% of the comments have at least 66.7% agreement (2/3). Our data shows that when the same comments are evaluated by five raters, 40.1% (n=117) of the comments have at least 66.7% agreement. With seven raters, 43.5% (n=127) have at least 66.7% agreement, and for nine raters, the value is 44.9% (n=131). 0 10 20 30 40 50 60 70 80 90 100 Three Five Seven Nine Percentage At least 66.7% agreement
- 13. Thanks! Dr. Joni Salminen jsalminen@hbku.edu.qa Qatar Computing Research Institute