A Web-Based Collaborative Reading Annotation System With Gamification Mechanisms To Improve Reading Performance

Contents lists available at ScienceDirect
Computers & Education
journal homepage: www.elsevier.com/locate/compedu
A web-based collaborative reading annotation system with
gamification mechanisms to improve reading performance
Chih-Ming Chena,∗
, Ming-Chaun Lia
, Tze-Chun Chenb
a
Graduate Institute of Library, Information and Archival Studies, National Chengchi University, No. 64, Section 2, ZhiNan Road, Wenshan District,
Taipei City 116, Taiwan, ROC
b
E-learning Master Program of Library and Information Studies, National Chengchi University, No. 64, Section 2, ZhiNan Road, Wenshan District,
Taipei City 116, Taiwan, ROC
A R T I C L E I N F O
Keywords:
Cooperative/collaborative learning
Learning environments
Teaching/learning strategies
A B S T R A C T
This study presents a web-based collaborative reading annotation system (WCRAS) with gami-
fication mechanisms to motivate students' annotations behaviors and promote students' reading
comprehension performance. The research participants were 55 fifth grade students from two
classes of an elementary school in northeastern Taiwan. A quasi-experimental design was
adopted to evaluate the effects of the experimental and control groups respectively using WCRAS
with and without gamification mechanisms to support digital reading on students' annotation
behaviors, collaborative interaction relationship, reading comprehension performance, and im-
mersion experience. The results showed that the experimental group made significantly more
annotations across almost all types of reading annotations and response annotations and had a
significantly higher degree of immersive experience and social interaction than the control group.
However, the difference in reading comprehension performance between the two groups has not
been found. In addition, although the experimental group provided more amounts of high-quality
annotations, the group did not generate more quantity of annotations related to reading com-
prehension aspects assessed by PIRLS. These research findings indicate that a gap between the
promotion of the quantity and quality of participation and learning outcomes exists. This study is
significant to address the need that designs the gamification mechanisms to encourage high-
quality annotations that can effectively enhance students’ reading comprehension. To better
associate game features and learning performance in the WCRAS with gamification mechanisms,
future studies are suggested to promote both annotation quality and connection between reading
achievement and gamification mechanisms.
1. Introduction
With the advance of information and communication technologies (ICT) in supporting reading, digital reading has emerged
rapidly and led to a change in human reading behaviors (Liu, 2005). Digital reading is not only a change of reading media from
printed text to digital text but also a new way of reading. By using digital devices, social collaboration and mobility of reading can be
better conducted than the use of traditional printed media (Cardoso, Ganito, & Ferreira, 2012).
https://doi.org/10.1016/j.compedu.2019.103697
Received 19 March 2019; Received in revised form 1 September 2019; Accepted 9 September 2019
∗
Corresponding author.
E-mail addresses: chencm@nccu.edu.tw (C.-M. Chen), mli1tw@gmail.com (M.-C. Li), vanessajim@msn.com (T.-C. Chen).
Computers & Education 144 (2020) 103697
Available online 13 September 2019
0360-1315/ © 2019 Elsevier Ltd. All rights reserved.
T

1.1. Collaborative annotating and reading
Many social annotation tools called web-based collaborative reading annotation system (WCRAS) in this study have been suc-
cessfully developed to support digital reading (Novak, Razzouk, & Johnson, 2012), specifically for collaborative digital reading. Kiili,
Laurinen, Marttunen, and Leu (2012) described collaborative reading as a socially contextualized learning activity that readers co-
construct meaning and knowledge through text-based interaction. According to social constructivist theory, knowledge is constructed
in a social context and emergence from the discourse (Prawat & Floden, 1994). Generally, social annotation tools were designed to
provide a social constructivist learning environment for collaborative digital reading. These tools not only support learners’ anno-
tating and highlighting activities that are usually adopted to perform in-depth readings in printed media, more importantly, they also
allow learners to collaboratively make annotations for sharing and discussing them with each other (Liu, 2005; Novak et al., 2012).
The features of social annotation tools would benefit learners to get more engaged in reading text (Mendenhall & Johnson, 2010). As
stated by Novak et al. (2012), by using social annotation tools, a new level of knowledge can be fostered through the aggregation and
discussion of the adding annotations on the digital texts made by many readers.
The advantages of using social annotation tools to support reading have been investigated by many studies. For example, Gao
(2013) investigated students' participation and interaction behaviors of using a social annotation tool, Diigo, to collaboratively read
an online text. The results showed that students participated actively in collaborative learning and engaged in various learning
behaviors including self-reflecting, elaborating, internalizing, and showing support to other students. Social annotation tool was also
employed to facilitate critical thinking and reading comprehension. Mendenhall and Johnson (2010) reported three studies using the
social annotation tool called HyLighter and concluded that students perceived the annotations were helpful for peer critique activity,
students who worked collaboratively in small groups could perform deeper thinking by interacting with peers, and students' critical
thinking skills were observed to be promoted. In addition, Yang, Yu, and Sun (2013) compared the effects of collaborative annotation
versus personal digital reading on primary school students' Chinese reading levels. Their study found that students who used Sharing
Unique Reading Feeling (SURF) to collaboratively make annotation significantly achieved better reading performance and higher
Chinese reading levels than personal digital reading. Moreover, their higher-level cognitive abilities (e.g., analyzing, summarizing,
and evaluating) could also be facilitated. When compared to making annotation individually, students who made collaborative
annotations and shared them with peers would perform better in reading comprehension, meta-cognitive skills (Johnson, Archibald,
& Tenenbaum, 2010), and learning outcomes (Hwang, Wang, & Sharples, 2007). The comparisons of using social annotation tools to
other forms of collaborative learning (e.g., traditional paper reading and other discussion tools) also showed favored outcomes on
students’ reading comprehension performance (Chen & Chen, 2014; Razon, Turner, Johnson, Arsal, & Tenenbaum, 2012; Su, Yang,
Hwang, & Zhang, 2010; Yang, Zhang, Su, & Tsai, 2011), use of reading strategies (Chen & Chen, 2014), and knowledge sharing (Su
et al., 2010; Yang et al., 2011).
Drawn from previous research, social annotation tools help students think and review carefully to understand peers' annotations
(Su et al., 2010). The interaction during collaborative annotating would also facilitate students' thinking and therefore recall the
reading better as well as their learning of basic comprehension and meta-cognition (Johnson et al., 2010). Because of the increased
engagement in social annotation learning, students' level of reading comprehension was promoted. Moreover, students’ deep reading
was also promoted through the interactions during collaborative annotating that their high-order capabilities of reading and cog-
nition could be developed (Mendenhall & Johnson, 2010; Yang et al., 2013). The higher level of interaction with the text by using
social annotation tools also helps students keep on task and become more engaged in the text (Mendenhall & Johnson, 2010). With
positive attitude and satisfaction of using social annotation tools, students would engage more in the reading and annotating.
Students’ learning achievements were also found to be positively correlated with annotation quantity made by the students
(Hwang et al., 2007; Su et al., 2010). Namely, the more the annotations a student made, the better the learning performance would be
achieved. However, it was found that students might become less motive to make annotations when they could easily retrieve
annotations provided by other students (Hwang et al., 2007). This is similar to the free-rider problem (Kim & Walker, 1984).
Therefore, how to effectively encourage students to continuously make rich and qualified annotations in digital reading with the
support of collaborative annotation tools would be an important issue.
1.2. Gamification and learning
In recent years, gamification has been increasingly employed to promote learning motivation, engagement, collaboration, and
effectiveness (Dichev & Dicheva, 2017; Dicheva, Dichev, Agre, & Angelova, 2015). Not like stand-alone educational video games,
gamification is “the use of game design elements in non-game context” (Deterding, Dixon, Khaled, & Nacke, 2011). Different game
mechanisms (e.g., points, badges, levels, and leaderboards) can be integrated into general learning contexts or platforms to make
learning more interesting and motivating. The effectiveness of gamification mechanisms in enhancing collaborative learning has been
examined by several studies. For example, Ding, Er, and Orey (2018) found gamification design would positively provoke students'
behavioral, emotional, and cognitive engagements in online discussion. In addition, most of the students participated in Ding et al.‘s
study stated that the gamification mechanism would motivate them to make more efforts in contributing posts both in higher quantity
and better quality. In other comparative studies, the gamification design was found to significantly enhance students' engagement in
collaborative learning. Students in the gamified learning environments would contribute more posts and reply more in online dis-
cussion forums as well as be more proactive to initiate discussion threads when compared to the non-gamified learning groups
(Barata, Gama, Jorge, & Gonçalves, 2013; Hew, Huang, Chu, & Chiu, 2016; Smith, Herbert, Kavanagh, & Reidsema, 2013). The
learning performance could also be facilitated by gamified learning environments. In Brown and Cairns’s (2004) study, students' final
C.-M. Chen, et al. Computers & Education 144 (2020) 103697
2

grades were found to be significantly improved after adopting gamification mechanisms. Besides, in a study conducted by Hew et al.
(2016), the majority of the students reported that they were motivated by the gamified course to try more challenging tasks. Students
in the gamified group also produced higher quality of artifacts than students in the non-gamified group. Moreover, Barata et al.
(2013) found a positive strong correlation between the number of students' posts and their final grades in the courses that employed
gamification design. According to the abovementioned studies, students' learning engagement and achievement can be fostered by
gamified learning context.
Although several WCRASs have been well developed and implemented (Estellés Arolas, Del Moral-Pérez, & González, 2010; Chen
& Chen, 2014; Estellés; Hwang et al., 2007; Johnson et al., 2010; Su et al., 2010), gamification design has not been applied in these
systems. Therefore, this study attempted to develop gamification mechanisms for the developed WCRAS to promote students’ an-
notation behaviors and reading comprehension performance in collaborative digital reading scenario.
1.3. Immersion experience and learning
Immersion experience was also investigated in the present study. While flow is generally used to describe the players' psycho-
logical status during game playing, many researchers have proposed that “immersion” is more appropriate to be used to describe the
degree of involvement in a learning activity (Cheng, She, & Annetta, 2015; Jennett et al., 2008). According to Brown and Cairns
(2004), immersion was identified as three stages including engagement, engrossment, and total immersion successively. The im-
mersion experience and its impacts on learning have been investigated in several studies (Cheng et al., 2015; Cheng, Lin, She, & Kuo,
2017; Hamari et al., 2016). Cheng et al. (2015) found that engagement is significantly correlated to both game performance and
science learning performance in a game-based learning study. In Cheng, Lin, She, and Kuo's (2017) study, they found that students
who had more prior knowledge would remain a certain degree of engagement in game-based learning and their engagement would
determine their learning outcomes. To investigate students' degree of involvement in a WCRAS with gamification mechanisms for
digital reading activity, the immersion experience was therefore examined.
1.4. Social interaction in collaborative reading
The WCRAS was developed to promote collaborative reading among students. However, how students interact with each other
when collaboratively making reading annotations was scarcely studied and limited to qualitative analysis (Gao, 2013). Social net-
works analysis, a method to study information exchange and interaction relationships among actors (Haythornthwaite, 1996) has
been adopted to investigate students' interaction patterns and engagement in computer-supported collaborative learning (CSCL) (de-
Marcos et al., 2016; Laat, Lally, Lipponen, & Simons, 2007; Rabbany, Elatia, Takaffoli, & Zaïane, 2014). For example, de-Marcos et al.
(2016) conducted a social networks analysis to visualize interaction networks of the students in a gamified course and explore the
relations of students' positions in the learning networks and their learning achievement. As stated by Palonen and Hakkarainen
(2000), social networks analysis could provide new information about the patterns and structure of students' interaction in colla-
borative learning that are relatively difficult to be uncovered by other analysis methods. To explore whether or not students would
interact differently when they use gamified and non-gamified WCRAS, social networks analysis was employed herein to explore
students’ interactions in collaborative annotating.
1.5. Research purposes and questions
In summary, the research purpose of this study is to examine the effectiveness of the gamified WCRAS on promoting students'
annotation behaviors, reading comprehension performance, and their degree of immersion as well as to explore the students' in-
teraction patterns in making collaborative annotations. Moreover, the quality of students' annotations and students’ opinions of using
gamified WCRAS are also investigated. The research questions of this study are as follows:
1. What were the effects of gamification mechanisms on students' annotation behaviors and their annotation quality in the colla-
borative reading activity?
2. What were the effects of gamification mechanisms on students' reading performance?
3. What were the effects of gamification mechanisms on students' immersion experience in the collaborative reading activity?
4. What were the effects of gamification mechanisms on students' social interaction in the collaborative reading activity?
5. What were the students' opinions about using gamified WCRAS?
2. Research methodology
2.1. Research participants
A total of 55 fifth grade students from two classes of an elementary school in the northeastern Taiwan participated in this study.
One class with 28 students (15 males and 13 females) was randomly assigned as the experimental group using the WCRAS with
gamification mechanisms to support digital reading while the other class with 27 students (14 males and 13 females) was assigned as
the control group using the WCRAS without gamification mechanisms. The students' midterm scores of Chinese language course in
school were adopted as their prior Chinese reading abilities; the average scores of the experimental and control groups were 91.07
3

(SD = 6.73) and 88.74 (SD = 6.68), respectively. There was no statistically significant difference between the two groups in students’
Chinese reading abilities (t = 1.29, p = .203 > 0 .05), confirming that the two groups have equivalent prior Chinese reading abil-
ities.
2.2. Research design and procedure
Based on a quasi-experimental design, the students in both groups respectively used the WCRAS with and without gamification
mechanisms to read a PIRLS article and made annotations together with their classmates. All of the students who participated in this
study were well informed about the research purpose and process in advance. After that, a 40-min instruction of the WCRAS was
provided to the students of both groups. The gamification mechanisms were additionally introduced to the experimental group during
this system instruction. After the instruction, the students in both groups were respectively given 100 min to perform a reading task
by making annotations collaboratively and sharing their ideas and comments with their classmates through the annotations. The
experiments of both groups were conducted using the class time that two consecutive class sessions were arranged for each group to
complete the reading task. The students in both groups read the whole article “An unbelievable night” and made annotations col-
laboratively for 100 min. The same annotation scaffoldings (i.e., annotation types of reading and responding annotations) were
provided to the students of both groups in the WCRAS. In the experimental group, gamification mechanisms were employed to
promote students’ annotation behaviors and guide their reading in increasingly depth by designing tasks requiring the use of different
annotation types from basic to advance. Although the certain types and quantity of annotations or the number of “hearts” getting
from others were required for level progression, students in the experimental group had the choice to use other annotation types and
made more quantity of annotations freely and they could decide whom to give the “heart.” As for the students in the control group,
they were free to make annotations with any annotation types and amounts as well as interact with each other without concerning
how many “hearts” they got from or gave to their classmates during the whole experimental session. During the whole experimental
session, students did self-learning with their peers without interference from the teacher. The teacher would only give assistance
when the students encountered technical problems or reminded the students when they made improper wordings in the annotations
and deleted it. All the annotating behaviors and annotation contents performed and generated by the students were recorded by the
system. After students completed the reading task, both groups took a reading comprehension test and filled out the WCRAS im-
mersion experience questionnaire immediately. Besides the test and the questionnaires, a semi-structured interview was conducted
within three days after the experiment.
2.3. Functions of the web-based collaborative reading annotation system (WCRAS)
2.3.1. Annotation functions of WCRAS
The WCRAS was developed to support digital reading by allowing students to read articles and make annotations collaboratively.
In addition, students can give responses to others' annotations to share and discuss their ideas about the reading article. To fulfill
these two purposes, two kinds of annotations are provided in the WCRAS: reading annotation and response annotation. Reading
annotations allow students to annotate and share their own ideas about the reading article and response annotations allow students to
give feedback to other students' annotations. In addition, reading and response annotation scaffolds are provided to help students
make appropriate annotations and guide them to read and discuss the reading article. The reading and response annotation types
were initially designed by an experienced Chinese language teacher based on her teaching experience in reading to support students
to read and understand the reading article and to discuss with others. The annotation types were designed to provide basic to
advanced scaffoldings. The basic annotations can help students elicit their prior knowledge (“I know”) and identify their new
knowledge (“New knowledge”) and questions (“Don't understand”) about the reading article. The advanced annotation types
“Different ideas,” “Additional information,” and “I want to say” can help students read and think of the reading article in more in-
depth and discuss with others. These annotation types were then validated by ten in-service elementary teachers who have five to
twenty years of Chinese language teaching experience. The descriptions of the reading and response annotation types are listed in
Table 1.
Table 1
Descriptions of reading and response annotation types.
Annotation type Reading Annotation Response Annotation
I know Provide understanding or known facts of an annotated text Give answers to the questions raised in other students' annotations
New knowledge Identify new knowledge learned from an annotated text Identify new knowledge learned from other students' annotations
Don't understand Indicate an annotated text that does not understand Indicate other students' annotations that do not understand
Different ideas Indicate the text that is different from what I think, and give
reasons
Indicate other students' annotations that are different from what I
think, and give reasons
Additional Information Provide supplementary information for an annotated text
by using an online search tool in WCRAS
Provide supplementary information for other students' annotations by
using an online search tool in WCRAS
I want to say Give comments to an annotated text and invite other
students to discuss their ideas
Respond to other students' comments or discussion of an annotated
text
Correction – Remind other students to correct their problematic annotations or
inappropriate use of the annotation types or wordings
4

To make a reading annotation, a student has to first select target texts, and then chooses an annotation type and writes down
annotation contents. A brief description is shown on the side of the chosen annotation type (Fig. 1(a)) to help students use proper
annotation types. For the annotation contents, students can type text, insert pictures or videos, provide webpage links, or use em-
bedded Google search tool to find additional information for the annotated text (Fig. 1(b)). Response annotations are used when
Fig. 1. An example of making a reading annotation in the WCRAS: (a) choose the text and select an annotation type; (b) add annotation content.
Fig. 2. An example of making a response annotation.
5

students want to give feedback to other students’ reading or response annotations. After students choose an annotation that they
would like to reply, the way to make a response annotation is the same as making a reading annotation (Fig. 2). Besides response
annotations, students can also click the “heart” icon to show their agreement or favor toward a reading or response annotation made
by other students. When students move mouse cursor over the text with annotations, they can see how many annotations have been
made for the text and click to read the contents of those reading and response annotations.
2.3.2. Gamification mechanisms of WCRAS
The gamification mechanisms of WCRAS were designed by the researchers of this study for two instructional purposes: one is to
motivate students' contributions of annotations and the other one is to guide students to read and comprehend the reading article. To
satisfy these two purposes, two popular gamification mechanisms (Dicheva et al., 2015), level and leaderboard, were employed to
gamify the WCRAS. Level is used to present the progress of one's achievement that motivates one to accomplish the tasks (e.g., Barata
et al., 2013; Mekler, Brühlmann, Tuch, & Opwis, 2017). Different task-related benefits can also be applied to different level that
motivates one to level-up (Goehle, 2013). A leaderboard is adopted to motivate one to make efforts on assigned tasks by providing a
reference for one to compare with others' performance (e.g., Ding et al., 2018; Domínguez et al., 2013; Hew et al., 2016; Richter,
Raban, & Rafaeli, 2015). The design of these two mechanisms to correspond to the instructional purposes of this study is described as
follows.
For the first purpose, different degrees of interactivity are arranged for different levels. Students have to make the required
quantity of reading and response annotations of each level so that they can read and respond to other students' annotations. Five
levels are set and corresponding roles are assigned to provide students a challenging and role-playing scenario. The roles are re-
spectively soldier, knight, bishop, castellan, and king from level 1 to level 5. Different features of annotation making will be unlocked
for each level to increase students’ interaction with each other. In the beginning level (level 1), students as soldiers can only make
their own annotations on the texts but not allowed to interact with others. The full features of annotation making and complete
interaction mode are not available until students become castellans (level 4). Students have to make contributions to reading and
response annotations as well as to receive responses from other students so that they can read complete annotations and responses
made by their classmates. To get responses from others depends on the content of the annotation one make that is decided by others
whether they respond or not. To make the last level more challenging and set the role of king more reasonable (i.e., there should be to
set a limited number of kings), only the top three students who contribute the most annotations can level up to king. Therefore, it is
possible to level down from king to castellan if students stop making annotations. To motivate students promoting the levels of roles,
a leaderboard mechanism is also adopted for students to compare their level statuses with their classmates.
For the second instructional purpose, the level-up tasks are arranged by requesting students to use different types of annotations.
In the beginning, students are asked to use basic annotation types such as “I know” and “Don't understand.” With the promotion of the
game levels, students are requested to apply more advanced annotation types in making annotations that would help them read and
think of the reading article and discuss with others. To attain the previously stated instructional purposes, the level achievement tasks
are assigned with the combination of different requirements of annotation quantities, annotation types, annotation responses and the
use of “heart” icon. The achievement tasks and the permission of annotation features for each level are listed in Table 2.
Besides level and leaderboard mechanisms, several gamification design principles are also adopted in this study including
achievement, visual status, clear goals, and feedback (Dicheva et al., 2015). Achievement is employed in level mechanisms that
students have to accomplish required tasks to level up. Clear goals and visual status are presented to students on a gamification board
Table 2
Achievement tasks and annotation feature permissions of each level.
Level Annotation Permission The task to Level Up
Soldier (Level 1) • Make personal reading annotations • Make 6 annotations on the text and use different annotation types for each annotation
Knight (Level 2) • Make personal reading annotations
• Make response annotations • Make 7 annotations on the text (including 4 “I know” and 3 “Don't understand”
annotations)
• Read peers' reading annotations • Respond to annotations of 3 other students
Bishop (Level 3) • Make personal reading annotations • Make 8 annotations on the text (4 “New knowledge” and 4 “Additional Information”
annotations)
• Make response annotations • Respond to annotations of 6 other students
• Read peers' reading annotations • Get responses from 3 other students
• Read peers' response annotations
Castellan (Level 4) • Make personal reading annotations
• Make response annotations • Make 8 annotations on the text (5 “I want to say” and 3 “Different ideas”)
• Read peers' reading annotations • Use 3 “Correction” annotations to respond to other students' annotations
• Read peers' response annotations • Give “heart” to other students' annotations
• Use the “heart” icon • The total number of annotations reaches the top three in the class
King (Level 5) • Make personal reading annotations
• Make response annotations • The total number of annotations contributed maintains the top three in the class
• Read peers' reading annotations
• Read peers' response annotations
• Use the “heart” icon
6

as shown in Figs. 3 and 4. As shown in Fig. 3, students can review their annotation records on the left side of the interface, and they
can see the assigned tasks for leveling up from the right side of the interface. Students can also know the levels they already achieved
and the current level of every student in the class (see Fig. 4). The last principle is to give students text feedback of promotion when
they achieve a level requirement (e.g., You are now the King!).
2.4. Research tools
2.4.1. Reading material and reading comprehension assessment
The reading article and reading comprehension assessment used in this study were chosen from The Progress in International
Reading Literacy Study (PIRLS). Each PIRLS article has its own comprehension question type and scoring guide. In this study, the
reading article was one of the selected articles of PIRLS 2006, “An unbelievable night.” All research participants did not read this
article before participating in the present study. In PIRLS reading assessment, two aspects of comprehension that include four
comprehension processes are evaluated (Mullis, Martin, Kennedy, & Foy, 2007). The first aspect is retrieving and straightforward
inferencing that includes the processes of retrieving explicitly stated information and making straightforward inferences from reading
passages. This aspect is referred to as direct and explicit comprehension in this study. The second aspect is interpreting, integrating,
and evaluating that includes the processes of interpreting and integrating ideas and information, and examining and evaluating
content, language, and textual elements in reading passages. The second aspect is referred to as inferential comprehension in this
study.
2.4.2. WCRAS immersion experience questionnaire
To investigate the differences of immersion experience between the students who used the WCRAS with and without gamification
mechanisms, the Game Immersion Questionnaire (GIQ) developed by Cheng et al. (2015) was revised in corresponding to the use of
WCRAS in this study. This revised questionnaire consists of 20 items of three stages of immersion experience including engagement (9
items), engrossment (7 items), and total immersion (4 items) successively according to the degree of immersion. Responses of the
Fig. 3. An example of a student's personal status of annotating task in the WCRAS with gamification mechanisms.
Fig. 4. An example of a class leaderboard in the WCRAS with gamification mechanisms.
7

items were made on a 5-point Likert scale. The Cronbach's α values for the overall questionnaire and the subscales of “engagement,”
“engrossment,” and “total immersion” were 0.94, 0.90, 0.92, and 0.84, respectively. This indicates good reliability of this revised
immersion experience questionnaire.
2.4.3. Log file
The annotation behaviors of the students were recorded directly into the log file database of the developed WCRAS. When
students made annotations, the target of annotations (i.e., the text being annotated or annotations to give responses), annotation
types, and annotation contents were all recorded. The behavior of using “heart” icon was also saved in the log file. The correctness
and completeness of the records were examined and verified using a testing task before the system was implemented.
2.4.4. Semi-structured interview
Students' comments about the use and the design of the gamification mechanism to promote their reading comprehension and
interaction were collected from both groups. The outlines of the semi-structured interview were designed by the researchers of this
study in accordance with the quantitative research questions to gather more in-depth qualitative evidences in order to discuss the
quantitative results. In order to compare students' perceptions about using WCRAS with and without gamification mechanisms,
students selected from the control group were asked to utilize the WCRAS with gamification mechanisms for 10 min before they were
interviewed. The interview data were recorded and transcribed by the researchers. The students’ comments were then classified in
order to discuss their annotations and interaction behaviors, reading comprehension, and immersion experience.
3. Experimental results
3.1. Analysis of annotation behaviors and annotation quality in both groups
3.1.1. Analysis of annotation behaviors in both groups
Table 3 shows the descriptive statistics of students' annotation behaviors. In total, the experimental group made 1268 annotations
including 899 (70.9%) reading annotations and 369 (29.1%) response annotations; the control group made 593 annotations in-
cluding 537 (90.6%) reading annotations and 56 (9.4%) response annotations. The total number of annotations made by the ex-
perimental group was more than twice of the quantity made by the control group; in the control group, students made the majority of
their annotations on the text but far less on responding others’ annotations; in the experimental group, students not only made more
reading and response annotations, the proportion of response annotations was also more than three times of the proportion of the
control group.
An independent samples t-test was further conducted to investigate the effect of gamified WCRAS on students' annotation be-
haviors in terms of annotation quantities of overall and different types of reading and response annotations. The results shown in
Table 4 indicate that the students in the experimental group made significantly more reading, response, and total annotations than
those in the control group. As for different types of reading and response annotations, the students in the experimental group also
provided significantly more annotations in almost all types. But no significant differences were found in “I know” and “additional
information” of reading annotation. The results indicate that the WCRAS with gamification mechanisms could not only positively
promote students’ total annotating behaviors but also facilitate them to make more reading and response annotations across almost
all types.
Table 3
Annotation frequencies and percentages of the two groups.
Annotation Type Control Group (n = 27) Experimental Group (n = 28)
Frequency Percentage Frequency Percentage
Reading Annotation
I known 214 39.9% 182 20.2%
New knowledge 40 7.4% 107 11.9%
Don't understand 77 14.3% 128 14.2%
Different ideas 18 3.4% 78 8.7%
Additional Information 135 25.1% 257 28.6%
I want to say 53 9.9% 147 16.4%
Total 537 100.0% 899 100.0%
Response Annotation
I known 15 26.8% 127 34.4%
New knowledge 0 0.0% 18 4.9%
Don't understand 2 3.6% 30 8.1%
Different ideas 3 5.4% 25 6.8%
Additional Information 0 0.0% 19 5.1%
I want to say 26 46.4% 75 20.3%
Correction 10 17.9% 75 20.3%
Total 56 100.0% 369 100.0%
8

3.1.2. Analysis of annotation quality in both groups
To evaluate the annotation quality made by the students, a qualitative content analysis (Boettger & Palmer, 2010) was conducted.
Students’ annotations were coded using the coding category presented in Table 5. The unit of analysis is “an annotation.” One of the
researchers of this study performed the coding process mainly. To ensure inter-rater reliability, about one-fifth of the total anno-
tations (375 out of 1861) of both group were randomly selected and coded both by the researcher and a research assistant who
assisted to conduct the experiment. The inter-rater Kappa reliability was 0.88 (p < .001) between the two coders, showing a high
consistency. Because this value is larger than 0.7, the codes of the annotations are reliable (Gwet, 2012). The coding of the rest
annotations was then completed by the researcher.
The analysis results of students' annotation quality are presented in Table 6. The pictures that were not text-related, as well as
unrelated text annotation that did not provide useful information for students to comprehend the reading article according to the four
comprehension processes evaluated in PIRLS reading assessment, were identified as low-quality annotations. To compare the an-
notation quality of the experimental and control groups, Person's chi-square tests were performed to examine the relationship be-
tween the groups and their annotation quality. The results showed that there were significant associations between the groups
(experimental group vs. control group) and different quality of annotations. The control group used more picture annotations while
the experimental group provided more text annotations (χ2
= 149.41, p = .000 < 0.05). The result also showed that the experi-
mental group made more high-quality text annotations than the control group (χ2
= 16.99, p = .000 < 0.05). However, there was
no significant association between the two groups (non-gamified group and gamified group) and different types of high-quality
Table 4
Group differences of annotation types between the control and experimental groups.
Annotation Type Control Group (n = 27) Experimental Group (n = 28)
Mean SD Mean SD t p
Reading Annotation
I known 7.93 9.45 6.50 3.00 0.75 .460
New knowledge 1.48 1.78 3.82 1.19 −5.75*** .000
Don't understand 2.85 2.05 4.57 1.64 −3.44** .001
Different ideas 0.67 0.96 2.79 1.42 −6.45*** .000
Additional Information 5.00 6.45 9.18 9.28 −1.93 .059
I want to say 1.96 4.23 5.25 4.38 −2.83** .007
Total 19.89 10.92 32.11 12.33 −3.89*** .000
Response Annotation
I known 0.56 1.05 4.54 3.27 −6.12*** .000
New knowledge .00 0.00 .64 1.13 −3.01** .006
Don't understand .07 0.27 1.07 1.98 −2.64* .013
Different ideas .11 0.32 .89 1.81 −2.25* .033
Additional Information .00 0.00 .68 1.42 −2.54* .017
I want to say 0.96 2.36 2.68 3.31 −2.22* .031
Correction 0.37 0.79 2.68 1.96 −5.75*** .000
Total 2.07 2.66 13.18 6.46 −8.39*** .000
Overall Annotation
Total 21.96 11.35 45.29 16.36 −6.16*** .000
*p < .05; **p < .01; ***p < .001.
Table 5
The coding category of the annotation content.
Category of Annotation Content Description
Pictures
Text-related One or more pictures that corresponded to the selected text. For example, a crocodile picture.
Not Text-related One or more pictures that did not relate to the selected text. For example, a logo of a website that has no meaning about the
selected text
Text
Unrelated information The information provided did not help students understand the selected text or comprehend the article. For example,
“Thanks to everyone's opinions”
Comprehension related
Word explanation The explanation of the selected text either provided by the students or copied from the online dictionaries or webpages. For
example, “Representing very slow observation.”
Retrieval State explicit information from the selected text. For example, “It turns out that the flamingo is an animal in the tropical
forest.”
Straightforward inferencing Make straightforward inferences from the selected text. For example, “I think he will not leave if giving him food.”
Interpreting and integrating Interpret and integrate ideas and information from the selected text. For example, “The magazine showed Anina where the
crocodile came from.”
Examining and evaluating Examine and evaluate the selected text. For example, “I think it should be Anina wetting pants or dreaming, so her parents
don't believe her.”
9

annotations (i.e., word explanation, retrieval, straightforward inferencing, Interpreting and integrating, and examining and evalu-
ating) (χ2
= 4.95, p = .18 > 0.05). These findings suggest that the gamification mechanisms designed in this study did promote
students to generate more related text annotations, but cannot guarantee that high-quality and useful annotations help promote their
reading comprehension assessed in this study.
3.2. Analysis of reading comprehension performance in both groups
To examine the effect of WCRAS with gamification mechanisms on promoting students' reading comprehension, analysis of
covariance (ANCOVA) was adopted to compare the reading performance of both groups. Students' prior reading ability was re-
presented by their Chinese language midterm scores as stated previously and used as the covariate in the analysis. Table 7 shows the
results. After controlling the effect of students’ prior reading ability, there were no significant differences in the reading performance
between both groups across all aspects of the reading comprehension and the overall performance. Both groups were above the
median score and got almost full scores in straightforward inferencing performance. This result indicates that whether using WCRAS
with or without gamification mechanisms, students could achieve well in their performances of reading comprehension to some
degree.
3.3. Analysis of immersion experience in both groups
The differences of the immersion experience between the two groups when using the WCRAS with and without gamification
mechanisms were compared by independent samples t-test. The results in Table 8 show that significant difference between both
groups was only found in the first-stage immersion experience “engagement” (t = −2.40; p = .022 < 0.05). This indicates that the
students who used the WCRAS with gamification mechanisms were more willing to invest their time and effort in making, sharing,
and discussing annotations than those in the control group. As for the immersive experiences in the second stage (engrossment) and
the third stage (total immersion), the two groups did not reach statistically significant differences. Nevertheless, the means of im-
mersion experience of the two groups in the second and third stages are all higher than the median of a 5-point Likert scale
(median = 3). This suggests that the students in the control group using the WCRAS without gamification mechanisms were still as
highly immersive as those in the experimental group.
Table 6
The annotation content coding results of the two groups.
Category of Annotation Content Control Group (n = 27) Experimental Group (n = 28)
Frequency Percentage Frequency Percentage
Total 593 100.0% 1268 100.0%
Pictures 245 41.30% 196 15.50%
Text-related 169 28.50% 161 12.70%
Not Text-related 76 12.80% 35 2.80%
Text 348 58.70% 1072 84.50%
Unrelated information 87 14.70% 164 12.90%
Comprehension related
Word explanation 109 18.40% 312 24.60%
Retrieval 61 10.30% 231 18.20%
Straightforward inferencing 87 14.70% 347 27.40%
Interpreting and integrating 0 0% 0 0%
Examining and evaluating 4 0.70% 18 1.40%
Table 7
ANCOVA results of reading comprehension performance.
Reading Comprehension PIRLS Comprehension Assessment
Control Group (n = 27) Experimental Group (n = 28)
Adjusted Mean SD Adjusted Mean SD F p
Overall reading comprehension 11.57 2.91 11.70 2.40 0.045 .832
Direct and explicit comprehension performance 7.47 1.55 7.37 1.23 0.072 .789
Retrieval 2.88 0.46 2.76 0.50 0.984 .326
Straightforward inferencing 4.59 1.25 4.61 1.10 0.005 .944
Inferential comprehension performance 4.10 1.86 4.33 1.71 0.327 .570
Interpreting and integrating 3.09 1.32 3.16 1.18 0.062 .805
Examining and evaluating 1.01 0.94 1.17 0.83 0.501 .482
10

3.4. Analysis of social interaction in both groups
To explore the students' interactions when they used WCRAS with and without gamification mechanisms, the social networks
analysis software, UCINET, was employed to analyze and visualize students' interactions in both groups. The interaction graphs of the
two groups are presented in Figs. 5 and 6, respectively. As shown in Figs. 5 and 6, the links among students in the experimental group
were significantly more than those in the control group. By calculating the ratio of the actual number of links to the maximum
possible number of links in the network, the network density of the experimental group was up to 0.55 while the control group was
0.13. The higher the density of a network, the more the students were connected to each other (Scott, 2000). Therefore, the result
indicates that students in the experimental group had more interactions with each other than those in the control group. In addition,
it can be observed in Fig. 5 that students in the experimental group all interacted more or less with each other as well as no one was
isolated from the network. In contrast, students in the control group did not interact much with other peers and near one-fifth of them
were isolated from the main interaction network. Three students (#11, #16, and #19) did not interact with other peers at all and two
students (#12 and #24) interacted with each other only. According to the results of social networks analysis, the WCRAS with
gamification mechanisms could promote students' willingness to give responses to others’ annotations and interact with more stu-
dents in the class, thus reducing the free-rider problem.
3.5. Interview results
In order to explore the students’ experiences and comments about using gamified WCRAS, 12 students from the experimental
group were randomly selected and interviewed after the experiment. In addition, 6 students from the control group were also
randomly selected and interviewed to gain an in-depth understanding of their opinions on the comparison between gamified and non-
gamified WCRAS. To help the students from the control group better understand the differences between gamified and non-gamified
WCRAS, each student was given 10 min to operate the gamified WCRAS before the interview. The results of the interview about
whether gamification mechanisms would promote collaborative annotation and reading were analyzed according to the following
aspects: (1) the annotation and interaction behaviors; (2) reading comprehension; and (3) immersion experiences.
(1) The annotation and interaction behaviors
Table 8
Differences of immersion experience between the two groups.
Control Group (n = 27) Experimental Group (n = 28)
Mean SD Mean SD t p
Stage1-Engagement 4.31 0.76 4.70 0.35 −2.40* .022
Stage2-Engrossment 3.53 1.01 3.89 0.97 −1.34 .185
Stage3-Total immersion 4.05 1.01 4.29 0.78 -.99 .329
*p < .05.
Fig. 5. Visualizations of students interactions in the control group.
11

According to the interview results, the design of levels to permit different degrees of interactivity did promote students' anno-
tation quantity. Most of the students agreed with that gamified WCRAS would motivate them to make more annotations because they
want to level up so that they could read annotations made by other students, discuss with others, and give “heart” to affirm other
students’ good annotations. Some students said that they made more annotations and interacted with others mainly because they
desired to challenge the levels. Most of the students also indicated that leaderboard would increase their motivation for making
annotation and interacting with others because they cared about their statuses in the class and want to gain recognition from other
students.
As for the quality of the annotations, most of the respondents indicated that they would improve the annotation quality because
they needed their annotations to be liked by their classmates to reach and remain at the highest level “King.” Some students said that
they would increase the quality of annotations because they could gain a sense of accomplishment if their annotations were liked by
their classmates. However, one student mentioned that some annotations even without high quality could still get “heart” (i.e., being
liked) from other students.
(2) Reading comprehension
According to the interview results, all the students agreed with that the annotation types can provide useful scaffoldings for
promoting reading comprehension. Some students specifically pointed out that the basic annotations types allowed them to know to
what degree they already understand the words and the article before further reading. Some other students said that the annotation
types helped them understand the annotations provided by other students better as well as enhanced their understanding of the
article or knew whether other students asked questions or responded to their annotations.
All the students also indicated that the design of using basic annotation types to advanced annotation types with the progress of
the levels could help them read the article with increasing understanding in-depth. However, students’ views are more divergent as to
whether gamification mechanisms could help improve their reading comprehension and assessment performance. Some students
stated that the increasing difficulties of the level tasks in gamified WCRAS were helpful for understanding the article and the reading
comprehension assessment. Some students felt no difference between gamified and non-gamified WCRAS to promote reading
comprehension. And one student said that using WCRAS without gamification mechanisms would allow him to concentrate on
reading and understanding the article because he would not always think about leveling up.
(3) Immersion experiences
According to the interview results, all students agreed with that the gamified WCRAS would make them engage more in the
collaborative annotation reading activity.
In summary, the gamification mechanisms designed for this study were viewed as useful for the interviewed students to promote
their annotation and interaction behaviors and increase their willingness to generate higher quality annotations. Moreover, anno-
tation scaffoldings were also valued in enhancing students’ understanding of the article. All students expressed a high immersion
experience of using gamified WCRAS.
Fig. 6. Visualizations of students interactions in the experimental group.
12

4. Discussions
The results of this study show that annotation behaviors, engagement, and interactions of students in the experimental group are
significantly superior to those of students in the control group. The interview of students further confirmed that they were motivated
by level and leaderboard mechanisms in collaborative annotation activities. The annotation features unlocked in different levels (e.g.,
read others’ reading and response annotations and the use of “heart” function) would motivate them to increase their contributions in
making annotations both in quantity and quality. These findings are similar to the results of literature, indicating that gamification
design would motivate students to contribute more posts in their online discussion (Barata et al., 2013; Ding et al., 2018; Hew et al.,
2016; Smith et al., 2013).
In addition, previous research has found a positive correlation between students' contribution of annotation quantity and their
course learning achievement (Hwang et al., 2007; Su et al., 2010) and between students' engagement and science learning perfor-
mance (Cheng et al., 2015). It is expected that when students made more annotations and engage more in the collaborative anno-
tation reading activity, they would perform better in their reading comprehension assessment. However, although students in the
experimental group made significantly more annotations and higher engagement than those of the control group, the differences
between the two groups in reading comprehension performance were not found in this study. The design of the level and the
leaderboard was aimed to give feedback about their individual achievement and show their statuses and reputation (Richter et al.,
2015) to motivate their efforts on reading and annotating collaboratively, so as to promote their reading comprehension. According
to the interview of the students, they were motivated to make more annotations because they wanted to level up to use more
annotation features, interact with others, and gain reputations. This indicates the success of using level and leaderboard mechanisms
to elicit students' behavior engagement. However, students viewed differently about the effectiveness of game mechanisms to fa-
cilitate their reading comprehension. Some students stated that they would be more concentrated in the content of the article when
they used WCRAS without gamification mechanisms because they would not have to care about the level tasks. As the students in this
study were in their middle childhood stage, peers groups become important sources for them to build self-concept and self-esteem
(Papalia & Olds, 1995) and social competition plays a critical role for their development (DelGiudice, 2018) so that they would
compete to show their competence. Therefore, they might pay their attention more on their reputation on the leaderboard. This
suggests that these students possibly tried to make faster progression on levels by completing the required tasks designed based on the
quantity of the annotations but spent little time to carefully read the article and the annotations made by other students or reply
response annotations meaningfully. Due to a lack of consideration on students' psychological characteristics, the gamification me-
chanisms adopted in this study only promoted students’ reading activity in competing the quantity of annotations and interactions
but not facilitate the quality of reading.
Another reason that might affect students' performance on reading comprehension is whether students could effectively learn
from the annotations generated by their peers. The WCRAS was designed to provide a self-learning environment to facilitate reading
comprehension by collaborative annotations. However, an excessive amount of annotations might make it difficult for students to
read and get useful information. Although no students mentioned about the difficulties of reading such many annotations, with a total
899 and 537 reading annotations made by the students in the experimental and control groups respectively, it is doubtable that
students would be able to read all the annotations and enhance their reading comprehension of the article by sharing and discussing
annotations with others. Gao (2013) indicated that it was hard to find the comments one needed in the first sight when there were too
many comments annotated to the reading texts. Students might randomly read and respond to few comments and did not develop a
comprehensive understanding of the content. According to the results of the present study, the average number of response anno-
tations made by the students in both groups was less than the reading annotations, especially for the control group. This might be
questionable how many reading annotations were actually read by the students and whether the ones they read would help them
understand the article. Moreover, Gao (2013) stated that when there were many small discussions based on annotated text, students
would report difficulties to gain a holistic understanding of the text. It was also possible that students found it more interesting to
read the comments than the article that they were distracted from their reading task. In Jan, Chen and Huang's (2016) study, they
found that students who are provided with high-grade annotation filter would perform better in digital reading performance than
those who read all annotations. Therefore, a good recommendation mechanism may be necessary to allow students to read high-
quality and useful annotations more efficiently.
In previous gamification research, several studies also found that gamification design could foster students' performance in
learning activities, but have no effects on students' final exams (Hew et al., 2016; Özdener, 2018). The results may imply a gap
between gamified learning activity and the final learning outcomes. For example, in Hillesund’s (2010) study, students were asked to
develop a questionnaire in groups. The study found that students posted more in discussion forums and produced higher quality of
questionnaire when using gamification mechanisms. However, the scores of the test that required students to recall factual knowledge
of the learning topic did not differ between gamified and non-gamified groups. For this present study, although students made
significantly more reading and response annotations that may imply more information providing and sharing in the experimental
group, the efforts that students made did not reflect on their reading comprehension. The analysis results of annotation quality in this
study could provide some evidences. Although the gamified group did make more high-quality annotations than did non-gamified
group, the two groups did not significantly differ in making annotations that would promote different comprehension aspects in-
cluding word explanations and four comprehension process defined by PIRLS. This implies that the gamification mechanisms did not
effectively promote the experimental group to generate more annotations corresponding to the aspects of reading comprehension
assessment. According to Chall’s (1983) reading developmental stages, the students in the present study would be in the stage that
begins to “reading for learning the new.” The goal of this stage is to develop advanced reading comprehension ability not only to
13

understand the facts from the text but also integrate information and draw inferences from what they read (Wolf, 2008). Students will
need guidance to develop their reading strategies. Although students expressed in the interviews that the annotation types could help
them comprehend the article, the currently designed annotation types seemed not to be able to guide students applying the four
comprehension processes of PIRLS in their reading activity, nor did the gamification mechanisms being designed to facilitate their
reading in corresponding to the four comprehension processes. Therefore, how to promote students' reading comprehension by
designing appropriate gamification and annotation mechanisms and make good connection between reading activity and compre-
hension assessment are important issues to be further studied.
5. Conclusions and future works
5.1. Conclusions
The gamification mechanisms were applied to the WCRAS in this study to facilitate elementary school students' collaborative
annotating behaviors and reading comprehension performance. Analytical results show that the use of gamified WCRAS can en-
courage students to make more quantity and higher quality annotations and promote their engagement in the collaborative anno-
tation. Additionally, when students used gamified WCRAS, they interacted with each other more than the students who did not use
gamified WCRAS. However, the differences in reading comprehension performances between the two groups were not significant. In
the previous section, the possible reasons were discussed from different aspects including students' psychological and learning
characteristics, the filtration of quality annotations, and the match of learning assessment with gamified learning activities. In
conclusion, students’ in-activity performance was enhanced by the gamification design but their reading comprehension performance
was not. The limitations of this study and the implications for future research, gamification design modifications, and pedagogical
implementation are addressed as follows.
5.2. Limitations of the study
The limitations of this study are three folds. First, the experimental time of the WCRAS in collaborative reading and annotating
was a one-time activity and limited to 100 min. Since students in the gamified group made abundant annotations, they would need
more time to read others' annotations to aggregate their understanding of the article. In addition, reading comprehension ability
relies on long-term development. The benefits of the gamification design on reading comprehension might need a longer time of the
experiment to be obtained. The second limitation is about the reading comprehension assessment. In this study, the reading article
and reading assessment were both selected from PIRLS 2006 because they were internationally used and developed with good
validity and reliability. However, among high-quality text annotation made by the students in the present study, there is a certain
amount of annotations made to provide word explanations that were not tested in PIRLS's reading assessment. The actual reading
comprehension considering other aspects might be underestimated. Lastly, the gamification design of this study was to encourage
students to make annotations. As for the appropriate use of annotation types and the quality of the annotation content, even though
students could use “correction” to respond to others, it would be problematic to fully rely on students' limited knowledge and reading
abilities . It is necessary for teacher to monitor the annotations made by students. However, due to a large amount of the annotations
generated by the students, the teacher in this study could only monitor and delete the improper wordings but could not provide the
advice on the annotation content.
5.3. Implications for future works
Suggestions for future studies in research design, system modification, and pedagogical implications are stated as follows. For
research design, design-based research is suggested to improve the WCRAS system and to develop the pedagogical design. The design-
based research is “a systematic but flexible methodology aimed to improve educational practices through iterative analysis, design,
development, and implementation, based on collaboration among researchers and practitioners in real-world settings, and leading to
contextually-sensitive design principles and theories.” (Wang & Hannafin, 2005). This approach has been adopted in many tech-
nology-supported learning studies to develop the learning model and modify the technology-supported learning tools tailored to the
pedagogical practices (e.g., Alvarez, Alarcon, & Nussbaum, 2011; Wong, Boticki, Sun, & Looi, 2011). In addition, a long-term ex-
perimental design is needed to evaluate the effects of WCRAS with gamification mechanisms. In Dichev and Dicheva’s (2017) review
of gamification studies, most of the experiments were conducted from several weeks to the whole semester. Hew et al.‘s study (2016)
conducted two studies to investigate students' learning engagement and performance. In the first study, a 3-day experiment was
performed and no significant result was found. In their second study, the researchers not only modify their gamification design but
also expand the experiment to 18 days. The results show that the effects of the gamification design on students' learning engagement
and performance were significant. On the other hand, reading comprehension ability needs long-term development. Therefore, it is
suggested that the duration of the experiment in this study can be extended to at least one semester to get more insights about the
impacts of gamified WCRAS on the effectiveness of fostering reading comprehension. Moreover, the assessment and learning ac-
tivities should be matched with each other so that students' reading comprehension can be better assessed and reflected from stu-
dents' collaborative reading and annotating activities.
Next, modifications of the WCRAS and its gamification design are suggested. First, the design of gamification mechanisms in this
study used the number of annotations and interactions as the task requirements for leveling up and the leaderboard was employed for
14

students to compare with their peers that motivated them to achieve higher levels. The results show that the gamification me-
chanisms can effectively promote students' willingness and interaction in making annotations, but the effectiveness of reading
comprehension promotion was not significant. This might indicate a lack of connection between the gamification design and the
reading comprehension processes. In future studies, game mechanisms can be adjusted to better associate students' annotation be-
haviors and the reading comprehension process by taking students' psychological and learning characteristics into consideration. As
for middle childhood students, they would perceive their competence by comparing with others and would need guidance for
learning from reading. Game mechanisms need to be designed to motivate them to compete for reading performance in both quantity
(i.e., annotation and interaction frequencies) and quality (i.e., reading comprehension). For example, in addition to meeting the
required numbers of annotations and interactions as designed in this study, challenges corresponding to different levels of reading
comprehension can be added as part of the gamification mechanisms in the future. Students have to answer those challenging
questions successfully to proceed to the next game level. The higher the game levels, the more sophisticated reading comprehension
questions are assigned for the challenge. The modified gamification mechanisms may help students advance their reading com-
prehension gradually through leveling up. Second, finding and reading high-quality annotations from a large number of annotations
might also influence students' reading comprehension of the article. In Jan et al.‘s (2016) study, they employed two quality anno-
tation filtering mechanisms (i.e., high-grade and master annotation filters) to promote students' reading performance. They found
that students would perform significantly better in digital reading when high-grade annotation filter was adopted compared to those
who read all annotations. In addition, Chen, Wang, and Chen (2014) adopted a voting mechanism for learners to identify high-quality
annotations. They found that this voting mechanism not only promotes annotators' reading comprehension by encouraging them to
make annotations with good quality but also foster other students' reading comprehension. The quality issue has also been addressed
by the researchers (Hew et al., 2016; Smith et al., 2013), indicating that gamification mechanisms were designed to promote the
quality of students' group projects and the discussion posts. With these gamification mechanisms, students were found to produce
high quality of their artifacts and posts. The gamification mechanism to promote high-quality annotation is also suggested. Although
teachers can check the quality of annotations and give feedbacks or guidance, it is difficult for them to deal with a large number of
annotations timely and efficiently. Therefore, developing quality annotation filtering mechanisms to provide suggestions of high-
quality annotations automatically or developing a motivate mechanism that can promote the quality of annotations for the WACRS
with gamification mechanism should be considered.
Lastly, some pedagogical implications of using gamified WCRAS are suggested. The gamified WCRAS examined in this study was
designed as a self-learning tool for students to read text-based material collaboratively with the support of social annotation and
gamification features. In recent years, the flipped classroom model has been advocated to transform the learning in the classroom into
student-centered and active learning practice (Francl, 2014). In a flipped instruction, students are expected to learn from videos or
other modes by themselves before going to the classroom for further discussion. Although the videos are commonly used in flipped
instruction (Herreid & Schiller, 2013), teachers have to make a great effort to find proper videos or take a long time to make their
own; students might also come to the class without watching videos due to a lack of self-regulated learning ability. Besides videos,
reading materials were also provided by some researchers as the pre-class homework (Jovanović, Gašević, Dawson, Pardo, &
Mirriahi, 2017). Research has shown that students used WCRAS for pre-reading would significantly achieve better in learning per-
formance and make better pre-reading by reviewing peer's annotations (Hwang & Hsu, 2011). A gamified WCRAS that support text-
based collaborative reading, therefore, can be considered for pre-class self-learning tool of supporting flipped instruction. With
gamification mechanisms and reading scaffoldings, students would more engage in reading and could collaboratively construct
knowledge with their peers that may enhance their reading comprehension and motivation for pre-class learning of flipped in-
struction.
Acknowledgement
This work was supported by the Ministry of Science and Technology, Taiwan [grant number MOST 106-2511-S-004 -005 -MY3].
Appendix A. Supplementary data
Supplementary data to this article can be found online at https://doi.org/10.1016/j.compedu.2019.103697.
References
de-Marcos, L., García-López, E., García-Cabot, A., Medina-Merodio, J. A., Domínguez, A., Martínez-Herráiz, J. J., et al. (2016). Social network analysis of a gamified e-
learning course: Small-world phenomenon and network metrics as predictors of academic performance. Computers in Human Behavior, 60, 312–321. https://doi.
org/10.1016/j.chb.2016.02.052.
Alvarez, C., Alarcon, R., & Nussbaum, M. (2011). Implementing collaborative learning activities in the classroom supported by one-to-one mobile computing: A design-
based process. Journal of Systems and Software, 84(11), 1961–1976. https://doi.org/10.1016/j.jss.2011.07.011.
Barata, G., Gama, S., Jorge, J., & Gonçalves, D. (2013). Improving participation and learning with gamification. Proceedings of the first international conference on
gameful design, research, and applications (pp. 10–17). New York, NY, USA: ACM. https://doi.org/10.1145/2583008.2583010.
Boettger, R. K., & Palmer, L. A. (2010). Quantitative content analysis: Its use in technical communication. IEEE Transactions on Professional Communications, 53(4),
346–357. https://doi.org/10.1109/TPC.2010.2077450.
Brown, E., & Cairns, P. (2004). A grounded investigation of game immersion. CHI ’04 extended abstracts on human factors in computing systems (pp. 1297–1300). New
York, NY, USA: ACM. https://doi.org/10.1145/985921.986048.
Cardoso, G., Ganito, C., & Ferreira, C. (2012). Digital Reading: The transformation of reading practices. Proceedings of the 16th international conference on electronic
15

publishing Guimaraes, Portugal.
Chall, J. (1983). Stages of reading development. New York: McGraw Hill.
Chen, C. M., & Chen, F. Y. (2014). Enhancing digital reading performance with a collaborative reading annotation system. Computers & Education, 77, 67–81.
Cheng, M., Lin, Y., She, H., & Kuo, P. (2017). Is immersion of any value? Whether, and to what extent, game immersion experience during serious gaming affects
science learning. British Journal of Educational Technology, 48(2), 246–263. https://doi.org/10.1111/bjet.12386.
Cheng, M. T., She, H. C., & Annetta, L. A. (2015). Game immersion experience: Its hierarchical structure and impact on game-based science learning. Journal of
Computer Assisted Learning, 31(3), 232–253. https://doi.org/10.1111/jcal.12066.
Chen, C. M., Wang, J. Y., & Chen, Y. C. (2014). Facilitating English-language reading performance by a digital reading annotation system with self-regulated learning
mechanisms. Journal of Educational Technology & Society, 17(1), 102–114.
DelGiudice, M. (2018). Middle childhood: An evolutionary-developmental synthesis. In N. Halfon, C. B. Forrest, R. M. Lerner, & E. M. Faustman (Eds.). Handbook of life
course health development (pp. 95–107). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-47143-3_5.
Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to gamefulness. Defining “gamification.” in proceedings of the 15th international
academic MindTrek conference: Envisioning future media environments (pp. 9–15). New York, NY, USA: ACM. https://doi.org/10.1145/2181037.2181040.
Dicheva, D., Dichev, C., Agre, G., & Angelova, G. (2015). Gamification in education: A systematic mapping study. Journal of Educational Technology & Society, 18(3),
75–88.
Dichev, C., & Dicheva, D. (2017). Gamifying education: What is known, what is believed and what remains uncertain: A critical review. International Journal of
Educational Technology in Higher Education, 14(9), https://doi.org/10.1186/s41239-017-0042-5.
Ding, L., Er, E., & Orey, M. (2018). An exploratory study of student engagement in gamified online discussions. Computers & Education, 120, 213–226. https://doi.org/
10.1016/j.compedu.2018.02.007.
Domínguez, A., Saenz-de-Navarrete, J., de-Marcos, L., Fernández-Sanz, L., Pagés, C., & Martínez-Herráiz, J.-J. (2013). Gamifying learning experiences: Practical
implications and outcomes. Computers & Education, 63, 380–392. https://doi.org/10.1016/j.compedu.2012.12.020.
Estellés Arolas, E., Del Moral-Pérez, E., & González, F. (2010). Social bookmarking tools as facilitators of learning and research collaborative processes: The Diigo Case.
Interdisciplinary Journal of E-Skills and Lifelong Learning, 6, 175–191. https://doi.org/10.28945/1298.
Francl, T. (2014). Is flipped learning appropriate. Journal of Research in Innovative Teaching, 71, 119–128.
Gao, F. (2013). A case study of using a social annotation tool to support collaboratively learning. The Internet and Higher Education, 17, 76–83. https://doi.org/10.
1016/j.iheduc.2012.11.002.
Goehle, G. (2013). Gamification and web-based homework. Primus, 23(3), 234–246. https://doi.org/10.1080/10511970.2012.736451.
Gwet, K. L. (2012). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among multiple raters (3rd
ed.). Gaithersburg, MD: Advanced
Analytics, LLC.
Hamari, J., Shernoff, D. J., Rowe, E., Coller, B., Asbell-Clarke, J., & Edwards, T. (2016). Challenging games help students learn: An empirical study on engagement,
flow and immersion in game-based learning. Computers in Human Behavior, 54, 170–179. https://doi.org/10.1016/j.chb.2015.07.045.
Haythornthwaite, C. (1996). Social network analysis: An approach and technique for the study of information exchange. Library & Information Science Research, 18(4),
323–342. https://doi.org/10.1016/S0740-8188(96)90003-1.
Herreid, C. F., & Schiller, N. A. (2013). Case study: Case studies and the flipped classroom. Journal of College Science Teaching, 42(5), 62–67.
Hew, K. F., Huang, B., Chu, K. W. S., & Chiu, D. K. W. (2016). Engaging Asian students through game mechanics: Findings from two experiment studies. Computers &
Education, 92–93, 221–236. https://doi.org/10.1016/j.compedu.2015.10.010.
Hillesund, T.. Digital reading spaces: How expert readers handle books, the Web and electronic paper. First Monday. (2010). Retrieved from http://firstmonday.org/ojs/
index.php/fm/article/view/2762 15-4.
Hwang, W.-Y., & Hsu, G.-L. (2011). The effects of pre-reading and sharing mechanisms on learning with the use of annotations. Turkish Online Journal of Educational
Technology - TOJET, 10(2), 234–249.
Hwang, W. Y., Wang, C. Y., & Sharples, M. (2007). A study of multimedia annotation of Web-based materials. Computers & Education, 48(4), 680–699. https://doi.org/
10.1016/j.compedu.2005.04.020.
Jan, J. C., Chen, C. M., & Huang, P. H. (2016). Enhancement of digital reading performance by using a novel web-based collaborative reading annotation system with
two quality annotation filtering mechanisms. International Journal of Human-Computer Studies, 86, 81–93. https://doi.org/10.1016/j.ijhcs.2015.09.006.
Jennett, C., Cox, A. L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., et al. (2008). Measuring and defining the experience of immersion in games. International Journal of
Human-Computer Studies, 66(9), 641–661. https://doi.org/10.1016/j.ijhcs.2008.04.004.
Johnson, T. E., Archibald, T. N., & Tenenbaum, G. (2010). Individual and team annotation effects on students' reading comprehension, critical thinking, and meta-
cognitive skills. Computers in Human Behavior, 26(6), 1496–1507. https://doi.org/10.1016/j.chb.2010.05.014.
Jovanović, J., Gašević, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017). Learning analytics to unveil learning strategies in a flipped classroom. The Internet and Higher
Education, 33, 74–85. https://doi.org/10.1016/j.iheduc.2017.02.001.
Kiili, C., Laurinen, L., Marttunen, M., & Leu, D. J. (2012). Working on understanding during collaborative online reading. Journal of Literacy Research, 44(4), 448–483.
Kim, O., & Walker, M. (1984). The free rider problem: Experimental evidence. Public Choice, 43, 3–24.
Laat, M. de, Lally, V., Lipponen, L., & Simons, R.-J. (2007). Investigating patterns of interaction in networked learning and computer-supported collaborative learning:
A role for social network analysis. International Journal of Computer-Supported Collaborative Learning, 2(1), 87–103. https://doi.org/10.1007/s11412-007-9006-4.
Liu, Z. (2005). Reading behavior in the digital environment: Changes in reading behavior over the past ten years. Journal of Documentation, 61(6), 700–712. https://
doi.org/10.1108/00220410510632040.
Mekler, E. D., Brühlmann, F., Tuch, A. N., & Opwis, K. (2017). Towards understanding the effects of individual gamification elements on intrinsic motivation and
performance. Computers in Human Behavior, 71, 525–534. https://doi.org/10.1016/j.chb.2015.08.048.
Mendenhall, A., & Johnson, T. E. (2010). Fostering the development of critical thinking skills, and reading comprehension of undergraduates using a Web 2.0 tool
coupled with a learning system. Interactive Learning Environments, 18(3), 263–276.
Mullis, I. V. S., Martin, M. O., Kennedy, A. M., & Foy, P. (2007). PIRLS 2006 international report: IEA's Progress in international reading literacy Study in primary schools in
40 countries. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College.
Novak, E., Razzouk, R., & Johnson, T. E. (2012). The educational use of social annotation tools in higher education: A literature review. The Internet and Higher
Education, 15(1), 39–49. https://doi.org/10.1016/j.iheduc.2011.09.002.
Özdener, N. (2018). Gamification for enhancing Web 2.0 based educational activities: The case of pre-service grade school teachers using educational Wiki pages.
Telematics and Informatics, 35(3), 564–578. https://doi.org/10.1016/j.tele.2017.04.003.
Palonen, T., & Hakkarainen, K. (2000). Patterns of interaction in computer-supported learning: A social network analysis. In B. Fishman, & S. O'Connor-Divelbiss
(Eds.). Fourth international conference of the learning sciences (pp. 334–339). Mahwah, NJ: Erlbaum.
Papalia, D. E., & Olds, S. W. (1995). Human development (6th
ed.). New York: McGraw-Hill.
Prawat, R. S., & Floden, R. E. (1994). Philosophical perspectives on constructivist views of learning. Educational Psychology, 29(1), 37–48.
Rabbany, R., Elatia, S., Takaffoli, M., & Zaïane, O. R. (2014). Collaborative learning of students in online discussion forums: A social network analysis perspective. In
educational data mining. Springer International Publishing, 441–466.
Razon, S., Turner, J., Johnson, T. E., Arsal, G., & Tenenbaum, G. (2012). Effects of a collaborative annotation method on students' learning and learning-related
motivation and affect. Computers in Human Behavior, 28(2), 350–359. https://doi.org/10.1016/j.chb.2011.10.004.
Richter, G., Raban, D. R., & Rafaeli, S. (2015). Studying gamification: The effect of rewards and incentives on motivation. In T. Reiners, & L. C. Wood (Eds.).
Gamification in education and business (pp. 21–46). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-10208-5_2.
Scott, J. (2000). Social network analysis: A handbook (2nd ed.). London: SAGE Publications.
Smith, E., Herbert, J., Kavanagh, L., & Reidsema, C. (2013). The effects of gamification on student learning through the use of reputation and rewards within
community moderated discussion boards. Presented at the AAEE 2013: 24th annual conference of the australasian association for engineering education (pp. 1–9).
16

Griffith School of Engineering, Griffith University. Retrieved from https://espace.library.uq.edu.au/view/UQ:326086.
Su, A. Y. S., Yang, S. J. H., Hwang, W.-Y., & Zhang, J. (2010). A Web 2.0-based collaborative annotation system for enhancing knowledge sharing in collaborative
learning environments. Computers & Education, 55(2), 752–766. https://doi.org/10.1016/j.compedu.2010.03.008.
Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research & Development, 53(4),
5–23. https://doi.org/10.1007/BF02504682.
Wolf, M. (2008). Proust and the squid: The story and science of the reading brain. Reprint editionNew York: Harper Perennial.
Wong, L.-H., Boticki, I., Sun, J., & Looi, C.-K. (2011). Improving the scaffolds of a mobile-assisted Chinese character forming game via a design-based research cycle.
Computers in Human Behavior, 27(5), 1783–1793. https://doi.org/10.1016/j.chb.2011.03.005.
Yang, X., Yu, S., & Sun, Z. (2013). The effect of collaborative annotation on Chinese reading level in primary schools in China. British Journal of Educational Technology,
44(1), 95–111.
Yang, S. J. H., Zhang, J., Su, A. Y. S., & Tsai, J. J. P. (2011). A collaborative multimedia annotation tool for enhancing knowledge sharing in CSCL. Interactive Learning
Environments, 19(1), 45–62. https://doi.org/10.1080/10494820.2011.528881.
Chih-Ming Chen is currently a professor in the Graduate Institute of Library, Information and Archival Studies at National Chengchi University, Taipei, Taiwan. His
research interests include digital library, e-learning, data mining, machine learning and intelligent agents on the web.
Ming-Chaun Li is currently a postdoctoral researcher in the Graduate Institute of Library, Information and Archival Studies at National Chengchi University, Taipei,
Taiwan. Her research interests include e-learning and game-based learning.
Tze-Chun Chen is currently a graduate student in the E-learning Master Program of Library and Information Studies at National Chengchi University, Taipei, Taiwan.
Her research interests include e-learning and game-based learning.
17

A Web-Based Collaborative Reading Annotation System With Gamification Mechanisms To Improve Reading Performance

More Related Content

Similar to A Web-Based Collaborative Reading Annotation System With Gamification Mechanisms To Improve Reading Performance (20)

More from Kate Campbell (20)

Recently uploaded (20)

A Web-Based Collaborative Reading Annotation System With Gamification Mechanisms To Improve Reading Performance