AcaWriter A Learning Analytics Tool For Formative Feedback On Academic Writing

Knight, S., Shibani, A., Abel, S., Gibson, A., Ryan, P., Sutton, N., Wight, R., Lucas, C., Sándor, Á.,
Kitto, K., Liu, M., Vijay Mogarkar, R., & Buckingham Shum, S. (2020). Acawriter: A learning
analytics tool for formative feedback on academic writing. Journal of Writing Research, 12(1),
141-186. https://doi.org/10.17239/jowr-2020.12.01.06
Contact: Simon Knight, Faculty of Transdisciplinary Innovation, University of Technology
Sydney, PO BOX 123, Broadway, NSW 2007 | Australia – simon.knight@uts.edu.au, ORCID:
0000-0002-8709-5780
Copyright: Earli | This article is published under Creative Commons Attribution-
Noncommercial-No Derivative Works 3.0 Unported license.
AcaWriter
A Learning Analytics Tool for Formative
Feedback on Academic Writing
Simon Knight1, Antonette Shibani1, Sophie Abel1, Andrew Gibson2,
Philippa Ryan1, Nicole Sutton1, Raechel Wight1, Cherie Lucas1,
Ágnes Sándor3, Kirsty Kitto1, Ming Liu1, Radhika Vijay Mogarkar1 &
Simon Buckingham Shum1
1 University of Technology Sydney | Australia
2 Queensland University of Technology | Australia
3 Naver Labs Europe | France
Abstract: Written communication is an important skill across academia, the workplace, and
civic participation. Effective writing incorporates instantiations of particular text structures –
rhetorical moves – that communicate intent to the reader. These rhetorical moves are
important across a range of academic styles of writing, including essays and research
abstracts, as well as in forms of writing in which one reflects on learning gained through
experience. However, learning how to effectively instantiate and use these rhetorical moves
is a challenge. Moreover, educators often struggle to provide feedback supporting this
learning, particularly at scale. Where effective support is provided, the techniques can be
hard to share beyond single implementation sites. We address these challenges through the
open-source AcaWriter tool, which provides feedback on rhetorical moves, with a design
that allows feedback customization for specific contexts. We introduce three example
implementations in which we have customized the tool and evaluated it with regard to user
perceptions, and its impact on student writing. We discuss the tool’s general theoretical
background and provide a detailed technical account. We conclude with four
recommendations that emphasize the potential of collaborative approaches in building,
sharing and evaluating writing tools in research and practice.
Keywords: writing analytics, learning analytics, genre theory, feedback, rhetorical moves

KNIGHT ET AL.  ACAWRITER: WRITING ANALYTICS FORMATIVE FEEDBACK E | 142
1. Introduction to AcaWriter and its Theoretical Basis
Academic writing is an essential skill that all higher education students must master
to succeed in their degree and professionally (OECD & Statistics Canada, 2000).
Students are asked to write documents in diverse styles that are assessed in various
contexts within a university. For instance, students write laboratory reports in
Science, legal opinions in Law, business reports across business disciplines,
research articles for higher degrees, and essays in the Humanities. Alongside these
traditional analytical genres, students also engage in other forms of reflective
writing to record their thoughts, feelings, and reactions to an event or situation and
reflecting on the learning process, in disciplines from Pharmacy, Engineering, and
Education, all while adhering to their disciplinary writing conventions. However, the
teaching and learning of academic writing are challenging as there is often a
misalignment between students’ and educators’ expectations of academic writing
(Lea & Street, 1998). This misalignment is compounded by the difficulty of providing
detailed explicit formative feedback on how to improve writing, due for example,
to resource constraints, lack of knowledge regarding writing pedagogy, and
inconsistency in writing program delivery (Lea & Street, 1998). Feedback is
necessary to improve students’ academic writing, as it allows students to review and
revise their writing, which is a key element in the writing process (Hayes & Flower,
1980). Feedback allows students to reflect on what they have done and how they
can improve, addressing the gap between their performance and their educator’s
expectations. Formative feedback encourages learners to focus on improvement,
rather than grade outcomes. Sharing pedagogic approaches to support this
formative feedback can aid educators in teaching students how to write effectively.
These factors motivated the development and implementation of an automated
feedback tool called AcaWriter.
Using technological advances, automated and semi-automated tools have been
developed to overcome the issues of time, effort and reliability in evaluating written
texts and providing formative feedback (Shermis & Burstein, 2013). The scope of
tools that evaluate writing varies from Automatic Essay Scoring (AES) systems that
provide a score based on the assessment of standardized writing to Automated
Writing Evaluation (AWE) systems that provide additional feedback to students on
their writing (Warschauer & Grimes, 2008). Intelligent tutoring systems for writing
instruction also exist that act as a strategy-oriented writing aid to help students with
well-defined problems (Roscoe & McNamara, 2013). A recent review (Strobl et al.,
2019) highlights the differences between writing tools in terms of their purpose and
introduces a new category called Interactive Writing Platforms (IWP) that provide
prompts and scaffold the writing process for students, but do not process their
input or give feedback (Strobl et al., 2019). Such technological advances have
provided new opportunities for improving teaching and learning practices.

143 | JOURNAL OF WRITING RESEARCH
Learning analytics has emerged as a field in which data from learning is used to
better understand the support of that learning (see, for example, LAK11, 2011;
Siemens & Long, 2011). Within the field of learning analytics, writing analytics has
been of particular interest in supporting the process of writing and providing
feedback on written products (see, for example, the workshop series Buckingham
Shum et al., 2016; Knight et al., 2017; Shibani et al., 2018; Shibani, Liu, et al., 2019). The
focus of this paper is one such writing analytics tool, AcaWriter, which provides
scalable formative feedback to students to improve their writing.
AcaWriter is a web-based writing analytics tool that provides automated
formative feedback on the rhetorical moves in students’ texts. It does this by
drawing on Swales’ (1990) genre analysis of texts in terms of their moves and
communicative goals and Hyland’s (2005) notion of metadiscourse. These rhetorical
moves are phrases and sentences that indicate or explain to the reader the writer’s
attitude or position in relation to what has come before in the text or what is ahead
or with respect to other texts. These moves include summarizing (for example, “it
is therefore clear that”), contrasting (for example, “conversely”), and emphasizing
(for example, “it is important to note that”). The sentences conveying the rhetorical
moves may contain words or phrases as metadiscourse cues that orient the reader,
and thereby support coherent and persuasive writing. Rhetorical moves thus
perform a communicative function by helping the reader follow the flow of the text
(Swales, 2004).
AcaWriter uses natural language processing (NLP) techniques to identify
sentences that communicate a specific rhetorical function. The software includes a
computational parser which processes each sentence in the text and identifies such
rhetorical moves. It is based on the concept-matching framework (Sándor, 2005),
which models the metadiscourse cues of identified rhetorical moves as
instantiations of syntactically related constituent concepts in sentences. Thus, the
syntactically related words of a Background move such as, “recent studies indicate”,
instantiate the constituent concepts [PREVIOUS] [PUBLICATION] [IDEA]. The tool
uses syntactic parsing to extract syntactic dependencies and a dedicated lexicon
that contains a list of words and expressions that may instantiate the constituent
concepts (as shown above). It then uses pattern-matching rules that identify
sentences containing pairs of instantiations of constituent concepts that are
associated with rhetorical moves.
The implementation of AcaWriter in the classroom has taken a genre-based
approach, in which writing is viewed as socially situated and purposeful (Hyland,
2007) and makes explicit the way language works to create meaning (Cope &
Kalantzis, 1993) and achieve its rhetorical goal. By taking this perspective towards
writing AcaWriter is able to be contextualized to fit different learning contexts
through targeting their particular genres. In this paper, we provide a detailed
presentation of the tool and its underlying theory (next section), before reporting

on the different learning contexts that AcaWriter currently provides automated
formative feedback for, through the description of three high-level deployment
contexts and evaluation in these contexts. We conclude summarizing our findings
across these three contexts, and directions for future research and development for
the AcaWriter tool.
2. Detailed Presentation of AcaWriter and its Parsers
2.1 Tool technical background
The AcaWriter tool is presented to the users (students and instructors) as a single
browser-based application. It is in fact comprised of three different open-source
web applications that interact: the AcaWriter web application, and two underlying
technologies, the Text Analytics Pipeline (TAP), and Athanor. Each of these
applications is deployed in a cloud environment, however, only the user interface
(UI) in the form of AcaWriter is accessible to the end user. The full set of
applications is available open-source, with Jupyter notebook guides (which
combine code blocks, their outputs, and rich text description) available to support
people through the tool design and use (Shibani et al., 2018), and educator
resources illustrating how AcaWriter can be integrated into writing improvement
tasks1.
The Athanor server is a web application that provides an API (application
programming interface). It first calls the Stanford CoreNLP software that for each
sentence of the input text produces a dependency tree, as well as part-of-speech
and named entity tags. Then it calls the Athanor rule engine that applies the pattern-
matching rules, which assign rhetorical move labels to sentences2.
The rhetorical moves associated with the text are supplemented by other custom
text analysis provided by TAP. TAP provides a unified API for a range of text analysis
functions, one of which is the rhetorical moves function provided by the Athanor
server. Others include basic text features (n-grams, text cleaning, paragraph counts,
etc.).
The user experience of AcaWriter is provided by the AcaWriter UI which
coordinates all of the necessary aspects of user interaction including authorization
of access, matching users to genre-based feedback associated with fields of study
or assignments, collecting the input text from the user, calling the appropriate
function from TAP, and providing a visualization of the resultant analytics back to
the user. This user experience can be customized in a number of ways, including
through the creation of “assignment codes” which, when entered by a user, preload
a particular feedback ‘genre’ for the writing submitted for that assignment. These
‘genres’ describe particular standards for writing for the particular audience in the
given assignments, for example in law or accounting, or for writing research

abstracts and reflective writing as will be discussed further below. These feedback
customizations are developed within the AcaWriter tool by mapping the UI to the
underlying features from TAP (Knight, Shibani, et al., 2018; Shibani, Knight, et al.,
2019)3, as detailed in the cases below.
2.2 AcaWriter user experience: Learning design and contextualizable genre
profiles
The AcaWriter tool contains two primary genre feedback modes to which
modifications and additions may be made:
 The analytical genre, which provides feedback on scholarly academic rhetorical
moves familiar to research writing and a wide range of university-level writing.
 The reflective genre, which provides feedback on scholarly reflective writing in
which students are asked to reflect on how their experience relates to their
professional practice and its development.
Basic functionality provides three tabbed pages to users: (1) A report, which shows
the input text highlighted with the moves identified, (2) a feedback tab, which
provides customizable messages regarding the moves identified, for example, if
particular functions are missing, and (3) a resources tab, which can provide learning
materials and examples. Each of these tabs can be modified for particular contexts,
for example to (1) focus on particular moves (by ignoring the presence/absence of
others), (2) Based on the identification of these moves, feedback is provided to
students – which may also be customized – for example, give custom feedback
indicating that a particular move has not been identified in the writing, or providing
a positive message if moves are in a theoretically desirable order and (3) link to
discipline-specific guidance and examples.
This specific instantiation of the linguistic rules to identify moves into student-
facing feedback has drawn on research and practice in well-established fields such
as User-Centred Design, Participatory Design and Co-Design (Buckingham Shum et
al., 2019) with instructors and linguists co-designing and developing rules
appropriate to the pedagogic context (see, for example, Knight, Buckingham Shum,
et al., 2018). The specific genre profiles and their instantiation into feedback are
described in full in the empirical work, which indicates how the specific moves
reported, feedback regarding those moves, and resources to support students can
be customized for a specific learning context (which can be assigned to ‘codes’ in
the system, which when entered by the user display particular feedback).
To implement AcaWriter and integrate it effectively across disciplinary teaching
and learning contexts, the tool is designed to support customization of new
feedback genres that provide context-sensitive feedback to students that is relevant
to their genre of writing. Such contextualization of learning analytics tools like
AcaWriter can augment existing pedagogical practices to help students make

effective use of the tools to support learning (Shibani, Knight, et al., 2019). This
augmentation is implemented through working with instructors who are subject
experts in their respective disciplines to co-design feedback and writing
interventions for students that are customized for their disciplinary context. That is,
in each of the implementation cases (discussed in detail below), the tool and its
evaluation is conducted in the context of specific learning designs, with tasks
defined such that a set of theoretically well-grounded tasks can be transferred
across different learning contexts.
2.3 AcaWriter instructor experience
The software is targeted at two primary end user groups: academics and students.
Through a “Manage Assignment” menu, academics (who are generally subject
instructors) may create an “Assignment code” to be provided to students, that (1)
marks students who use the code as submitting against that particular assignment
(and associated feedback-genre), and (2) can set up specific feedback, at present
from a set of pre-defined ‘feedback genres’. These genres provide some feedback
specific to the task, as described in the following sections. The instructions to
academics explain that:
When using AcaWriter to manage your assignment, it’s important to choose
the correct genre type. This allows AcaWriter to provide you with the
relevant feedback support on your writing. There are a number of different
genres you can choose from, and these can be found in the dropdown menu
on the left-hand side of the page. Each genre will provide you with different
Figure 1. Sample AcaWriter ‘add a new assignment’ screenshot; instructors may base
their assignment feedback on a general analytical or reflective genre, or on
previously modified discipline-specific genres (e.g., reflective pharmacy writing).

types of feedback, and you can learn more about the feedback by clicking
on each of the genres and reading the corresponding descriptions. After
you’ve added a new assignment, the system will automatically generate a
unique assignment code. This assignment code can be given to the students.
Students then login to AcaWriter and subscribe to this assignment by
selecting the tab “Enter my assignment code”, create a draft for their
assignment and keep refining and adjusting the text based on iterative
feedback generated in AcaWriter.
As shown in Figure 1, selecting a genre allows creating a new document to get
feedback for, and displays a short explanation on the right, with a link to More info.
Figure 2. Detailed writing task descriptions with introduction, learning design overview,
description of the analytics genre profile (how the analytics are mapped to feedback provided)
and research regarding its implementation linked to instructor instructions.

This takes the academic to the detailed Writing Activity with Writing Analytics
description as shown in Figure 2.4 It describes the application of the technology to
the particular pedagogic context through expressing a particular learning design
(tasks students undertake), the way the tool is modified for that design (the genre
profile), and any research context and evaluation of relevance to the context. These
foster the sharing of approaches to using AcaWriter, reflecting the need for
technologies to develop both technical and social infrastructure in order to achieve
implementation.
2.4 AcaWriter student experience
On signing in, students are offered tabs to either create a new document choosing
from the generic Analytical or Reflective feedback genres or enter an assignment
code provided to the student by the instructor as in Figure 3. The assignment code
automatically selects the genre allowing the student to receive customized
feedback for their subject.
Figure 3. AcaWriter user interface for adding a new piece of writing.
An information portal for students and staff has been developed, presenting the
above orientation information to different groups.5
3. Empirical Studies
The AcaWriter tool has been implemented across a number of contexts. Here we
describe three such implementations at a large metropolitan university in Australia:
 First, we describe work to implement the analytical genre first in the context of
an undergraduate law class, subsequently adapted to an undergraduate
accounting class, both within the context of specific course assignments. These
implementations are described separately, reflecting the design iterations
undertaken to develop the work (in law), and then transfer it (to accounting).
The final design iteration is discussed for both law and accounting. While both

sites are presented alongside evidence of impact (particularly in legal
education), the focus of the accounting context was to determine if the
AcaWriter implementation could be successfully transferred to another
undergraduate context.
 Second, we describe the work to implement the analytical genre in a
postgraduate research writing context. Specifically, this section provides
evidence to determine if the AcaWriter tool can be successfully adapted and
used in a postgraduate research writing context to teach Ph.D students the
structure of academic writing, in a European/Australian model in which
candidates undertake three years of research, with no coursework
requirements.
 Third, we describe work to implement and evaluate the use of AcaWriter to
provide feedback in the reflective genre, in the context of a pharmacy course.
Across this work, a design-based approach to the research and implementation was
taken (Brown, 1992; Collins, 1992). Design approaches offer “a systematic, but
flexible methodology aimed to improve educational practices through iterative
analysis, design, development, and implementation, based on collaboration among
researchers and practitioners in real-world settings, and leading to contextually-
sensitive design principles and theories’’ (Wang & Hannafin, 2005, pp. 6–7). The
focus of this approach is hence on developing theory and practice through iterative
practical design problems (in our case, improving student writing across
disciplinary contexts) rather than experimental design approaches. Together these
three studies provide evidence for our main research questions:
1. Can the AcaWriter tool be customized to varied learning contexts to improve
student writing?
2. Do students and instructors perceive AcaWriter to be useful to improve
academic writing in their specific context?
4. Disciplinary academic writing contexts: law & accounting
4.1 Disciplinary academic writing context one: Legal education
The law context provides evidence for the question, does the use of the analytical
AcaWriter tool result in improvements in students’ writing?
Learning context and background (law)
The first learning context in which AcaWriter was applied was a core undergraduate
law subject on civil practice, described in detail in Knight et al., (2018). Law places
core emphasis on clear and persuasive writing with the use of appropriate
arguments. The specific type of writing that students were asked to produce in this

subject was argumentative law essays of about 2000 words. These academic essays
outline legal arguments for or against a particular proposition from a list of
assigned, provocative topics, with a key requirement that students adopt a position
for or against the proposition contained in the essay question. In the written
assignment, the instructor had developed a marking rubric consisting of the
following elements: Statement of argument, Statement of essay plan, Identification
of issues, Analysis, Sustained thesis and Original insight, and Engagement with
literature/cases. As part of the submission process, students were required to
submit a self-assessment of their essay. The unit was located in the middle of the
overall course structure, and hence students were familiar with the conventions of
submitting written university assignments. However, early in the marking process,
the instructor observed that students would often overestimate the quality of their
writing, and thus required more support to develop their evaluative judgment
(Boud, 2000) of the criteria. AcaWriter was contextualized to this subject to help
students write better essays using a new feedback genre that aims to help students
self-assess their essays based on the assessment criteria.
Methods (law)
Implementation site and participants (law)
Following the design-based approach discussed earlier, the AcaWriter tool and the
writing intervention for law students were developed over the course of five
semesters in design iterations, with roughly 280 to 400 students in each cohort.
Improvements to the task design and the tool were made in each iteration based
on the feedback from the previous iteration (from instructors and students), until a
stable design was reached and evaluated, discussed in detail in Shibani, Knight, and
Buckingham Shum (in press). This final iteration – discussed below – used the most
recent version of the tool and task design. Evaluation data comes from 90 students
who completed all parts of the writing intervention without technical issues in class.
Implementation and materials (law)
To contextualize AcaWriter feedback for law, the types of rhetorical moves
AcaWriter could identify were mapped to writing criteria elements of a law essay.
With this mapping as the foundation, three tabs of feedback from AcaWriter were
developed: the first tab provided a highlighted report of the key rhetorical moves
for that genre of writing that were found in the students’ writing. A sample analytical
report with highlighted moves is given in
Figure 4.

Figure 4. Sample analytical report for law in AcaWriter highlighting rhetorical moves in the
writing. Top pane indicates the editor (left) and feedback (right); bottom pane shows the
detailed feedback.

The second tab provided further feedback messages for the student based on the
rhetorical moves that are missing in their writing. A screenshot of this second tab
showing feedback messages is given in Figure 5.
The third tab provided examples and mapping to the assessment criteria for
students to learn more about those rhetorical moves. In the context of the law essay,
the rubric elements were mapped directly to one or more rhetorical moves (see
Table 1). The mapping of rhetorical moves to assessment criteria and the crafting of
context-sensitive feedback for the subject were conducted with the instructor who
is a subject matter expert.
Table 1. AcaWriter tags mapped to Law assessment rubric elements with example sentences.
Assessment Rubric
Element
AcaWriter Tag Example Sentence
Engagement with law
and scholarly
literature
B – Background The Concept of good faith has
previously been thought to be a
work-in-progress in Australia
Statement of thesis/
argument
S – Summary This article will trace the origins of
good faith and its development in the
common law.
Essay plan S – Summary This essay contains three parts. The
first part will talk about […]
Critical analysis,
evaluation, original
insight
C – Contrast
E – Emphasis
However, where the obligations are
found in statute and they conflict with
contractual obligations, it is important
to note that the former must prevail.
Drawing together
themes and reaching
a logical conclusion
S – Summary In conclusion, the reasonable
behavior required under the standard
of good faith does not preclude
strong bargaining techniques […]

Figure 5. Sample feedback messages for law from AcaWriter on missing rhetorical
moves.
Pedagogic design and procedures (law)
To introduce students to rhetorical moves in academic writing and the use of
AcaWriter to improve their writing, learning tasks were designed to integrate
AcaWriter into the law subject by co-designing them with the subject instructor.
This was to ensure that the students were well-inducted to use the tool
appropriately and understand its relevance for their subject-related writing. The
task design of the law intervention is shown in
Figure 6.

Figure 6. Task design of writing intervention using AcaWriter in Law.
The writing task in the law context was a one-time activity that was facilitated in class
during a tutorial session. The task started with an introductory reading or a video in
which the instructor explained the concept of rhetorical moves and discourse
markers and their application for the students’ own writing genre. The students
then logged in to an online platform called AWA-Tutor/ AWAT (Shibani, 2018) which
facilitated the writing activities, as described in Shibani et al. (2017).
1. First, students completed a matching exercise by dragging and dropping
elements of the assessment criteria to sentences with rhetorical moves to
understand the mapping between them.
2. Then, they viewed an exemplar text that was marked up with revisions, in order to develop
their understanding of how introducing rhetorical moves could improve the quality of an
essay.
3. They then assessed an exemplar text, using guided questions; a text they revise or redraft
in the next stage.
4. In the next stage, revision, the students revised or redrafted the exemplar provided in step
3. At this revision stage, a between-groups design was adopted in which students either
received automated feedback from AcaWriter to revise their essays, or did not. This design
was implemented in order to study the impact of AcaWriter on student revisions and their
perceptions of the writing intervention. This task design also exposes students to a shared

exemplar text to improve, and allows shared discussion around that text, and thus has
pedagogic value.
5. After this main revision task, students discussed with their peers to provide feedback on
each other’s revised essays from that point, which they applied to further revise the text
in the next step.
6. Finally, students completed feedback questions on the usefulness of the activity and were
provided a guide to working with AcaWriter on their own essays in the future.
This set of tasks was co-designed with the instructors, to address a number of
pedagogic needs. First, they engage students with the assessment criteria and their
application to exemplars. Second, the tasks exemplify and provide space to practice
the activity of revising a text to improve it. Third, the tasks support students in
developing their ability to assess the quality of texts, including through self-
assessment, to understand the quality of their writing and how they can improve it.
Measures of impact (law)
A number of measures were adopted to assess the impact of the AcaWriter tool and
intervention on student writing, and the student’s perceptions of the usefulness of
the learning design (with and without the tool).
Perceived usefulness (law). To study the impact of AcaWriter interventions, student
and staff perceptions of the writing intervention and automated feedback were
studied. Students’ responses to the questions in the final feedback survey (the
penultimate step in
Figure 6) were used to study their perceived usefulness. Students from both groups
(receiving and not receiving automated feedback) answered the questions: ‘How
did you find this task in helping to improve essay writing’ (rating scale of 1-5, where
1-not useful at all and 5-very useful), and ‘Please explain why’. Students who
received AcaWriter feedback in the task were also asked what feedback they found
helpful and what they did not find helpful. Qualitative responses were analyzed
using the questions as a deductive (top-down) guide, and an inductive approach to
understanding the types of responses made and their implications for our design
work, as detailed in prior studies (Shibani et al., in press, 2017; Shibani, Knight et al.,
2019). As such, this approach was intended to understand the typology of
responses, rather than to quantify the qualitative analysis through the development
of a coding scheme. Representative examples are given to exemplify the kinds of
responses made.
Analysis of revisions made (law). To further study the impact of AcaWriter on
students’ writing, we analyzed the revisions students made to a draft that was
provided to them from the online activity described in the Pedagogic design and

procedures section. Analysis of these revisions using the between-groups design
allowed comparison of revised texts between groups, by using a standardized base
or initial draft from which all students worked.
Student revised texts were scored using the standard rubric on which their own
essays were assessed, and which the draft they were given was aiming to address.
These were scored by a subject expert. This assessor was an experienced grader,
with expertise in scoring these assignments developed over a number of semesters,
including through making use of exemplars and tutor-moderation as part of their
own practice. Texts were scored across the criteria in Table 1, on a simplified scale
of 0-3 (0 indicating that a text had been degraded or reduced in quality on that
criterion, 1 indicating no change, 2 indicating minor improvement, and 3 major
improvement).
Rhetorical moves (law). In addition, the number of rhetorical moves detected by
AcaWriter in the revised text was counted automatically for each revised text of
student; the original base text did not contain any rhetorical move.
Results (law)
Perceived usefulness (law)
From 90 students who completed all parts of the activity in law, responses to the
question ‘How did you find this task in helping to improve essay writing’ indicate
the mean perceived usefulness score for students who received AcaWriter
feedback (M = 3.67, SD = 0.79, N = 46, 61% selecting ‘Useful’ or ‘Very useful’) was
higher than those who did not receive AcaWriter feedback (M = 2.95, SD = 0.96, N =
44, 32% selecting ‘Useful’ or ‘Very useful’) during the revision task. The effect of
AcaWriter feedback in the activity was found to be significant in a Welch two-
sample t-test: t (83) = 3.9, p < .001 with a large effect (Cohen’s d = 0.82, 95% CI [0.38,
1.25]).
Further qualitative responses from students in Law showed that many students
found the activity helpful and appreciated the feedback provided to support their
writing.
I find it very useful that there is a new tool which goes beyond grammar,
spelling etc and assists in ensuring the essay will follow a better structure
and include discourse markers. (Law student)
This task has provided a lot of clarity on the impact discourse markers make
on essay writing. These will be extremely useful to apply in my future essays.
(Law student)
Indeed, some students made specific reference to the ways that the feedback
encouraged them to think about their word choice, and how that communicated
their stance (or rhetorical position) to the reader, for example:

I think what is being taught is something I was already aware of. However,
by being forced to actually identify ways of arguing, along with the types of
words used to do so, it has broadened my perspective. I think I will be more
aware of the way I am writing now. (Law student)
There was a variety of feedback that was useful in revising the essay,
including the importance of using rhetorical moves to enhance the quality
of the text, deepen the argument and allow for the reader to obtain a more
developed understanding of what the essay is trying to convey. This drew
my attention to how using discourse markers improve the essay and allow
for the reader to be able to flow on with the essay with ease, comfort and
guidance. (Law student)
Analysis of revisions made (law)
As described above, the AcaWriter interventions included tasks that asked students
to revise a draft that was provided to them. Out of the total 88 valid revised drafts,
45 were from students in an AcaWriter feedback group, and 43 from students in a
group who did not receive feedback.
On average, the AcaWriter feedback group obtained higher scores for the
revised essay (M = 7.98, SD = 2.49, N = 45) than the no-feedback group (M = 7.16, SD
= 1.21, N = 43). Welch two-sample t-test showed borderline significant difference
between groups: t (64) = 1.96, p = 0.055, with a small observed effect (Cohen’s d =
0.41, 95% CI [-0.02, 0.84]), suggesting a need for further evaluation to assess impact.
From the scores of students’ valid improved essays (N = 88), performance bands
were created as Improved, Degraded and Neutral, based on the lower and upper
quartiles of grade summary statistics. If the score was greater than 8, it was classified
as improved, if less than 7, it was classified as degraded, and neutral otherwise. The
proportion of students in the feedback groups falling in those categories is shown
in Figure 7 which indicates a larger proportion of students in the AcaWriter group
improved their text than in the no-feedback group.
A significant difference was found between the two groups’ revision performance
in a chi-squared test: X-squared (2) = 12.75, p < 0.005. This effect can be attributed to
the automated feedback in the AcaWriter feedback group, which helped students
to improve their essays including through incorporating rhetorical moves. Students
in the no-feedback group produced more degraded or neutral essays than
improved essays.

Figure 7. The proportion of students in AcaWriter and no-feedback groups
categorized by revision performance (numbers indicate row-wise proportion % (n),
while the vertical display shows proportion in each revision quality category).
Rhetorical Moves (law)
A further analysis was conducted to compare the number of rhetorical moves
across the two groups in law. The number of moves was measured by running the
essays through automated analysis from TAP. Importantly, the original text included
0 rhetorical moves, yet to meet the needs of the assessment criteria, rhetorical
structures were required, as indicated by the explicit feedback provided in the tool.
Students in the AcaWriter feedback group introduced a statistically significant
higher number of rhetorical moves (M = 3.76, SD = 2.8, n = 45) than students in the
no-feedback group (M = 1.16, SD = 1.02, n = 43) in their improved essays: t(56) = 5.81,
p < 0.001. The effect size was large (Cohen’s d = 1.22, 95% CI [0.75, 1.68]), indicating
that using the automated feedback from AcaWriter helped students to understand
how the use of rhetorical moves mapped to their assessment criteria, and how to
apply them to the provided draft to improve it. These rhetorical moves also had a
moderate positive correlation with the grades discussed earlier: r = 0.46, n = 88, p <
0.0001, showing that there is a relationship between the presence of rhetorical
moves and essay quality.
4.2 Disciplinary academic writing context 2: Accounting education
Building on this work, the second learning context was an undergraduate
management accounting subject. The following section provides evidence
regarding the question, can the analytical AcaWriter tool be transferred to another
undergraduate context?

Learning context and background (accounting)
Throughout their degree accounting students are expected to develop, practice and
refine their ability to communicate in a professional business manner. This subject
focused on enhancing students’ written communication skills, specifically their
ability to ‘convey information clearly and fluently in high-quality written form
appropriate for their audience’. This skill development was targeted through
formative tasks associated with an individual writing assignment of about 1500
words. In the current study context, students were asked to write a business report,
proposing and substantiating a definition of performance for a specific organization
(selected from a pre-defined list). The report was assessed against four elements of
assessment criteria: 1. Organizational analysis, 2. Defining performance, 3.
Justification of their definition of performance, and 4. Written communication. The
cohort comprised a diverse mix of local and international students, with varying
levels of English language ability. To help students improve their written
communication in their business reports, a feedback genre in AcaWriter was
contextualized for the subject, involving co-design with accounting academics,
based on the learning and technology design conducted in law.
Methods (accounting)
Implementation site and participants (accounting)
The accounting subject had roughly 400 to 600 business students in each cohort and
ran every semester. The cohort had a wide mix of local and international students,
who had varying levels of English ability. Two instructors from the subject worked
with the researchers to co-design AcaWriter feedback and writing intervention for
students. They used the validated task design from the law intervention and
transferred it to their new disciplinary context. Data presented in the paper comes
from 302 students (all receiving AcaWriter feedback) who completed the online
activity in week 1 of the task design, and the marks of 403 students in the overall
cohort.
Implementation and materials (accounting)
AcaWriter feedback was contextualized for Accounting by mapping the rhetorical
moves AcaWriter can identify to criteria elements of the business report students
wrote for the subject. Similar to law, three tabs of feedback from AcaWriter were
developed for the accounting context. The first tab displayed the analytical report
of the student’s writing with the key rhetorical moves highlighted in the writing as
shown in Figure 8.

Figure 8. Sample analytical report for Accounting in AcaWriter highlighting rhetorical
moves in the writing.
Feedback to improve their writing was provided in the second tab by showing
missing rhetorical moves that were deemed important for a business report.
Examples of phrases from the business writing context that can highlight those
moves were provided in the feedback messages as shown in
Figure 9. Sample feedback messages for Accounting from AcaWriter on missing
rhetorical moves .

Figure 9. Sample feedback messages for Accounting from AcaWriter on missing rhetorical
moves
The third tab in AcaWriter for Accounting provided the mapping between rhetorical
moves and the assessment criteria elements, and reflective questions that
encouraged students to reflect on their use of rhetorical moves, as shown in Figure
10.

Figure 10. Mapping of AcaWriter rhetorical moves to assessment criteria for
business reports.

Pedagogic design and procedures (accounting)
As in the law context, in the accounting context, the AcaWriter tool was introduced
in a writing intervention with tasks to induct students into its effective use. The
writing tasks were spread out over six weeks (half a teaching session) as illustrated
by the task design in Figure 11.
Figure 11. Task design of writing intervention using AcaWriter in Accounting.
In week 1, students completed an online homework activity. This activity was similar
in structure to the law writing intervention, but the students completed the tasks
independently at home before coming to class. In the activity, students were
provided with an example text based on students’ work from the previous semester.
Working with this text, students reviewed instructor feedback and comments,
assessed the work using the assignment marking rubric, and practiced using
AcaWriter by reviewing feedback and making changes to the text. The instructor
explained the use of rhetorical moves in business writing using a recorded video
embedded as part of the task. This online activity thus inducted students into the
use of AcaWriter feedback for their disciplinary context, and was the main part of
the writing intervention. In the following tutorial, the students shared their
experiences of using AcaWriter with their peers. Students also used AcaWriter
feedback to develop their own reports for the assignment. Having prepared a
mandatory at draft in Week 4, students then used AcaWriter, in conjunction with
several other tools and feedback sources (Turnitin, peer and self-evaluations) to
improve their written communication in their assignment prior to submission in
Week 7.
Measures of impact (accounting)
Perceived usefulness (accounting). The same measure of perceived usefulness was
used in accounting to study student feedback (although there is no comparison
group in this context since all students received AcaWriter feedback). Perceived
usefulness provided us with an understanding of student perceptions about the
tool and the intervention.
In addition, the impact of the tool and design work was investigated through
instructor interviews to understand the transfer of the tools and tasks from the law
to the accounting contexts. While sample quotes demonstrating impact are
provided in the current paper, a comprehensive study and extended quotes on

instructor perspectives are reported in other work (Shibani et al., in press), in which
a deductive-inductive approach to develop themes is described (Eisenhardt, 1989;
Thomas, 2006), as with the survey data the aim of this approach is not to create an
exhaustive coding scheme, but to provide an interpretive typology to understand
the interview case data.
Final assignment score (accounting). In addition, an analysis of marks scored by
students in the assignment was conducted to provide an indication of writing
performance. The final marks scored by students in their assignment contributed
to overall credits to pass the subject and was used as the performance indicator.
The instructors and tutors scored these assignments based on the assessment
rubric for business reports and moderated the results between the markers for
consistency of marking.
Since the main writing intervention described above was an online homework
task, not all students completed it. Whether students completed the activity or not
was used to study the association between completing the intervention and the
final assignment mark scored by the student. To complete the online activity,
students were required to complete all tasks in the online platform AcaWriter Tutor
(AWAT), which required watching the introductory instructor video, viewing an
exemplar marked up by instructors, assessing a sample report, improving it using
AcaWriter feedback, and providing feedback on the task.
Results
Perceived usefulness (accounting)
In accounting, data was analyzed from 302 students (all receiving AcaWriter
feedback) who completed the online activity in week 1. Analysis indicated that 66%
of students selected 4 or 5 (highest) in relation to the perceived usefulness score
(M = 3.8, SD = 0.90) showing that many students found the activity useful to improve
their report writing.
Further qualitative responses showed that many students found the activity helped
to improve their writing and appreciated the feedback provided:
By doing this exercise, it shows me a clear idea on how we could approach
our writing for the upcoming individual report assessment task. Not just that,
but will generally help in gaining confidence in writing a clear and concise
business report future uni projects or for work. (Accounting student)
It's like having a tutor or another person check and give constructive
feedback on your work. (Accounting student)
It was interesting to see what AcaWriter was able to pick up and the feedback
it gave allowed me to edit the sample doc a lot faster than normal.
(Accounting student)

Final assignment score (accounting)
To further study the impact of the AcaWriter intervention, marks that measure
student performance in their writing assignment were obtained from the
instructors. These marks were available for all students of the cohort. The
assignments were scored from 0 to 30 for written communication and this mark was
used for analysis, N = 403, M = 20.67, SD = 4.87, with a large range from 5.5 to 29.5.
Results show that students who completed the online activity with AcaWriter
scored higher marks for their assignments on average (M = 21.15, SD = 4.87, n = 274)
than students who did not complete the activity (M = 19.64, SD = 4.72, n = 129).
Welch two-sample t-test showed a significant difference between groups: t(258) = -
2.98, p < .001, with a small observed effect (Cohen’s d = -.31, 95% CI [-0.52, -0.10]),
although caution should be drawn in interpretation given the potential
confounding factor of student motivation.
Instructor feedback on AcaWriter in both law and accounting
The instructors who taught the subject and marked students’ writing (in both
academic writing disciplinary contexts) also thought AcaWriter and the writing
interventions helped them to efficiently provide individual formative feedback to a
large cohort of students:
I got a lot of positive feedback from students about the fact that it was
automated and I think that’s an important part of the story because what
we’re talking about is saying to the students, paste your essay into this tool
and then it’s going to give you some feedback because your lecturer doesn’t
have enough time to do that for you. (Law staff)
We can’t afford to give formative feedback when we have 400 students
because it already takes us maybe about 20 hours to mark one class of these
assignments and so we can’t have the tutors spend that time again giving
formative feedback. So, we had to do it in a way that is time efficient.
(Accounting staff)
These instructors also noticed how the provision of this formative feedback
resulted in increased performance in their students’ writing:
I noticed a change and it was profound that suddenly the discourse markers
were everywhere… And suddenly I noticed their essays were better. And
they will be better in court and they will be better lawyers for it. (Law staff)
Overall, since we’ve been working with CIC around written communication over
the course of the last four of five semesters, we have seen a marked improvement
in students’ written communication. Overall, their individual assignment pass-rate
is going up... We are seeing improvements in the number of students who are either

meeting or exceeding the expectations around written communication (Accounting
staff)
5. Higher Degree by Research (HDR) context
As well as for undergraduate reports and essays, the analytical AcaWriter tool has
been adapted for use in a postgraduate research writing context to provide students
with ad-hoc feedback on the structure of their academic writing. In this context,
AcaWriter not only identifies the presence or absence of rhetorical moves but also
provides feedback on the order of the moves as is expected in academic writing.
This section describes and provides evidence of the impact of the implementation
in this learning context.
5.1 Learning context and background (HDR)
It is often assumed and expected that higher degree research students know ‘how
to write’. However, supervisors report that writing is a challenge for research
students (Aitchison et al., 2012), that students lack familiarity with, and struggle to
learn, disciplinary writing and rhetoric (Aitchison et al., 2012; Maher et al., 2014;
Thompson, 2016), and that students with English as a second language have
particular difficulty understanding the rhetorical nature of research writing
(Paltridge & Starfield, 2007). This writing includes both the dissertation or thesis
document itself, as well as increasing pressure for junior scholars to produce
scholarly peer-reviewed outputs (Catterall et al., 2011; Lee et al., 2010; Lee & Kamler,
2008). Thus, while writing is fundamental to the success of research students, many
lack skills in applying their discipline-based discourse conventions with few
possessing requisite experience in writing for an academic audience at the start of
their candidature (Torrance et al., 1992).
For HDR students to succeed in their degree, they need to position themselves
in their discourse community by conforming to disciplinary writing conventions.
One way for students to analyze the writing conventions used in the introduction
sections of articles within their discipline is by using the Create a Research Space
(CARS) model developed by Swales (1990; Swales & Feak, 2004). The CARS model
allows students to explicitly see how authors achieve an authoritative voice in their
introductions. Each of the CARS moves has a specific rhetorical function that is
achieved through appropriate linguistic expressions. Identifying the rhetorical
function and language choices used to achieve the rhetorical move explicitly
conveys the link between the rhetorical goal and the language used. By exposing
this link directly students are able to see how rhetorical moves and language used
are important in situating themselves within their discourse community.
To provide students with formative feedback on their research writing, a
feedback context was developed based on this Create a Research Space (CARS)
model in AcaWriter. The CARS model was chosen as it systematically shows the

connection between rhetorical intent and the language used to achieve that
rhetorical function. The CARS model has also been used to teach HDR students the
structure of research article introductions in pedagogical contexts ( Cai, 2016;
Kuteeva, 2013),including – as described in this issue – in other learning technologies
(see, Cotos, Huffman and Link, 2020). At present, AcaWriter provides feedback on
students’ research article abstracts and introductions.
Methods (HDR)
Implementation site and participants (HDR)
Pilot work has been conducted to evaluate the HDR context. This intervention was
piloted with groups of three or four students in three separate sessions. Overall, 12
students participated in the study, helping us to gain a preliminary understanding
of whether the writing analytic tool and learning design were useful for students.
The students were at different stages of the doctoral journey. Five students were
within the first year of their candidature (which, follows the UK/Australian model of
a three year doctorate, with no coursework requirements), six were in their second
year, and one in their final year. Eleven were international students and one was a
domestic student. Three students withdrew from the intervention after the first
workshop.
Implementation and materials (HDR)
AcaWriter’s parser was mapped to the CARS model to create the AcaWriter CARS
parser (see Table 2) to help students identify rhetorical moves in research article
introductions.
AcaWriter’s moves were mapped to the CARS moves by examining their
communicative functions, and comparing them to the three CARS rhetorical moves.
The mapping was validated by first conducting a genre analysis on various research
articles annotating the CARS moves. After this, the emerging CARS parser in
AcaWriter was tested to see that it identified the same moves. Next, the Elsevier
STEM corpus (Elsevier, 2015) was analyzed using the parser, where sentences were
checked to see which were tagged and if they corresponded with the CARS moves.
Once the testing was complete, a rules-based system was employed to the CARS
parser so that feedback would be generated when the moves do not follow the
same order as the CARS model and if moves are missing.
To help doctoral students improve their scholarly writing, it is essential that
AcaWriter’s feedback is clear, specific, and actionable so that students understand
what they need to do in order to revise and improve their text. The analytical report
that is generated by AcaWriter highlights the moves that students have written (see
Figure 12). Each AcaWriter tag is assigned to a color that corresponds to each CARS
move used in the mapping. This was done so that students are able to clearly

distinguish which sentences matched back to the CARS moves. Highlighting the
moves in students’ texts encourages them to think critically about their writing.
Table 2. AcaWriter tags mapped to CARS moves (adapted from, Swales & Feak, 2004)
CARS Rhetorical Moves AcaWriter Tags
Move 1 – Establishing a research territory:
by showing that the general research area is important,
central, interesting, problematic, or relevant in some way
(optional)
by introducing and reviewing items of previous research in
the area (obligatory)
E - Emphasis
B - Background
Move 2 - Establishing a niche:
by indicating a gap in the previous research, raising a
question about it, or extending previous knowledge in
some way (obligatory)
C – Contrast &
Q – Question
Move 3 - Occupying the niche:
by outlining purposes or stating the nature of the present
research(obligatory)
by listing research questions and hypotheses (optional)
by announcing principle findings (optional)
by stating the value of the present research (optional)
by indicating the structure of the research paper / thesis
(optional)
S – Summary
N – Novelty
S – Summary
For example, if a sentence is tagged as Move 2, but the student did not intend to
express move 2, it allows to them go back and see if they have expressed their
meaning effectively, and in turn, revise the sentence. Students are also given clear,
actionable feedback messages when their texts are missing moves and if they are in
a different order to the CARS model. These feedback messages prompt students to
revise their texts specifically related to each move. An explanation about each move
is given, alongside guidance on language that students can use to achieve the
rhetorical move (see Figure 13) with additional guidance provided in the Resources
tab. AcaWriter supports students writing practice as they go through the recursive
nature of writing, guided by feedback to help them achieve their communicative
goals.
Pedagogic design and procedures (HDR)
A writing intervention using AcaWriter was designed to help students understand
the concept of rhetorical moves and apply the moves in their writing.

Figure 12. AcaWriter's highlighting of rhetorical moves in a research abstract, colour-coded
by moves from the CARS model.

Figure 13. AcaWriter's feedback on the presence, absence, and sequencing of CARS moves.
The intervention consisted of two face-to-face workshops for doctoral students on
how to write an introduction and abstract using the CARS model and AcaWriter.
The workshop followed the teaching and learning cycle (TLC) of decon-
struction:building the field, joint construction, and independent construction (Rose
& Martin, 2012; Rothery & Stenglin, 1994). The TLC was applied in the sessions as it
raises student’s rhetorical awareness, which is needed in HDR research writing,
through a sequence of structured connected stages and learning activities. The TLC
approach allows students to become independent writers as they go through the
workshop stages and scaffolded activities.
The first workshop involved discussions about writing in general, writing abstracts
and introductions, and writing in their disciplines. In the same workshop, a mini-
lecture of the CARS model was given to students where an introduction was
deconstructed according to the CARS model. Once the facilitator was confident
that the students understood the model, students were then asked to annotate
abstracts and introductions according to CARS where more discussion took place
about the moves the students found and if they followed the CARS model. The next
activity involved students critically analyzing an introduction from a sample article,
they were then asked to submit this introduction to AcaWriter for feedback and
reflect on whether they agreed with AcaWriter’s feedback.
The second stage of this activity is where joint construction took place, as
students revised the introduction according to their analysis and AcaWriter’s
feedback in pairs or individually. In the final activity in this session, students
submitted an introduction or abstract from their discipline to AcaWriter, where

again they reflected on whether AcaWriter identified the moves or not, or if they
found moves that AcaWriter did not detect. In the second workshop, students
independently constructed their own abstract or introduction in AcaWriter, and
requested feedback when needed.
Measures of impact (HDR)
Perceived usefulness (HDR). The researcher conducted a focus group and interviews
with students who participated in the workshops, regarding their experience of
using AcaWriter. The semi-structured focus group (n = 4) and interviews (n = 3) were
recorded, verbatim transcribed and coded using an inductive (bottom-up)
approach.
Impact on writing (HDR). From the pilot workshops described above, pre and post-
tests were conducted at the start and end of the workshops to measure student
understanding of the CARS rhetorical moves (a strategy also adopted by Anthony &
Lashkia, 2003; and, Cotos, 2010). Both tests contained two tasks. In the first task,
students received a list of 13 sentences and were asked to annotate each sentence
for the CARS moves. Students were given a score of 1 if they annotated the sentence
with the correct move and a score of 0 for an incorrect move. The second task asked
students to read an introduction and annotate the introduction for the three CARS
moves. Scores of 1 were given if they identified the move that each section
belonged to correctly. If students annotated part of the section correct and the
other incorrectly they were given 0.5 and if the move was completely incorrect they
were given 0. In both cases different, but equivalent, sample materials (sentences,
and introductions) were used pre- and post-test. Students were not provided
feedback on their results in the pre-test (as the workshop directly addressed the
task aims).
Results (HDR)
Perceived usefulness (HDR)
Common themes emerged from the data: thinking about writing, structure,
planning, and usefulness of automated feedback. All students found the
intervention and AcaWriter useful. Students stated that the automated feedback
messages and highlighting helped them reflect on their writing and its rhetorical
structure:
It makes you think about if you have the moves […] it gives you some hints,
a message, ok you’re missing this (Participant 3)
Some students described AcaWriter as a helpful tool for planning:

It was useful for planning, seeing if your writing is in the right order, in an
understandable order, so not only like having the moves, but you have a
sequence that makes sense (Participant 4)
I think one of the main contributions of AcaWriter is to help you structure
your thinking (Participant 1).
The same participant mentioned that AcaWriter helped them to critically analyze
their writing
I think it was very useful to use a piece of writing of my own and then when
the software gives the feedback, maybe you think you’re having the moves,
you have the right structure but then it, it happened in my case the software
told me ok, you’re missing move 1, but I thought that it was there...I think in
that sense, it challenges you in the way that you’re thinking (Participant 1).
Impact on writing (HDR)
While most students reported that AcaWriter helped them learn the CARS
rhetorical moves, some stated that they needed more time to become more familiar
with AcaWriter and the CARS framework, and thus longer interventions are planned
for future evaluation.
As three people left the study after the first session, only data from the
remaining students (n = 8) were included in the pre and post-test. In task 1
(annotating 13 sentences), the average number of sentences correctly annotated by
students increased from 8.00 (SD = 3.25) pre-test to 9.50 (SD = 3.25) post-test. In task
2 (introduction annotation), students (n = 8), improved their scores from 1.81 (SD =
0.59), to 2.00 (SD = 0.76). However, in both tasks not all students improved their
performance individually. These results suggest that understanding rhetorical
moves used in research writing is a challenging skill to learn and that more time is
needed to learn to apply the moves, an evaluation that is underway.
5.2 Reflective writing context
While the previous sections described and evaluated the use of the analytical genre
of AcaWriter, this final learning context describes and evaluates whether AcaWriter
can successfully be used to provide effective feedback for the reflective writing
genre.
Learning context and background (reflective writing)
Reflective practice is an important feature of lifelong learning (Ryan, 2011). As Boud,
Keogh, and Walker (1985) note:

Reflection is an important human activity in which people recapture their
experience, think about it, mull over & evaluate it. It is this working with
experience that is important in learning (p.43)
Thus, reflection can help learners develop an understanding of their own
experience of learning (Boyd & Fales, 1983). This makes reflection a particularly
important element of degree programs in which experiential components of
professional practice play an important role, notably teacher education, nursing,
pharmacy and allied health disciplines, engineering, and architecture. Reflective
writing tasks are advocated as an effective way to assess reflective practice in tertiary
contexts (Griffin Jr et al., 2010; Ryan, 1988), making them a key component of
reflective practice as students document their experiences through journals, blogs,
or reflective essays. Reflective writing allows students to express how they have
learned from their experiences and to connect theory with practice.
However, guiding reflection is challenging, with concerns around
operationalizing deep or surface reflection (Hatton & Smith, 1995; Lew & Schmidt,
2011; Sumsion & Fleet, 1996), and developing language that is shared between
students and instructors across disciplines to teach and asses the writing (Ryan,
2011). Indeed, a review of reflective writing teaching across disciplines in an
Australian institution (Ryan & Ryan, 2013) found that academics struggle to explain
what they are looking for in quality reflections, and thus encounter issues in guiding
students in completing their reflective writing tasks.
The development of the reflective writing aspect of AcaWriter was theoretically
informed by the above research on the use of reflection in education, and research
on the linguistic characteristics of reflective writing identified through analysis of
reflections across a number of disciplines (Gibson et al., 2017). A key aim in
developing the tool was that it provide pedagogic support contextualized to the
disciplines into which it was applied. As such, the distillation of the theory was also
contextualized by discussions through a co-design methodology with teachers with
an interest in reflective writing representing a range of disciplines (e.g. Pharmacy,
Accounting, Engineering). The primary pedagogical aim of providing actionable
feedback to students set the direction for this work. The result generated a
reflective framework that both synthesized theory in reflective writing and that took
into account practical disciplinary approaches to assessing and supporting
reflection.
The reflective framework (see Figure 14) visualizes five levels describing
increasing reflective depth from Impression and Interpretation through
Internalisation and Integration, to Intention. The linguistic features were also
ordered from thoughts and feelings through challenge and self-critique to potential
solution and learning opportunity. The framework also incorporated elements to
assist with using it for actionable feedback. The linguistic features were simplified

into 3 main categories: context (thoughts and feelings), challenge (challenge and
self-critique), and change (potential solution and learning opportunities).
Further, guiding questions were provided that aimed to shift a student from the
less reflective impression and thoughts through to the more reflective intention
and learning opportunity. For example, “How is this a problem that challenges me?”
or “How can I learn from other perspectives?”. Each element of the framework was
designed to capture common aspects of respected theories in a way that was
relevant to the teachers who might use it.
Figure 14. Reflective Writing Framework, from Gibson et al., (2017, p. 3).
Methods (reflective writing)
Implementation site and participants (reflective writing)
Pharmacy students are required to reflect on their clinical practice in relation to
their academic learning. As part of the course requirements, students are required
to keep a reflective e-portfolio and document weekly reflections derived from their
weekly clinical placement experiences. Data comes from 59 Master of pharmacy
students who participated in a one and a half hour AcaWriter workshop.
Implementation and materials (reflective writing)
The implementation involved assessing how key linguistic features of the narrative
could be identified in students’ written reflections. A two-pronged approach was
adopted. First, reflective rhetorical moves were developed to map the reflective
writing framework of Gibson et al., (2017), in a process similar to the analytical
parser of AcaWriter (described above). Secondly, expression analysis derived from
earlier analysis (Gibson et al., 2016) was used to capture key features of reflection
such as the use of affective vocabulary, and first-person personal pronoun centric
expressions of knowledge and belief or self-critique.

The detection of reflective rhetorical moves was based on the concept matching
analysis framework described in the introduction, developed to directly relate
sentences to the three categories (context, challenge, and change). Matching
sentences were annotated in AcaWriter with a blue square (context), pink circle
(challenge) and green triangle (change). A fourth component highlighted
sentences that referred specifically to the author (i.e., the self). If these self-
references co-occurred with one of the 3 categories, the sentence was emboldened.
Figure 15 illustrates this visual grammar in AcaWriter, along with some sample
feedback statements.
Expressions were identified using rules, based on finding groups of words that
involved first-person personal pronouns (i.e., I, me, my) and then looking for
additional epistemic verbs (think, believe, wonder, guess) or modals (could, should,
would). Additionally, words with affective strength were also identified. These
expressions and/or words were annotated in the text via styles of underlining: plain
represented expressions indicating belief, learning or knowledge; dotted
represented expressions of self-critique; and dashed represented words associated
with strong feelings.
Pedagogic design and procedures (reflective writing)
The purpose of the 1.5-hour AcaWriter workshop was four fold: (i) to provide
exemplars of reflective writing at different levels; (ii) introduce students to the
capabilities of AcaWriter for immediate formative feedback on their writing tasks;
(iii) facilitate self and peer reflection; and (iv) introduce a validated reflective rubric
(Lucas, Smith, et al., 2019) which aligned to the elements of reflection such as those
derived from the AcaWriter parser output.
In the Masters, students were asked to document in an e-portfolio their experiences
from their weekly clinical placements that included one critical incident (interaction
with patient, carer, family member, another healthcare provider and/or a situation
that has arisen as a response to a patient enquiry) from the clinical environment and
reflect on the impact this has had on their learning. They were asked to draw from
their previous personal and/or professional experiences with special attention to
how this learning may have challenged their own personal beliefs. Prior to the
AcaWriter workshop, students were asked to bring along one or two reflective
statements derived from their e-portfolios for use during the interactive workshop.
During the workshop, students were first provided with exemplars of poor and
high-quality reflections. These were discussed as a class, following which students
were asked to identify aspects of the reflections that address the rubric criteria. The
students then viewed AcaWriter feedback for these exemplars, before working
individually to view feedback on their own writing. They then engaged in peer
review of their reflections with the AcaWriter feedback.

Measures of impact (reflective writing)
Perceived usefulness (reflective writing). An earlier version of the reflective genre of
AcaWriter provided a prompt to users after each prompt, with the question: “Did
Figure 15. The visual grammar of reflective feedback and sample feedback.

you find the feedback on your writing helpful?” on a 1-7 scale, where 1 is low and 7
high, alongside a comment box for written feedback.
Impact on writing (reflective writing). To analyze the impact on writing, a surface-
level content analysis of revisions made between feedback requests was conducted
to investigate the number of reflections submitted to AcaWriter, and where the
same reflection was revised and resubmitted, what the nature of those revisions
was.
Results (reflective writing)
Evaluation of the reflective writing parser deployed in an pharmacy context
(reported in, Gibson et al., 2017; Lucas, Gibson, et al., 2019), indicated that generally
students responded positively, with some evidence of impact on writing.
Perceived usefulness (reflective writing)
As reported in Gibson et al., (2017), of 59 students, 30 used the tool to support their
writing. Of this thirty, 18 (60%) posted more than once, making a total of 120 posts
to the tool. Each post was prompted to provide student feedback to the question
“Did you find the feedback on your writing helpful?” on a 1-7 scale; 63 posts were
responded to, with 54 (85.7%) on the 5-7 (positive) range of responses.
As reported in Gibson et al. (2017), qualitative responses to the tool were also
largely positive, noting:
I was fascinated by how it works and can see its implication in future, to
determine which phrases need more work/ which can be improved. (Student
A)
It details where I’ve made reflective statements and shows where I can
improve as well as add to and fill in aspects to which I have not confirmed.
(Student B)
Prompted me to follow through with the refection to the last step of the
process - I had written about my thoughts and feelings, discussed
challenges, but had not followed through with reflecting on how this can
lead to change…. The reports also direct me to write more personally, using
language that evokes emotion, and less descriptively (Student C)
A comment by one student indicated that they saw the potential for AcaWriter to
assist them in improving their grade, and even suggested that it should provide a
grade (a strategy we have not focused on, instead providing formative feedback):

This system has allowed me to identify the strengths and weaknesses of my
reflection, highlighting on what criteria I have addressed and which ones I
haven’t. I wish there was feedback on how I could improve to get full marks
and wish this reflection gave a mark at the end. (Student D)
No Pharmacy students left clearly negative comments. However, some early use of
the software by students from other subjects highlighted a theme in their more
negative comments: criticism either at what the software did not do or towards a
lack of clarity in what needed to change. These are comments that we have sought
to address by creating rule-based actionable feedback statements based on the
presence and absence of rhetorical moves. For example “Doesn’t elaborate on the
features that are lacking and often they appear there but are not recognized. Good
way of highlighting other points.” (Student E). “it’s not clear what needs improving”
(Student F) and “comments are not clear enough” (Student G). “I don’t understand
what [AcaWriter] reproach to my work. It is said that there isn’t a good balance but
in the text I can’t see how… not clear” (Student H).
Thus, while students were generally positive about the tool, this data indicates
that they wanted more actionable feedback; subsequent versions have introduced
such formative feedback. A later pilot reported similarly positive results (reported
in, Lucas, Gibson, et al., 2019), with students indicating on a scale that the tool
supported their self-directed learning, was relevant to pharmacy professional
practice and enhanced their confidence in their ability to write their submissions.
Impact on writing (reflective writing)
Further analysis of the actionability of the feedback investigated the revisions made
between feedback requests to identify how the feedback provided might be
influencing the revisions made. Five of the 18 repeat posters appeared to modify
drafts, with two of these submitting multiple versions of the same draft text. The
remainder (13 users) submitted different reflections for each of their posts, which
aligns with the needs of the assignment for which they were working.
With the exception of addition of new information, most draft modifications
appeared to improve the quality of the reflection. For example:
I made sure to be understanding and not force the customer to purchase
just for the sake of receiving a sale (ethics, social responsibility).
was changed to:
Initially I was confused as to why this would be an issue like isn’t it exactly
the same thing? But for good pharmacy practice, I decided to be
understanding and not force the customer to purchase a product just for the
sake of receiving a sale (ethics, social responsibility).

The first of these had no sentence tagging, whereas the second was tagged with a
pink circle (Challenge). Students also changed their writing to introduce how they
felt about a situation. For example, “This situation didn’t sit well with me.” was
changed to “This situation didn’t sit well with me, I felt as it these patients didn’t
receive the best care possible.”
6. Discussion and Future Directions
Across the contexts that AcaWriter has been deployed, we have co-designed
learning tasks that integrate the tool into meaningful teaching and learning
activities to provide formative feedback to students specifically targeting their
discipline. The sections above describe these integrations and implementations,
and the evidence of impact thus far. There is some evidence of impact – most
notably in the law and accounting contexts – which suggests that students who
receive feedback from AcaWriter are more likely to improve draft texts and to
incorporate rhetorical moves into their writing. Automated writing feedback tools
like AcaWriter can thus provide additional writing support to students within and
outside the classrooms in ad-hoc ways. However, further work is required to
understand how the tool supports learning and how it might be developed further.
In particular, impact on writing is assessed by investigating changes in a confined
text-revision task (law, accounting) or on a single task (reflection), that in the legal
and accounting context was assessed by tutors using their established rubric, rather
than a psychometrically validated measure. Further work is also required to
understand how learning from a single task and intervention transfers to other
contexts (and writing tasks). Other changes, including the incorporation of
rhetorical moves, are measured using the automated tool; while such measurement
provides useful insight, further work should be conducted to ensure (1) that the
moves added are appropriate, and (2) to understand other changes made by
students, including the addition of rhetorical structures that may not be
automatically detected.
In the HDR context, measurement of improvement is challenging given the
distinct nature of each student research project. In this work a measure of how well
students could identify and annotate rhetorical moves was piloted, however further
work is required to validate this evaluation at larger scales and using a set of test
samples that would allow for random selection of items, counterbalancing, and so
on. Further work to establish robust tasks and reliable assessment, both of which
should be authentic to the classroom context, would strengthen this evaluation.
While the perceived usefulness data indicate positive responses to both the task
design and tool, and is supported by a comparison between groups who used the
tool and those who did not (in law), there is potential for response bias in the use
of such self-report instruments. Finally, although there seems to be a relationship
between engagement with the tool and tasks, and subsequent scores, motivation is

a confounding factor in such analysis; students who complete the learning tasks
may simply be more motivated to work on their assignments.
Both students and instructors recognize the imperfect nature of the feedback
provided, but feedback (within ‘perceived usefulness sections’) indicates that they
nonetheless believe it supports reflection on writing, revision, and attention to
important details, including through disagreement with the tool; they thus might
be said to ‘embrace imperfection’ (Kitto et al., 2018). The design approach we adopt
supports this iterative work, through supporting stakeholder engagement, and
ensuring that use of the tool can align closely with existing good practices to
support learning.
In the following, we summarize the impact of the AcaWriter tool, highlighting
future directions, and in particular making explicit recommendations for the
community.
Recommendation 1: Develop shared corpora for testing writing analytics models
The AcaWriter tool provides formative feedback to students engaged in scholarly
analytic and reflective genres of writing. The rhetorical expressions identified by
the tool are well established in the literature as communicative acts that are
important across a range of scholarly writing. In our development of the AcaWriter
tool and underlying parsers, we have evidence that these moves feature in both
published academic works (e.g. through analysis of open access corpora of writing),
and student writing. Analysis of student writing has included both the British
Academic Written English (BAWE) corpus (Nesi et al., 2004) and student writing at
our own institution. However, the former consists only of work of relatively high
quality, while the latter cannot be made public. Work to investigate relationships
between rhetorical features, other features of the text, and outcome variables
(notably criterion-referenced grades), is hampered by the lack of established
graded corpora of student writing of varying qualities, across established tasks (see
recommendations 2 and 4). Sharing such a corpus will foster collaboration in
tackling the development of writing analytics to provide formative feedback to
students.
Recommendation 2: Develop a set of established shared tasks for evaluating writing
analytics interventions
While identifying predictive relationships between text features and outcomes is
one significant method for evaluation, a – perhaps clearer – approach is through
identifying change associated with direct interventions. Across the implementation
contexts described above, there is clear evidence of impact, although further
evaluation is required in each case. Implementing and integrating for impact in
authentic learning contexts, rather than in lab-based studies, has been a significant
contribution of the AcaWriter program of work, demonstrating the potential of co-

design and augmentation of existing practices in tool development. In our
discussion, we highlight a number of approaches we have taken to a task-oriented
evaluation, including asking students to revise a text that they are provided. These
approaches allow ready comparison of conditions, as well as holding a pedagogic
benefit in their own right. We thus see significant potential in developing a shared
library of such evaluation tasks. Further work is required to evaluate the impact of
AcaWriter across deployment contexts and tasks, and to design evaluation tasks
appropriate to the learning context, particularly in a country without widespread
adoption of standardized college writing assessments (Knight et al., in press).
Recommendation 3: The community should work together to develop shared open-
source writing analytics systems, where relations between textual features and feedback
can be mapped, and adopted or adapted to particular assessment tasks
The open-source and modular nature of the AcaWriter tool have been a significant
asset to us in developing and implementing the tool across contexts. Because of
this open-source nature, elements of the AcaWriter suite have been adopted at
other institutions, as well as being used in tutorials designed to teach academics –
both more and less technical – about the potential of writing analytics, using Jupyter
notebooks that display code blocks alongside explanatory rich text and the outputs
of the code (Knight et al., 2017; Shibani et al., 2018)6. There is significant potential
for this approach to develop capacity across institutions, using an extensible and
modifiable system. However, the current feature set is limited in scope, largely
based on rhetorical moves, and thus work is needed to build a shared library of
feature-feedback relationships that can be adopted and adapted for particular
assessment tasks and wider learning designs.
Recommendation 4: A shared library of learning designs should be developed to provide
patterns for the adoption and adaption of writing tools
Underpinning all of our work has been a learning design approach, in which we
have augmented existing successful practices of formative assessment to
contextualize the AcaWriter tool to particular assessment tasks by mapping textual
features to feedback structures. This has had an impact, and instructors are positive
about this approach to integrating writing analytics in practice. To scale this
approach to further contexts, we intend to develop a user interface for the
middleware component of the AcaWriter system that more readily allows end-users
to map features to feedback. However, in doing this, further research on the user
experience and learning design concerns of such an approach will be required; the
success of implementing AcaWriter is in its integration with learning design.
As already described, we have begun to develop a library of open access
(Creative Commons licensed) learning designs, which demonstrate to educators
how AcaWriter can be aligned with different kinds of student writing tasks and

assessment criteria. This is how we conceive ‘contextualized scalability’ (Shibani,
Knight, et al., 2019). As with other design pattern approaches, by providing
‘templates’ for educators to adopt or adapt, we propose that a learning design
exchange will help accelerate the rate at which the community can develop and
refine effective uses of the tool.
In conclusion, the AcaWriter project has taken a holistic approach to co-
designing formative feedback for students, using NLP technologies and learning
design to do so. We have detailed how we are assessing the impact of the tool once
it is integrated into a well-designed writing activity and aligned with assessment
criteria. To move research and impact forward in this space, we derive four key
recommendations from our own work, including the need for shared datasets,
evaluation tasks, technical approaches, and learning designs.
Acknowledgements
Our thanks to colleagues and students for their involvement in this research. The
empirical work report draws on prior work as noted, alongside the PhD theses of
Antonette Shibani (within the section "Disciplinary academic writing contexts: law
& accounting", including reproduction of figures), and Sophie Abel (section Higher
Degree by Research Context). Parts of the work were supported by grants from the
Australian Technology Network (ATN) and UTS Teaching and Learning grants with
the support of DVC-Education Professor Shirley Alexander.
Notes
1. Academic Writing Analytics project: https://cic.uts.edu.au/tools/awa
Open-source release: https://cic.uts.edu.au/open-source-writing-analytics
Educator resources: http://heta.io/resources and illustrative Jupyter notebook
https://github.com/uts-cic/ALASI2018-WritingAnalyticWorkshop
2. For Athanor technical tutorials, see http://heta.io/online-training-in-rhetorical-parsing
3. At present, this requires work on the underlying code. However, developing a tool that
interacts with this middleware layer to support end-users (such as academic staff) to create
such mappings is on the AcaWriter roadmap.
4. Higher Education Text Analytics project resources: http://heta.io/resources
5. AcaWriter information portal for university staff and students: https://uts.edu.au/ acawriter
6. See, particularly, https://github.com/uts-cic/ALASI2018-WritingAnalyticWorkshop

References
Aitchison, C., Catterall, J., Ross, P., & Burgin, S. (2012). ‘Tough love and tears’: Learning doctoral
writing in the sciences. Higher Education Research & Development, 31(4), 435–447.
https://doi.org/10.1080/07294360.2011.559195
Anthony, L., & Lashkia, G. V. (2003). Mover: A machine learning tool to assist in the reading and
writing of technical papers. IEEE Transactions on Professional Communication, 46(3), 185–
193. https://doi.org/10.1109/TPC.2003.816789
Boud, D. (2000). Sustainable assessment: Rethinking assessment for the learning society.
Studies in Continuing Education, 22(2), 151–167. https://doi.org/10.1080/713695728
Boud, D., Keogh, R., & Walker, D. (1985). What is reflection in learning. In D. Boud (Ed.),
Reflection: Turning experience into learning (pp. 7–17). Kogan Page.
Boyd, E. M., & Fales, A. W. (1983). Reflective Learning: Key to Learning from Experience. Journal
of Humanistic Psychology, 23(2), 99–117. https://doi.org/10.1177/0022167883232011
Brown, A. L. (1992). Design Experiments: Theoretical and Methodological Challenges in
Creating Complex Interventions in Classroom Settings. Journal of the Learning Sciences,
2(2), 141–178. https://doi.org/10.1207/S15327809JLS0202_2
Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-Centred
Learning Analytics: Design Frameworks, Tools and Techniques. Journal of Learning
Analytics, 6(2), 1–9. https://doi.org/10.18608/jla.2019.62.1
Buckingham Shum, S., Knight, S., McNamara, D., Allen, L., K., Betik, D., & Crossley, S. (2016).
Critical Perspectives on Writing Analytics. 6th ACM Learning Analytics and Knowledge
Conference, 481–483. https://doi.org/10.1145/2883851.2883854
Cai, (Luna) Jing. (2016). An exploratory study on an integrated genre-based approach for the
instruction of academic lexical phrases. Journal of English for Academic Purposes, 24, 58–
74. https://doi.org/10.1016/j.jeap.2016.09.002
Catterall, J., Ross, P., Aitchison, C., & Burgin, S. (2011). Pedagogical approaches that facilitate
writing in postgraduate research candidature in science and technology. Journal of
University Teaching & Learning Practice, 8(2). https://ro.uow.edu.au/jutlp/vol8/iss2/7/
Collins, A. (1992). Toward a Design Science of Education. In New Directions in Educational
Technology (pp. 15–22). Springer. https://doi.org/10.1007/978-3-642-77750-9_2
Cope, B., & Kalantzis, M. (1993). The Powers of Literacy: A Genre Approach to Teaching Writing.
Routledge.
Cotos, E., Huffman, S., & Link, S. (2020). Understanding graduate writers’ interaction with and
impact of the Research Writing Tutor during revision. Journal of Writing Research 12(1),
187-232. https://doi.org/10.17239/jowr-2020.12.01.07
Cotos, E. (2010). Automated writing evaluation for non-native speaker English academic
writing: The case of IADE and its formative feedback. Iowa State University.
Eisenhardt, K. M. (1989). Building theories from case study research. Academy of Management
Review, 14(4), 532–550. https://doi.org/10.5465/amr.1989.4308385
Elsevier. (2015). New open access resource will support text mining and natural language
processing. Elsevier Connect. https://www.elsevier.com/connect/new-open-access-
resource-will-support-text-mining-and-natural-language-processing
Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C., Knight, S., Nicolova,
N., & Jarvis, W. (2017). Reflective Writing Analytics for Actionable Feedback in Authentic
Assessment. 7th International Conference on Learning Analytics & Knowledge, 153–162.
https://doi.org/10.1145/3027385.3027436
Gibson, A., Kitto, K., & Bruza, P. (2016). Towards the discovery of learner metacognition from
reflective writing. Journal of Learning Analytics, 3(2), 22–36.
https://doi.org/10.18608/JLA.2016.32.3
Griffin Jr, J. E., Lorenz, G. F., & Mitchell, D. (2010). A study of outcomes-oriented student
reflection during internship: The integrated, coordinated, and reflection based model of

learning and experiential education. The Journal of Cooperative Education & Internships,
44(1), 42–50. https://scholarsarchive.jwu.edu/univ_office/2/
Hatton, N., & Smith, D. (1995). Reflection in teacher education: Towards definition and
implementation. Teaching and Teacher Education, 11(1), 33–49. https://doi.org/
10.1016/0742-051X(94)00012-U
Hayes, J. R., & Flower, L. (1980). Identifying the organization of writing process. In L. W. Gregg
& E. R. Steinberg (Eds.), Cognitive Processes in Writing (pp. 3–30). Erlbaum.
Hyland, K. (2005). Metadiscourse: Exploring writing in interaction. London: Continuum.
Hyland, K. (2007). Genre pedagogy: Language, literacy and L2 writing instruction. Journal of
Second Language Writing, 16(3), 148–164. https://doi.org/10.1016/j.jslw.2007.07.005
Kitto, K., Shum, S. B., & Gibson, A. (2018). Embracing Imperfection in Learning Analytics.
Proceedings of the 8th International Conference on Learning Analytics and Knowledge,
451–460. https://doi.org/10.1145/3170358.3170413
Knight, S., Allen, L. K., Gibson, A., McNamara, D., & Buckingham Shum, S. (2017). Writing
analytics Literacy—Bridging from Research to Practice. 496–497. https://doi.org/10.1145/
3027385.3029425
Knight, S., Buckingham Shum, S., Ryan, P., Sándor, Á., & Wang, X. (2018). Academic Writing
Analytics for Civil Law: Participatory Design Through Academic and Student Engagement.
International Journal of Artificial Intelligence in Education, 28(1), 1–28. https://doi.org/
10.1007/s40593-016-0121-0
Knight, S., Gibson, A., & Shibani, A. (in press). Implementing Learning Analytics for Learning
Impact: Taking Tools to Task. Internet and Higher Education. https://doi.org/10.1016/
j.iheduc.2020.100729
Knight, S., Shibani, A., & Buckingham Shum, S. (2018). Augmenting Formative Writing
Assessment with Learning Analytics: A Design Abstraction Approach. In J. Kay & R. Luckin
(Eds.), 13th International Conference of the Learning Sciences: Rethinking Learning in the
Digital Age. Making the Learning Sciences Count (Vol. 3, pp. 1783–1790). International
Society of the Learning Sciences. https://drive.google.com/drive/u/1/folders/1
wTHiZuSn8Wn84Q1WpAL-ihQ6SvaDWP17 https://doi.org/10.1145/3303772.3303785
Kuteeva, M. (2013). Graduate learners’ approaches to genre-analysis tasks: Variations across
and within four disciplines. English for Specific Purposes, 32(2), 84–96.
https://doi.org/10.1016/j.esp.2012.11.004
LAK11. (2011). 1st International Conference on Learning Analytics and Knowledge 2011 |
Connecting the technical, pedagogical, and social dimensions of learning analytics. LAK11.
https://tekri.athabascau.ca/analytics/about
Lea, M. R., & Street, B. V. (1998). Student writing in higher education: An academic literacies
approach. Studies in Higher Education, 23(2), 157–172. https://doi.org/10.1080/0307507981
2331380364
Lee, A., & Kamler, B. (2008). Bringing pedagogy to doctoral publishing. Teaching in Higher
Education, 13(5), 511–523. https://doi.org/10.1080/13562510802334723
Lee, A., Kamler, B., & Aitchison, C. (2010). Introduction: Why publishing pedagogies? In C.
Aitchison, B. Kamler, & A. Lee (Eds.), Publishing Pedagogies for the Doctorate and Beyond
(pp. 13–23). Routledge.
Lew, D. N. M., & Schmidt, H. G. (2011). Writing to learn: Can reflection journals be used to
promote self-reflection and learning? Higher Education Research & Development, 30(4),
519–532. https://doi.org/10.1080/07294360.2010.512627
Lucas, C., Gibson, A., & Buckingham Shum, S. (2019). Pharmacy Students’ Utilization of an
Online Tool for Immediate Formative Feedback on Reflective Writing Tasks. American
Journal of Pharmaceutical Education, 83(6). https://doi.org/10.5688/ajpe6800
Lucas, C., Smith, L., Lonie, J. M., Hough, M., Rogers, K., & Mantzourani, E. (2019). Can a reflective
rubric be applied consistently with raters globally? A study across three countries.
Currents in Pharmacy Teaching and Learning. https://doi.org/10.1016/j.cptl.2019.06.004

Maher, M. A., Feldon, D. F., Timmerman, B. E., & Chao, J. (2014). Faculty perceptions of common
challenges encountered by novice doctoral writers. Higher Education Research &
Development, 33(4), 699–711. https://doi.org/10.1080/07294360.2013.863850
Nesi, H., Gardner, S., Thompson, P., & Wickens, P. (2004). British Academic Written English
Corpus. Oxford Text Archive. http://purl.ox.ac.uk/ota/2539
OECD, & Statistics Canada. (2000). Literacy in the Information Age—Final Report of the
International Adult Literacy Survey. OECD. http://www.oecd.org/edu/skills-beyond-
school/41529765.pdf
Paltridge, B., & Starfield, S. (2007). Thesis and Dissertation Writing in a Second Language: A
Handbook for Supervisors. Routledge.
Roscoe, R. D., & McNamara, D. S. (2013). Writing pal: Feasibility of an intelligent writing strategy
tutor in the high school classroom. Journal of Educational Psychology, 105(4), 1010–1025.
https://doi.org/10.1037/a0032340
Rose, D., & Martin, J. R. (2012). Learning to write, reading to learn: Genre, knowledge and
pedagogy in the Sydney School. Equinox Sheffield.
Rothery, J., & Stenglin, M. (1994). Exploring literacy in school English (Write it right resources
for literacy and learning). Sydney: Metropolitan East Disadvantaged Schools Program.
Ryan, M. (1988). The teachable moment: The washington center internship program. New
Directions for Teaching and Learning, 1988(35), 39–47. https://doi.org/10.1002/
tl.37219883505
Ryan, M. (2011). Improving reflective writing in higher education: A social semiotic
perspective. Teaching in Higher Education, 16(1), 99–111.
https://doi.org/10.1080/13562517.2010. 507311
Ryan, M., & Ryan, M. (2013). Theorising a model for teaching and assessing reflective learning
in higher education. Higher Education Research & Development, 32(2), 244–257.
https://doi.org/10.1080/07294360.2012.661704
Sándor, Á. (2005). A framework for detecting contextual concepts in texts. ELECTRA Workshop
on Methodologies and Evaluation of Lexical Cohesion Techniques in Real-World
Applications (Beyond Bag of Words), 80–89. https://acl-arc.comp.nus.edu.sg//~
forecite/services/omnipage/thang/key_pdf/H-97.pdf#page=80
Shermis, M. D., & Burstein, J. (2013). Handbook of Automated Essay Evaluation: Current
Applications and New Directions. Routledge.
Shibani, A. (2018). AWA-Tutor: A Platform to Ground Automated Writing Feedback in Robust
Learning Design. 8th International Learning Analytics and Knowledge Conference,
Sydney, Australia. https://solaresearch.org/core/companion-proceedings-of-the-8th-
international-learning-analytics-knowledge-conference-lak18/
Shibani, A., Abel, S., Gibson, A., & Knight, S. (2018). Turning the TAP on Writing Analytics.
Companion Proceedings 8th International Conference on Learning Analytics &
Knowledge (LAK18). 8th International Conference on Learning Analytics & Knowledge,
Sydney, Australia. https://doi.org/10.1145/3170358.3170384
Shibani, A., Knight, S., & Buckingham Shum, S. (in press). Educator Perspectives on Learning
Analytics in Classroom Practice. Internet and Higher Education. https://doi.org/10.1016/
j.iheduc.2020.100730
Shibani, A., Knight, S., Buckingham Shum, S., & Ryan, P. (2017). Design and Implementation of
a Pedagogic Intervention Using Writing Analytics. In W. Chen, J.-C. Yang, A. F. Mohd Ayub,
S. L. Wong, & A. Mitrovic (Eds.), 25th International Conference on Computers in Education
(pp. 306–315). Asia-Pacific Society for Computers in Education. http://icce2017.
canterbury.ac.nz/sites/
default/files/proceedings/main/C3/Design%20and%20Implementation
%20of%20a%20Pedagogic%20Intervention%20Using%20Writing%20Analytics.pdf
Shibani, A., Knight, S., & Shum, S. B. (2019). Contextualizable Learning Analytics Design: A
Generic Model and Writing Analytics Evaluations. Proceedings of the 9th International

Conference on Learning Analytics & Knowledge, 210–219. https://doi.org/10.1145/3303772.
3303785
Shibani, A., Liu, M., Rapp, C., & Knight, S. (2019). Advances in Writing Analytics: Mapping the
State of the Field. Companion Proceedings of the 9th International ACM Conference on
Learning Analytics & Knowledge (LAK19), 251–255. https://doi.org/10.21256/zhaw-5531
Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education.
EDUCAUSE Review, 46(5), 30.
Strobl, C., Ailhaud, E., Benetos, K., Devitt, A., Kruse, O., Proske, A., & Rapp, C. (2019). Digital
support for academic writing: A review of technologies and pedagogies. Computers &
Education, 131, 33–48. https://doi.org/10.1016/J.COMPEDU.2018.12.005
Sumsion, J., & Fleet, A. (1996). Reflection: Can we assess it? Should we assess it? Assessment &
Evaluation in Higher Education, 21(2), 121–130. https://doi.org/10.1080/0260293960210202
Swales, J. (1990). Genre Analysis: English in Academic and Research Settings. Cambridge
University Press.
Swales, J. (2004). Research genres: Explorations and applications. Ernst Klett Sprachen.
Swales, J., & Feak, C. (2004). Academic writing for graduate students: Essential tasks and skills
(Vol. 1). University of Michigan Press.
Thomas, D. R. (2006). A General Inductive Approach for Analyzing Qualitative Evaluation Data.
American Journal of Evaluation, 27(2), 237–246. https://doi.org/10.1177/1098214005283748
Thompson, P. (2016). Genre approaches to theses and dissertations. In K. Hyland & P. Shaw
(Eds.), The Routledge Handbook of English for Academic Purposes (pp. 378–391).
Routledge.
Torrance, M., Thomas, G. V., & Robinson, E. J. (1992). The writing experiences of social science
research students. Studies in Higher Education, 17(2), 155–167. https://doi.org/10.1080/
03075079212331382637
Wang, F., & Hannafin, M. (2005). Design-based research and technology-enhanced learning
environments. Educational Technology Research and Development, 53(4), 5–23.
https://doi.org/10.1007/BF02504682
Warschauer, M., & Grimes, D. (2008). Automated Writing Assessment in the Classroom.
Pedagogies: An International Journal, 3(1), 22–36. https://doi.org/10.1080/
15544800701771580

AcaWriter A Learning Analytics Tool For Formative Feedback On Academic Writing

More Related Content

Similar to AcaWriter A Learning Analytics Tool For Formative Feedback On Academic Writing (20)

More from Scott Faria (20)

Recently uploaded (20)

AcaWriter A Learning Analytics Tool For Formative Feedback On Academic Writing