Annotations With A Tablet PC Or Typed Feedback Does It Make A Difference

Annotations with a Tablet PC or typed feedback: does it
make a difference?
Lyn Brodie
University of Southern Queensland, Toowoomba, Australia
Lyn.Brodie@usq.edu.au
Birgit Loch
University of Southern Queensland, Toowoomba, Australia
Birgit.Loch@usq.edu.au
Abstract: The University of Southern Queensland (USQ) has adopted a flexibility agenda
and thus, assignments in many disciplines are now submitted, marked and returned
electronically. However, since some markers are more comfortable marking hardcopy
assignments, electronically submitted assignments may still be printed, annotated, and
the handwritten feedback scanned or typed to facilitate electronic return of assignments.
The latter may limit the quantity, depth and placement of comments provided.
This research focuses on a course where model answers or a standard set of comments
cannot be generated and detailed individual marker feedback is required. Marking
rubrics have been designed to standardize marking but in the implementation process, the
level of individual feedback has deteriorated. This pilot study investigates how the
feedback students have received previously when assignments were marked with typed
comments, compares to this year’s experience when the same marker was given a Tablet
PC and was able to provide handwritten feedback.
Introduction
The work presented in this paper has developed from a USQ Learning and Teaching Fellowship which
provided Toshiba Portege M750 Tablet PC’s to all faculties to evaluate benefits to lecturers and effect
on student learning. It has been undertaken in collaboration between the Faculty of Engineering and
Surveying (FoES) and the Faculty of Sciences (FoS).The project described in this paper was to
investigate and extend the use of Tablet PC’s for providing electronic feedback for assessment items.
USQ has a large cohort of distance students and is well known for its development of distance
education materials. However the use of electronic submission and electronic marking of assessment
items, particularly within FoES has been slow to develop.
Anecdotal evidence suggests reasons for this are:
• The particular content of assessment items: largely mathematical or containing technical drawings,
and not essay based. Hence it is sometimes difficult and time consuming for students to ‘type’
assessment items.
• Equity: to ensure students who have limited access to both computing and internet facilities are
not disadvantaged and may still pursue tertiary education.
• Staff reluctance to move from traditional paper based submissions: for ease and flexibility in
marking, staff prefer making annotations on paper and do not wish to commit to marking when
they must have access to a computer. They perceive electronic assessment as more time
consuming.
• Use of sessional staff for marking: electronic submission and assessment requires all sessional
markers to have access to computing facilities and the university network.
• A lack of understanding on how to establish ‘good practice’ in electronic marking and feedback.
The course at the focus of this investigation has incorporated electronic submission for several years.
Considerable development, testing and refining online marking rubrics has been done which are now
ISBN 1 876346 59 0  2009 AAEE 2009
765

shown to be reliable and repeatable (Brodie & Gibbings, 2009). Evidence suggests that a limiting
factor of the rubrics is that markers tend to provide less detailed feedback, relying on the generic
descriptors of each level of the rubric. Spaces for specific comments are provided on the electronic
rubric but the authors hypothesise that specific comments, positioned at the correct place in the student
submission, are required for enhanced student learning. We also believe that a Tablet PC provides a
tool to enable markers to quickly provide this feedback in the electronic environment.
This paper is one in a series of investigations into the effective use of Tablet PC’s and related
technologies in higher education to promote student engagement and effective learning and teaching
practices at USQ. It describes initial results from a pilot study of electronic assignment marking with a
Tablet PC in an Engineering course.
Background
Written feedback on assignments is often the only source of feedback for distance education students.
Higgins et al. (2002) reported on the findings of a three year project investigating the meaning and
impact of assessment feedback for students in higher education. The report discussed quality, quantity
and language of comments and their relevance to the student’s learning, timeliness of feedback and the
importance of personal rather than generic feedback. Students said they thought they deserved
feedback as a matter of fairness as they invested their time in writing an assessment item, and they
wanted to have strengths and weaknesses of their work highlighted. Feedback, regardless of the
method, was important “to provide a grade, but also to focus on generic deep skills”. In addition,
feedback can be seen as dialogue (Sadler, 1989 as cited in Hatzipaganos and Warburton, 2009), which
is fundamental to the process of ‘closing the loop’ in assessment.
At USQ, electronic assignment submission is now encouraged in all Engineering and Surveying
courses. Reasons for this decision were of a financial nature, but also the educational benefits to
distance students were realised as the assignment turnaround time is reduced considerably if
assignments are handled entirely electronically. As Jones (1996) commented, “the speed with which
appropriate feedback is provided to students is essential to learning”, and timing of feedback is critical
if students are to both value it and benefit from it (Gibbs & Simpson, 2004-5). In a study by Redkedal
(1983), course completion rates were increased from about 70% to 90% when assignment turnaround
times were reduced from 8.3 days to 5.6 days. With mail delivery processes, turnaround times of three
weeks or more are not uncommon at USQ and students often do not receive assignment feedback until
after end of semester exams are completed. This means that students may not receive any feedback on
concepts introduced and assessed late in a semester (Buchanan & Farrands, 1995).
We argue that fast feedback on assessment items requires electronic marking as well as electronic
submission and thus elimination of any manual handling. A number of studies in the last few years
have reported on electronic marking of assessment items with a focus on handwritten annotation. For
instance, at the Open University (OU) in the UK, a comparison was made between feedback provided
by assignment markers in a large technology course where some markers typed comments, and others
were using the digital inking feature of a Tablet PC to write comments (Fisher, 2008, p. 6). Students’
and markers’ attitudes to the use of technology in the assessment process were evaluated, and
outcomes of this study include that “student engagement with electronic feedback could depend on
two things, the technology they use to generate their feedback, and their level of typing and ICT and
technical competencies”. The Tablet PC allowed greater creativity in assessment (p.1) and the author
suggests that pedagogy has been extended from pen and paper to electronic writing. Students liked the
personal feel of handwritten feedback, while markers liked to be able to write a comment immediately
while reading an assignment before the thought was lost, and they valued the ability to write feedback
close to the point being raised.
Another study from the OU reports on a trial of marking physics assignments with a Tablet PC
(Freake, 2008). Advantages and disadvantages of marking with a Tablet PC are listed. Issues included
the ability to annotate electronically similar to how this had been done on paper in the past, reduced
paper shuffling, improved turnaround time, ergonomic considerations and issues with the reliability of
the technology.
20th
Australasian Association for Engineering Education Conference University of Adelaide, 6-9 December 2009
ISBN 1 876346 59 0  2009 AAEE 2009
766

Methodology
This investigation is closely linked to practice and uses an Action Research methodology. Action
Research follows a series of cycles involving planning, acting, observing, reflecting and implementing
(Lewin, 1946). Action research is typically based in one institution and involves the researcher as an
active participant trying to solve a particular problem or issues that arise out of a particular teaching
practice.
The first phase of this research is modelled on the Action Research process introduced by Salmon
(2002, p. 201). Phase 1 is a pilot of the technology implementation within one course and focussing on
one particular application. Phase 2 will follow on from monitoring and reflecting on phase 1
outcomes and extending the pilot to additional markers and courses. Phase 2 is dependent on
renewing and replanning our research questions and will continue after phase 1 has been sufficiently
analysed. This paper focuses on disseminating the initial outcomes of phase 1.
The authors hypothesise that
1. Hand written comments provided by using the Tablet PC are more comprehensive and detailed
than those typed in the rubric previously.
2. As the hand written comments can be placed at specific points, these changes, corrections and
feedback are more readily used by the students in future submissions especially with respect to
formatting, spelling, grammar and referencing comments.
3. Markers prefer a more traditional approach to marking, i.e. hand written comments as opposed to
typing in a limited field in a marking rubric.
4. Students prefer personalised and specific comments allowed by handwritten annotations.
Phase 1 (Pilot phase) Details
Over 2 consecutive semesters (semester A and semester B) the research team observed one marker of
ENG1101 Engineering Problem Solving 1. In this course all student submissions, marking and
feedback are completed online and no paper submissions are accepted. The experienced sessional
marker used for this research was responsible for providing feedback and marking three team reports
for four teams of eight students in each semester, A and B i.e. 12 assessment pieces. The assignment
requirements were similar for each semester. In semester A the marker used comprehensive rubrics
developed as electronic forms (Brodie & Gibbings 2009). The form has text boxes which allow for
specific comments to be added for each criterion, and each level of achievement for a particular
criterion has detailed generic comments built in which provide additional feedback and guidance. In
semester B the marker was provided with a Tablet PC which allowed hand written annotations to be
added to the submitted assignments, similar to paper and pen marking of a traditional hard copy
submission. The final annotated files were returned to the students in addition to the rubric which
shows the actual marks for each criterion and level of achievement. Feedback by way of typed
comments in the comment field of the rubric (semester A) and hand written electronic comments on
the actual student submission from semester B, were reviewed and compared for type and quantity.
A secondary area of investigation involved collating both marker and student perceptions of the
feedback provided. For the marker this included views on time taken to mark, ease of providing
feedback and confidence in the quality of feedback provided. The students were surveyed to
determine their preference in assessment feedback and the use of this feedback in subsequent
submissions.
Student perceptions on the use of the feedback were validated by reviewing subsequent student
submissions and following specific errors, requests for clarification and further information to
determine if feedback was actually applied.
Results
Comparison of marker annotations and typed comments
All comments, hand written annotations as well as typed comments, were sorted into clusters based on
similarity for comparison. This process produced a coding schema of six main categories as shown
20th
ISBN 1 876346 59 0  2009 AAEE 2009
767

below. All codes where recorded as being either at a specific point or just as a general comment on
the page. These results were then compared with typed comments entered into the marking rubric from
previous semester submissions following the same coding.
Table 1 Coding schema for hand written annotations and typed comments
Category Examples
Generic Comment e.g. good, tick , cross
Specific encouragement
/feedback
Report structure including formatting, spelling, grammar, punctuation
Content both relating to technical aspects and teamwork aspects
including project management
Referencing Citation errors, correct method indicated, reference requirement
Clarification Not understanding e.g. ‘not sure what you are trying to say here’; ‘your
discussion [on this point] is not clear to the reader’
Request for further information or discussion e.g. direction for areas not
considered; points for further consideration
Linkages Between content in the report e.g. ‘you should make further reference to
the project objectives’
To other resources e.g. ‘you could research this area further see…’
Specific
instructions/feedback for
future submissions
‘You need to restructure the report in line with the submission
requirements; there are several critical technical errors in this design’
(followed by detailed reasons and explanations for the errors and how to
correct them)
Analysis of coding
The number and type of comments in each of the six major coding categories for marking rubric
(typed) and Tablet PC (hand written) were compared. For this small sample, there were no significant
differences in length or quality of feedback provided. Obviously there was some feedback which
would only be provided by the use of a Tablet PC eg. ticks, crosses and the placement of generic
comments such as ‘good’ at specific points in the student submissions, but the marker usually
provided some feedback by typed comments, e.g. “Introduction was good”; “second dot point in
Objectives is not sufficiently justified” etc. Similarly actual correction of spelling, grammar,
punctuation, formatting and referencing errors could not be done via the marking rubric and typed
comments only. However, this particular marker provided typed comments which were quite detailed
or gave specific examples in the typed feedback e.g. “check spelling carefully e.g. summary not
summery” but not all errors or modifications where specifically mentioned.
Coding of comments for the main categories of clarification, linkages and specific
instructions/feedback for future submissions showed no significant difference in quantity or quality
between the Tablet PC feedback and typed comments.
However anecdotal evidence suggests this may be indicative for this marker only. A review of
feedback given by two other markers randomly chosen showed that relatively little specific feedback
was provided for the same assessment piece. General comments referred students to overall
improvements e.g. “take greater care with proofreading”; “justify assumptions with more references”;
“check citation method” etc. Further research in phase 2 of the action research plan will investigate
the effect of specific comments on future student submissions.
Given detailed feedback was provided via annotations with the tablet PC, it was disappointing to
observe that even specific errors were not corrected in subsequent submissions of the assessment item.
In the first submission only three sections of a formal report were to be completed; Introduction,
Objectives (Design specifications) and Evaluation methods. For the final submission, the report was
to be completed in its entirety, resubmitting earlier components. However none of the teams took the
effort to make more than minor changes to the sections already assessed and most teams made no
changes at all to previously assessed sections even though the report was to be reviewed in total.
Marker Perceptions
The marker was keen to try the new technology and explore its options and possible advantages to
students. At the end of the semester, she was still supportive of the use of the Tablet PC, but indicated
20th
ISBN 1 876346 59 0  2009 AAEE 2009
768

a number of technical problems. These problems could be overcome with a small amount of training
or reconfiguration of the Tablet default settings. These included the default thickness and colour of
writing and the need to continuously reactivate the inking tool every time when moving between the
typed rubric in a separate document to provide a mark, and adding ink annotations on the next section
of the student assignment.
General comments and feedback from the marker regarding the Tablet were:
• The Tablet was easy to use.
• The eraser at the top of the stylus was useful for quickly editing comments. This was much easier
than editing typed comments in the rubric.
• The Tablet quickly converted hand written comments to text to be added to the rubric.
• Marking time was increased due to the addition of detailed comments, specifically correcting
spelling, formatting, grammar and punctuation, but the belief was that this was beneficial to the
students and hence worthwhile.
Student Perceptions
There was a poor response to the online student survey, most likely due to the late timeframe in the
semester when the survey was released. Only three students out of a possible 29 responded. A further
19 students were contacted by telephone and agreed to complete the survey verbally. Whilst this
meant that responses were not anonymous, results for the course had already been released and
students were aware that they were free to comment honestly.
All 22 students agreed that
• Electronic submission and feedback was better/easier than traditionally submitted assignments
(i.e. paper based, posted or handed into the administration office).
• The Tablet PC provided better feedback than typed comments.
• The feedback was useful and timely.
However 15 students admitted they did not read the feedback in detail, but only scanned it; five
admitted they only looked at the final mark and not what was missing or done poorly in the
submission as indicated by either the marking rubric or the feedback. The main reasons students gave
for not looking at the feedback in detail were
• Lack of time.
• Not my task within the team.
• Knowledge that those sections of the report already marked would not be reassessed and hence
there was no point in revisiting them.
• Technical problems.
The technical problems related to the fact that assignments were submitted and marked in Microsoft
Word format. In some cases when viewed on different computers and with different versions of
Microsoft Office, annotations had moved and were unreadable.
Discussion and Conclusion
The original hypotheses suggested that using a Tablet PC would provide more comprehensive and
detailed comments than those previously typed into the comment fields of the rubric. This pilot phase
of the investigation did not support this view. However anecdotal evidence still suggests that with a
larger sample of markers and assessment items, evidence may be found to support this view. This will
be investigated in future research.
Secondly it was hypothesised that the detailed feedback provided by hand written comments at the
relevant points in the submission would be more readily used by students in future submissions.
Again this was shown to be false. Whilst it could be argued that a larger sample of student work may
show a different result, this finding clearly has implications for the current assessment strategy and
may result in changes which give students more encouragement or incentives to actually use or
respond to the feedback.
20th
ISBN 1 876346 59 0  2009 AAEE 2009
769

Both markers and students showed a preference for handwritten comments. Both felt it was more
personalised and specific and hence of benefit. However the failure of students to act on feedback,
indicates that this personalised feedback may, in reality, be of little benefit if the overall assessment
scheme does not encourage students to learn from the comments given.
The pilot investigation undertaken and described in this paper provides data and information on which
to base future research into the effective and efficient use of technology and specifically Tablet PCs in
higher education and assessment. For instance, the categories in Table 1 provide a basis for further
literature reviews and investigations.
References
Brodie, L., & Gibbings, P. (2009). Comparison of PBL Assessment Rubrics. Paper presented at the Research in
Engineering Education Symposium. from http://eprints.usq.edu.au/5480/
Buchanan, R., & Farrands, P. (1995). Can Simulations Help Students Understand Programming Concepts - A
Case Study. Proceedings of Learning with Technology: The Twelfth Annual Conference of the Australian
Society for Computers in Learning in Tertiary Education, Melbourne.
Fisher, W. (2008). Digital ink technology for e-assessment. Paper presented at Education and Information
Systems, Technologies and Applications: EISTA 2008.
Freake, S. (2008). Electronic marking of physics assignments using a Tablet PC. New Directions, 4 (December
2008), 12-16.
Gibbs, G., & Simpson, C. (2004-5). Conditions under which assessment supports students’ learning. Learning
and Teaching in Higher Education(1), 3-31.
Hatzipanagos, S., & Warburton, S. (2009). Feedback as dialogue: exploring the links between formative
assessment and social software in distance learning. Learning, Media and Technology, 34(1), 45-59.
Higgins, R., Hartley, P. and Skelton, A. (2002). The Conscientious Consumer: reconsidering the role of
assessment feedback in student learning. Studies in Higher Education, 27(1).
Jones, D. (1996). Solving some problems of University Education: A Case Study [Electronic Version]. AusWeb.
Retrieved 4 August 2009, from http://cq-pan.cqu.edu.au/david-
jones/Publications/Papers_and_Books/96ausweb/index.txt.html.
Lewin, K. (1946). Action research and minority problems. Journal of Social Issues, 2(4), 34-46.
Redkedal, T. (1983). Enhancing student progress in Norway. Teaching at a Distance, 23, 19-24.
Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18,
119-144.
Salmon, G. (2002). Approaches to Researching Teaching and Learning Online. In Networked Learning:
Perspectives and Issues (pp. 195-212): Springer Verlag.
Copyright © 2009 Remains the property of the author(s). The author(s) assign to AaeE and educational non-profit institutions a
non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full
and this copyright statement is reproduced. The author(s) also grant a non-exclusive licence to AaeE to publish this document
in full on the World Wide Web (prime sites and mirrors) on electronic storage and in printed form within the AaeE 2009
conference proceedings. Any other usage is prohibited without the express permission of the author(s).
20th
ISBN 1 876346 59 0  2009 AAEE 2009
770

Annotations With A Tablet PC Or Typed Feedback Does It Make A Difference

More Related Content

Similar to Annotations With A Tablet PC Or Typed Feedback Does It Make A Difference (20)

More from Amy Cernava (20)

Recently uploaded (20)

Annotations With A Tablet PC Or Typed Feedback Does It Make A Difference