Improving Fairness on Students' Overall Marks via Dynamic Reselection of Assessors

International Journal on Integrating Technology in Education (IJITE) Vol.6, No.2, June 2017
DOI :10.5121/ijite.2017.6201 1
IMPROVING FAIRNESS ON STUDENTS’ OVERALL
MARKS VIA DYNAMIC RESELECTION OF ASSESSORS
Zhuhan Jiang1
and Jiansheng Huang1
1
School of Computing, Engineering and Mathematics, Western Sydney University, New
South Wales, Australia
ABSTRACT
A fundamental subject delivered at the tertiary level could have a cohort of several hundreds of students
distributed into multiple campuses. The running of such a unit typically calls for a teaching team of which a
major task is to fairly mark all students’ various assessment items. It is well observed that a given
assessment is likely to receive different marks if it is given to different markers, often regardless of how
detailed the marking criteria are, especially when the content is of subjective or opinion based nature. In
this work, we propose an effective strategy to improve the fairness on the students’ overall marks by
accepting that markers may have inherent marking leniency of different magnitude and by dynamically
reselecting markers for different groups of students in such a way that the students will eventually share a
similar amount of marking leniency in their overall marks. This strategy is completely objective, purely
based on the markers’ previous marking statistics, and is independent of the design and interpretation of
the marking criteria.
KEYWORDS
Fairer overall marks, marker reselection, objective strategies, effective algorithms.
1. INTRODUCTION
The burgeoning of e-learning or blended learning has brought about radical changes to the
education sector in recent years, ranging from the fast delivery of all forms of learning materials
and interactive activities, to the more effective learning models and teaching pedagogies [1-3]. An
important component in any formal learning is to conduct various assessments on the students
and give their marks and marking feedback accordingly [4]. Hence, it’s crucial for all students to
receive fair and equitable assessment marks on the work they have done [5].
When a student enrols into a course or subject to study theories or technologies, he will have a set
of expectations on what knowledge or skills are to be gained and what form of accreditation or
certification could be endowed for the learning achievement. Since the topic and skill coverage
within the subject in general complies quite well with the students’ expectations, how a subject
delivery is conducted impacts a great deal more on the student satisfaction on taking the subject
or unit, especially for subjects that are offered to massive number of students [6,7] online or
concurrently on multiple campuses. More specifically, the equity or fairness of assessments
across the students, as well as the individual’s expectation on the grade outcome, will be the
major factors affecting the students’ satisfaction rate or the students’ review on the teaching
performance [8,9]. The consistency and fairness on the student assessments are also crucial to
keeping students’ positive learning attitude and motivation. This is in general much less an issue
when the class size is small as the total number of involved disparate teaching staff would be
small too. For a large cohort of students, different campus lecturers and many different tutors or
markers will make the assessment consistency not an easy task at all [10,11,6]. Among many
popular assessment pedagogies, there is currently also a trend to adopt peer assessments [12,13]
or even automated feedback and marking system [14] somewhere. This may be partly because for
large classes peer assessments or the like implicitly reduce the assessment marking tasks by the
instructors and tutors. Of course, when peer assessments are heavily involved it may be even
harder to maintain the consistency and fairness across the students although it may in principle be

2
incorporated into our later proposed marker selection strategy as long as we treat these marking
peers just like the formal markers.
The marking inconsistency is typically due to the subjective assessment of certain items, in that
what marks should be given is not unequivocally set by the marking criteria. In theory the more
detailed the marking criteria are, the less marking inconsistency there will be. However it’s
simply not possible to specify all potential circumstances in the marking criteria, and therefore
subjective assessment is unavoidable. The subjective tendencies, or “biases”, are often inherent of
a person’s personality regardless of how hard a person may try to be fair. Hence it’s best to accept
that marking biases are unavoidable and then try to find ways to alleviate their impact rather than
pretending that such biases can be eradicated completely. To study any tertiary subject, a student
typically needs to submit several assessments due on different dates. We propose to explore the
marking statistics up to the last marked assessment to estimate how much marking leniency or
bias a student may have “suffered” this far, and then utilize the predicted marking profiles to
reselect markers for the next assessment so that the predicted accumulated biases will be pro-rata
evenly distributed among the students. An initial effort has already been made [15] in this regard
where markers are reallocated student-wise via algorithms utilizing means and standard
deviations for various assessments. This means the reallocation of marking duties will be student-
wise rather than group or class-wise, making its administration and in-class marking feedback
inconvenient or impossible. One of the main purposes of this work is thus to design a strategy to
reassign students group or class-wise to the markers, with a neater algorithm, so as to achieve
fairer overall marks for all students. We will also develop an approach to synchronize marking
profiles established from different cohorts of students in the past so that proper candidate markers
may be better selected for future marking consistency. These extra components of teaching
support may be eventually implemented as certain in-house tools similar to those in [16] that have
been gradually developed largely on a need-to basis to complement the generic support provided
by the web system currently adopted by the University.To recap the main purpose of this work,
we reiterate that the running of a large tertiary subject requires in general the support of many
assessors to mark or grade the student work, and these assessors will almost inevitably exhibit
inconsistent marking leniency no matter how detailed the marking criteria are formulated,
especially on the matters of subjective nature. Our proposed solution is therefore to reallocate the
assessors objectively for theseparatepieces of submitted student work so that each student will
share an estimated equal amount of accumulated marking leniency overall, thus leading to a fairer
and more balanced total assessment mark for each student.
This work is organized as follows. We first in section II explore the pitfalls of subjective biases
and typical various traditional strategies to reduce the marking discrepancies. We then in Section
III set up a proper framework to deal with group-wise marking allocations to assessors of
inhomogeneous marking profiles, and propose a simpler but more pertinent and efficient method
to reallocate the markers for the different assessments so as to even out the total accumulated
marking discrepancies as much as possible, with the corresponding implementation examples
demonstrated in Section IV. Section V then extends the use of marking profiles to cross-cohort
and multi-cohort profiles so as to improve the initial or later supplementary selection of the
markers for better marking consistency and fairness. The conclusions are finally summarized in
Section VI.
2. THE PITFALLS OF SUBJECTIVE BIASES
Fairness in assessing student work is essentially embedded into the marking or grading itself, and
it has been a topic under consideration for decades if not for centuries. A variety of teaching and
assessment methodologies, many in common sense and mostly independent of the use of
technologies, have already been in existence. Our consideration of fairness here will lay stress on
the equity across the students, rather than whether it is fair to assess students in a particular form

3
or on a particular topic. The assessment equity [17, 18] is often more subjective than the concept
of what and how to assess, where a vast number of contributory factors could range from the type
of assessments (written, oral, practical, peer, formative, group), to the topic complexity and
pertinence (fundamental, challenging), and further to the extent of marking feedback (evaluative,
descriptive). There are also many factors that can affect the assessment equity among the
students. Although formal marking criteria, possibly with the use of marking rubrics, can go a
long way to make marking or grading accountable, inherent individual “biases” are much harder
to defend against. Such biases are often demonstrated by an essentially the same work receiving
quite different marks when marked by different markers due to largely the markers’ individual or
subjective inclinations. This type of biases is generic and can contribute significantly to the
breach of the fairness to the students’ work and efforts.
With the technological advances in e-education, it has now become possible to reduce those
biases more objectively. As the first part of the undertaking, we assume that a teaching instructor
would pre-empt the potential partiality as much as possible by going through some of the
following strategies:
1. Formalize the assessment criteria as much as possible. This will obviously reduce the
mark fluctuation range for any given assessment to mark. In other words, if the same
assessment item is given to different markers, the difference of the marks this way is
likely to be smaller.
2. Subjective criteria should be replaced by the alternative objective criteria as much as
possible. For instance, instead of asking a marker to rate a programming style to see if it’s
excellent or good or something else, the marking criteria might specify objectively what
features are considered good, and therefore how good a program is can be more
objectively determined by counting the number of those desirable features. Some form of
auto-marking, such as that for the programming tasks [16], can be of good assistance.
It’s not difficult to imagine that some markers are just by nature more lenient than the others.
Although additional training on the markers can help to a good extent, it’s not always feasible or
practical. Hence if different assessment items for the same student get to be marked by suitably
different markers of a given marking team, then it’s possible for the students to get a similar
amount of marking leniency on the whole and therefore get fairer overall marks. The simplest yet
very effective strategy is to swap markers properly to make each student have a fair share of the
marking leniency. An algorithm can be formulated [15] to assign a suitable marker for each
individual student to mark the next assessment item so that the total accumulated leniency is
evened out as much as possible after such a re-assignment of the markers. This approach is
however individual-based, and the reallocated marking duties could go across a number of
different tutorial groups or even campuses, and therefore create considerable extra management
work on both the unit coordinator and the markers. Moreover, students may share some common
problems associated with a specific group, and a group-based reallocation would allow more
conveniently the relevant tutor for the group to address such problems. For these reasons we will
develop below strategies that will make full use of the fact that the markers will always be
assigned to mark assessments group or class-wise, where the group sizes are largely fixed due to
the fixed class or lab room size. It is perhaps important to note here that swapping markers is
much better a choice than, say, rescaling different groups of students differently according to who
their markers are. This is because it is often impossible to formally justify the use of different
rescaling for the different groups to the students, while in the case of reallocating markers all
existing marks can still be deemed “absolutely” correct on the papers.
3. IMPROVING FAIRNESS ON STUDENTS’ OVERALL MARKS
We now proceed to design a new algorithm for the markers’ group-wise reallocation. Let there be
a total of M groups or classes of students, with the group size being approximately the same T,

4
and let there be a pool of N available markers denoted by 1, 2, …, N respectively. Some markers
may not mark anything at all while some others may only mark some assessments. For the i-th
group of students, their designated marker j for the k-th assessment will be denoted by j=ρ(i,k).
Although almost all of our later concepts or methodologies will be explained in just words, for the
alternative more concise descriptions we nonetheless first introduce some notations for a few
basic concepts or quantities.
Suppose K assessments have already been marked. For the k-th assessment, 1≤k≤K, we assume
that its weight in marks is wk, and that each marker j∈J≡{1,2,…,N} was allocated the student
groups Ik
(j)
⊆I≡{1,2,…,M} and has already marked the allocated work for precisely the students of
the set Sk
(j)
, a total of Nk
(j)
≡ |Sk
(j)
| students who did submit the k-th assessment and belonged to the
groups allocated to marker j. For any given marker j∈J, set Sk
(j)
could be an empty set ∅, or could
consist of the students of one or several of the M student groups specified by Ik
(j)
. Let {xk,s
(j)
} for
all s∈Sk
(j)
denote the marks for all the students marker j marked for the k-th assessment. We
define the following variants of mark averages µ(j,k), ν(i), µ*(j), andµ(k) respectively by
, =
,

∈

,

0; = , , ,

∈ ;
∗ = 10 #

, $ %

, ̅ =
,

∈

∈(
, 1

where the mark average µ(j,k), conceptually being the average mark by the j-th marker on the k-th
assessment, is defined in the above only when marker j did mark some student work, i.e. Nk
(j)
0.
Otherwise, that is, if Nk
(j)
=0, we simply set µ(j,k)=µ(k), which is defined as the average mark of
all the submitted student work on the k-th assessment. Among all the assessments markerj marked
so far, the score percentage by markerj is measured by µ*
(j) whose value will always be
normalised to range from 0 to 10. For notational convenience, we here adopt 10 rather than 1 for
µ*
(j) to represent 100%. A value 10 here would mean that corresponding marker would give full
marks to every student work he marks. Let µ(i,k) denote the average mark of the submitted work
by the students in the i-th group on the k-th assessment. In the case of all groups of students
having exactly the same number of students and all students submitted all their assessments for
marking, then all quantities in (1) can be derived from the mark averages µ(i,k) on each group
for each i∈I and k, see (2) later. In other words, there will be no need in this case to delve into the
granularity of individual student marks once all the group averages are calculated.
Fig. 1: Estimate and rank average marks for markers
With the preparations in the above, we are ready to reallocate the marking duties for the (K+1)-th
assessment. For each group i, we will take ν(i) as the measurement of the total marking “biases”

5
the students in group i so far experienced and will take µ*
(j) as the prediction of the mark
“percentage” for marker j for marking the next assessment. Our strategy is thus to allocate the
most generous markers to the groups that experienced the most biases. The advantage of this
strategy is that no marks are ever rescaled explicitly, but the total accumulated over several
assessments will be objectively made fairer in that students are getting similar share of the
toughest and the most lenient markers.
We now first sort student groups so that the groups of lower marks on average will appear earlier
in the listing. More precisely, we set up a one-to-one mapping φ:I→I such that whenever a, b∈I
and a≤b we have ν(φ(a))≤ν(φ(b)). Suppose for the next assessment item, each marker j is
contracted to mark Mj≥0 student groups. We represent the marking duties by the set D={ (j,n):
j∈J, 1≤n≤Mj, and n is an integer}, where each pairing (j,n) indicates an additional student group
marker j needs to be assigned to mark. We now sort the marking duties so that their
corresponding markers will have the more generous ones appear earlier in the listing. That is, we
set up another one-to-one mapping ψ:I→D such that whenever a, b∈I and a≤b we have
µ*
(a′)≥µ*
(b′) where (a′, x)=ψ(a) and (b′, y)=ψ(b). Then for the next assessment item, the i′-th
student group will be assigned to marker j′ such that φ(a)=i′ and (j′, n)=ψ(a) for some a∈I, j′∈J,
and integer n≥1. In plain words, we assign the most lenient marker to the student group which has
been marked by the toughest markers in the previous assessments, see Fig. 1 for the estimation of
the accumulated average marks for each marker and the subsequent ranking of these markers,
where column g represents groups, column m represents markers, and column a represents mark
averages. We note that the averages in Fig. 1 are always represented by a value between 0 and 10,
with 10 indicating 100% of all the achievable marks, and this representation will also be adopted
in the later tables or diagrams of this work. The complete reallocation procedure is summarised in
the Markers Reallocation Algorithm (A1) framed out in the above.

6
In a perfect situation where all student groups are of absolute equal size and all students submit
the assessments for marking, then the algorithm can be simplified to the Simplified Reallocation
Algorithm (A2). We note that the simplified algorithm A2 in the above can be applied even if the
group sizes are only approximately identical and a small portion of students may not submit some
of their assessments, as the algorithm will nonetheless still reallocate markers for a fairer overall
marks in general. The default µ*
in step 4 may also be set to µ*
=∑N
j=1µ*(j)Mj/ (∑N
j’=1 Mj’). This
way the markers who won’t participate in the next marking task will be deemed irrelevant in
predicting the average percentage in the next marking task.
4. IMPLEMENTATIONS
Fig. 2: Data on who marked which student
groups and the mark averages with K=1.
Fig.3: Data on markers’ averages with K=1. The marking
profiles µ*
(j′) are ordered in shaded columns.
Fig.4: Sort group mark averages in shaded columns
and define φ:I→I by i′=φ(i) there. We then have
ν(φ(a))≤ν(φ(b)) whenever a,b∈I with a≤b.
To best illustrate the reallocation process, we now apply it to a simple case. We assume there are
4 student groups of 20 students each, and there are 5 available markers. Hence I={1,2,3,4} and
J={1,2,3,4,5}. For the 1st
assessment item, marker 1 is assigned to student group 1 and group 2;
marker 2 is assigned to group 3 and marker 3 is assigned to group 4. Suppose that the average
marks for the 1st
assessment item for the 4 groups are 6, 7, 9, and 6 respectively, see Fig.2. For the
Group Marker
j
Marks
average
Overall
average
i ρ(i,K) ̅(j,K) ν(i)
1 1 6 6.5
2 1 7 6.5
3 2 9 9
4 3 6 6
j µ(j,K) µ*
(j) Mj (j,1), ..,
(j,Mj)
µ*
(j′) j′
1 6.5 6.5 1 (1,1) 9 2
2 9 9 2 (2,1), (2,2) 7 4
3 6 6 0 7 5
4 7 7 1 (4,1) 6.5 1
5 7 7 0 6 3
Groupi Mark
averageν(i)
Sortedν(i′) i′
1 6.5 6 4
2 6.5 6.5 1
3 9 6.5 2
4 6 9 3

7
2nd
assessment item, we have marker 1 available or contracted to mark just 1 group, marker 2
available for 2 groups, and marker 4 available for 1 group again, see the 5th
column of Fig.3. We
thus apply our above proposed algorithm manually to assign the markers to the suitable group/s
so that the marks for the first 2 assessment items will overall be better balanced on the marking
fairness in terms of the potential subjective biases.
First we calculate the average marks for each group from (1) or A2 with K=1, see the results
forµ(i,K) and ν(i) in Fig.2. Next we find the mark averages for each marker, and represent the
marking of 4 student groups as the 4 pairs in the duty set D, see Fig.3. The value 7 in the column
for µ(j,K) there is the average of the column for µ(i,K), see (2) and the line below. Then we
reorder the group indices i′ to make tabulated group averages ν(i′) increase downwards, see the
shaded columns in Fig.4. We then reorder the marking duty pairs (j′,n′)∈D so that the
corresponding markers’ averages µ*
(j′) decrease downwards, see the middle 3 columns
(shaded in light blue) in Fig.5.
Fig.5: Sort the marking duties on the left to the 3 middle
columns by markers’ profiles, match with sorted group
averages in dark shade on the right, then i′→j′ maps the
groups to the markers.
After appending the shaded columns on the right of Fig.4 to the right of Fig.5, we can complete
the construction of φ(i) and ψ(i) in Fig.6, and then read off the marker reallocation ρ(i,K+1)
directly from the table there. For reading convenience, we added sub-indices (a)-(f) to
Fig.6 to indicate the sequential steps to arrive at (f) from (a). Hence, for the marking of the 2nd
assessment item, marker 2 is assigned to group 1 and group 4, marker 1 to group 3 and marker 4
to group 2.
Fig.6: Mappings for φ, ψ, and ρ. The shaded columns are the
group to marker mapping ρ(⋅, K+1): i′→j′.
As an example of applying this algorithm to a subject we delivered to a cohort of about 500
undergraduate students, the reallocated student groups can be directly shown to the markers via
the web, see Fig.7, and can be accessed or downloaded by the markers in a single go.
We applied this reallocation strategy to an actual subject delivery in the 2nd
semester of 2016
containing, among others, 2 assignments and a final exam. Before our measurement of the
accumulated biases, we first remove the incomplete and irregular samples that often correspond to
underdisciplined or extremely poor-performing students.
(j,n)∈D µ*
(j) µ*
(j′) (j′,n′)∈D j′ ν(i′) i′
(1,1) 6.5 9 (2,1) 2 6 4
(2,1) 9 9 (2,2) 2 6.5 1
(2,2) 9 7 (4,1) 4 6.5 2
(4,1) 7 6.5 (1,1) 1 9 3
Rank Group Rank Duties Group Marker
i φ(i) i (j,n)=ψ(i) i′ j′
1 4 1 (2,1) 1(a) 2(f)
2(c) 1(b) 2(d) (2(e),2) 2 4
3 2 3 (4,1) 3 1
4 3 4 (1,1) 4 2

8
Fig. 7: Marking duties reallocated on web via the simplified A2 algorithm
Although we can treat all students as a single group, it is more illustrative to break students into
several groups according to their final exam marks being G1:40-49.9%, G2:50-69.9%, and
G3:70-100%. The largest of these groups has 121 students. We treated the final exam marks as
more trustworthy as they are also more consistently marked, other marks will be compared
against the exam marks. Since marks’ consistency is largely relative, we will measure the
consistency of the marks, {xs} and {ys} respectively, for two assessments by finding α and β such
that such that {αxs+β} is closest to {ys} in the sense of the least squares. The table in Fig.8 lists
the least squares errors respectively for the 3 groups G1 (20 students), G2 (121 students) and G3
(87 students). We see that the accumulated errors have been reduced when assignments 1 and 2
are added together by compensating each other’s marking leniencies.
Fig.8: Estimated errors when comparing with the final exam marks.
Exam 40-49.9% 50-74.9% 75-100%
Item 1 2.5727 4.9335 6.3558
Item 2 2.7338 4.9253 6.4984
Items 1+2 2.5340 4.9181 6.2808

9
While the algorithm in the above will try to objectively ensure that the marking of the student
assessments is as fair as possible, we make sure that this message is also well disseminated to the
students’ mind so that they will also feel so themselves. We note that our proposed objective bias-
balancing approach is the first of its kind as far as we know, although the first exploration of this
kind was initiated in our previous work [15]. Our current work however will now be able to
support the logistic fact that markers typically have to be assigned class-wise or tutorial group-
wise, and the improvement on the fairer overall marks for the students such as those illustrated in
Fig. 8 is on a similar scale as that exhibited in [15] when students have to be reassigned to
different markers on the individual basis, therefore not suitable to beconducted class-wise, and
also more involved statistical quantities such as variousstandard deviation of student marks will
have to be made effective use of.
5. EXTENSION TO CROSS-COHORT BIAS-PROFILING
The advantage of utilising our proposed marker reallocation scheme is that it’s effective, easy to
implement, and applicable to all cases where students are grouped into tutorial classes and their
assessment tasks are marked group-wise by the same marker. Nevertheless, there are still many
different application scenarios that can induce further improvements on the fairness of the student
marks. In what follows we will address in several subsections such specific application scenarios.
5.1. MARKING PROFILES AND ITERATIVE UPDATE
While our proposed marker reallocation scheme suffices for a single cohort of students, it’s
possible to retain certain essential form of marking statistics or profiles for potential future use. In
this subsection, we will identify a set of crucial statistical features that can be easily extracted,
archived and later made use of when needed. Suppose we have the marks of a total of K
assessments marked by some of the markers in J. Then we may choose to retain only the
following suite ℑ of data, also referred to as the cohort marking profile later,
ℑ = )∗: ∈ +, ν: ∈ , ,, %, -

., 5
where w=∑K
k=1wk, W(j)
=∑K
k=1wk⋅Nk
(j)
, and the rest of notations are already defined in (1) or
nearby. These data are more essential in the sense that an update on ℑ due to including the marks
for the next assessment, i.e. the (K+1)-th assessment, can be calculated on the basis of the marks
for the next assessment alone, without having to resort to any earlier data on the marks other than
those of ℑ in (5). To this end, we just need to observe that µ(j,K+1) and µ(K+1) in (1) can be
calculated from the marks and marking allocations for the (K+1)-th assessment alone, while the
accumulative µ*(j), ν(i), w and W(j)
can be shown to be updatable to µ′*(j), ν′(i), w′ and W′(j)
via
µ′∗
= 0µ∗ + 10µ, 2 + 134

5 61 + 34

7
, ν′ = ν + µρ, 2 + 1, 2 + 1,
34

= %44

-

,
%′ = % + %4, -8

= -

+ %44

. 6
This shows ℑ is a decent choice of the minimum data to keep for the marker reallocation
algorithms. Once the marking of all the assessments is completed, we can treat the data in ℑ as
the statistical profiling for the involved markers and archive them for the potential future use.
Since the quantity W(j)
in (5) indicates the amount of the work j marked, we can normalize it into
W*(j)
via W*(j)
= W(j)
/ ∑N
s′=1W(s)
, so that ∑N
j=1W*(j)
=1. Hence W*(j)
may be treated as a form of
relative fidelity on j’s marking profile, as the more is W*(j)
, the more trustworthy is the profile
µ*(j). If all students submit all of the prescribed assessments, then all the W*(j)
could also

10
completely determine all the W(j)
because ∑N
j′=1W(j)
= ∑k′wk∑j Nk
(j)
= wTs and thus W*(j)
= W(j)
/(wTs),
where Ts denotes the total number of students for the cohort. For the case of having M groups of
students each with T students and the total marks weight is w, we then have W(j)
= W*(j)
wTM. The
advantage of using the percentage oriented W*(j)
is that they add up to 100% and are more
representative to their relative significance.
5.2. MERGE PROFILES ON TWO SEPARATE COHORTS
Suppose we currently have marking profiles ℑ′ and ℑ″ obtained from two cohorts of students.
The cohort profile ℑ′ corresponds to the primary cohort where notations will be mostly primed,
and ℑ″ corresponds to the secondary cohort where the notations will carry the double primes. If
the set of common markers, J≡J′∩J″, contains 2 or more markers, then it’s possible to incorporate
the marking profiles for some additional markers from J″ into that of J′ for the primary cohort.
Obviously the larger the cardinality |J| is, the more meaningful this synchronization will be. We
note that for a random variable of normal distribution, two values µ (mean) and σ (standard
deviation) will suffice its determination, and one normal distribution can be easily rescaled to
match another normal distribution. In fact, for each marker j∈J if one denotes by µ(j)
and σ(j)
respectively the mean and the standard deviation of the marks { Xi
(j)
}i , i=1, …, L(j)
, then for any
µ#
and positive σ#
the new marks Y i
(j)
transformed by this rescaling
;

= #
+ 6

−

7 @#
@

⁄ , = 1, … , C

, 7
will be another normal distribution bearingµ#
and σ#
as the new mean and standard deviation. In
other words, (7) easily rescales one normal distribution into another that exhibits a more desired
µ#
and σ#
.
Now we use the common markers in J≡J′∩J″ as the pivot to rescale the profile ℑ″ towards ℑ′ as
close as possible for the synchronization. For notational simplicity, we enlist J by J={ s1, s2, …, sn
}, and use xj and yj to represent µ″*(sj) and µ′*(sj) respectively. We now establish a mapping
y=P(x), with P(x)=∑m
j=0aj⋅xj
and m≥1, such that the weighted error
EF, … , EG = H
6I
− JKI
L7
M
N

, H
M
= -′

-′′

, 8
is minimized. The non-negative weight λj is to ensure that its significance is proportional to the
size of the past training data, or student marks to be more precise. In order to synchronize the
profiles in ℑ″ with those in ℑ′, we need to rescale the profiles in ℑ″ to fit as closely as possible
within the perspective of ℑ′ characterized by the profiles of the pivoting common markers. This is
in fact a standard Least Squares problem, and the solution is E = QΛQST
QΛI, Λ =
diagλ, … , λN, where
E = Z
EF
⋮
EG
, I = Z
IF
⋮
IN
, Q = ]
1

⋮

G
1

M
⋮

M
G
…
⋮
⋮
…
1

N
⋮

N
G
^. 9
We choose m=2 although m can be higher as long as m≤n. However, a small m like 2 or 3 is more
stable in general. We are now ready to merge some marking profiles from one cohort with that of
another cohort. There are two slightly different ways merge them. The first case is to add the
marking profiles for some of the markers from the secondary cohort ℑ″ to the primary cohort. In
this case, we in general expect the primary cohort ℑ′ is still on-going and the data there is more

11
pertinent, and the merge is only required because new markers need to be introduced for the
marking of the next assessment, typically due to some existing markers unexpectedly terminated
their marking commitment. Hence we will retain the existing cohort profile specified by ℑ′ as
much as possible, and the merging is done only through adding extra markers in a selected subset
of J″J′. The cohort profile ℑ′ is then extended by including the following subset ℑe
where
+ = +8
∩ +88
,ℑb
= c8∗
= J 688∗
7 , -8

= d

, ∈ +b
⊆ +88
+′g, and
d

= -′-∗hh
i -∗h

Improving Fairness on Students' Overall Marks via Dynamic Reselection of Assessors

More Related Content

What's hot (16)

Similar to Improving Fairness on Students' Overall Marks via Dynamic Reselection of Assessors (20)

Recently uploaded (20)

Improving Fairness on Students' Overall Marks via Dynamic Reselection of Assessors