Developing a comprehensive empirically based research framework for classroom based assessment

Developing a comprehensive, empirically based
research framework for classroom-based assessment
Language Testing 2011
Kathryn Hill and Tim McNamara
University of Melbourne, Australia
Presented by:
Amirhamid Forough Ameri (ahfameri@gmail.com)and Saeed Shiri
January 2016

Abstract
This paper presents a comprehensive framework for researching classroom-based
assessment (CBA) processes, and is based on a detailed empirical study of two
Australian school classrooms where students aged 11 to 13 were studying Indonesian as
a foreign language. The framework can be considered innovative in several respects. It
goes beyond the scope of earlier models in addressing a number of gaps in previous
research, including consideration of the epistemological bases for observed assessment
practices and a specific learner and learning focus. Moreover, by adopting the broadest
possible definition of CBA, the framework allows for the inclusion of a diverse range of
data, including the more intuitive forms of teacher decision-making found in CBA
(Torrance & Pryor, 1998). Finally, in contrast to previous studies the research
motivating the development of the framework took place in a school-based foreign
language setting. We anticipate that the framework will be of interest to both researchers
and classroom practitioners.

Introduction
 Previous studies have focused on issues such as
 validity and reliability (e.g. Gipps, 1994; Heurta-Macias, 1995),
 criteria and standards (e.g. Leung, 2007; Leung & Teasdale, 1997b) and
 the influence of external assessment and reporting regimes on classroom
practices (e.g. Brindley, 1998, 2001; Clarke & Gipps, 2000; Davison,
2004).
 Relatively fewer studies, however, have focused on the actual processes
of classroom-based assessment (research on which this framework is
intended to guide).

Introduction
 The framework we propose is based on a detailed empirical study of two
Australian school classrooms where students aged 11 to 13 were
studying Indonesian as a foreign language.
 Although essentially a bottom-up study, an initial orientation to the
research reported on here was developed from the existing literature.
 In this case, two themes from the literature guided the investigation: the
dimensions and scope of CBA, as well as the way these two aspects
intersected with each other.

Introduction
Dimensions
McNamara (2001) sets out three critical dimensions of assessment:
evidence
use
interpretation

Introduction
 According to McNamara, CBA is ‘[a]ny deliberate, sustained and
explicit reflection by teachers (and by learners) on the qualities of a
learner’s work’ and the use of this information, for example, ‘as an aid to
the formulation of learning goals…’ (2001, p. 343).
 However, the literature reveals significant diversity in how each of these
dimensions is understood (Table 1).

Introduction
 The definition adopted for the study is designed to reflect this diversity
and, in line with ethnographic principles, to admit all possible evidence.
We thus propose the following amended definition of CBA:
 Any reflection by teachers (and/or learners) on the qualities of a learner’s
(or group of learners’) work and the use of that information by teachers
(and/or learners) for teaching, learning (feedback), reporting,
management or socialization purposes.
 Note this definition of CBA incorporates both formative (or assessment
for/as learning) and summative assessment (assessment of learning).

Introduction
Scope
 In terms of scope, the following focal research questions, reflecting
issues identified by Leung (2005) and Rea-Dickins (2006) respectively,
were used to guide the empirical study:
 1. What do language teachers do when they carry out classroom-based
assessment?
 2. What do they look for when they are assessing learners?
 3. What theory or ‘standards’ do they use?
 4. Do learners share the same understandings?
o The relationship between scope and dimensions is set out in Table 2.

Design of the study
 As the focus is on understanding processes rather than describing
outcomes, existing CBA research is essentially qualitative in nature,
typically drawing on discourse-oriented social-constructivist
approaches to cognition and language use.
 Torrance and Pryor (1998) drew on theories of classroom interaction,
constructivist theories of learning and theories of motivation and
attribution to interpret their data (classroom observation and interviews).
 Our study used ethnographic methods, in particular, participant
observation and case studies, and a ‘grounded’ approach to analysis
(Rea-Dickins, 2001).

Design of the study
 Research context
 The study took place in two Indonesian language classrooms in Victoria,
Australia, and coincided with the introduction of a new curriculum and
standards framework, VELS (VCAA, 2008).
 Reasons for choosing Indonesian:
 Firstly, as this was an ethnographic study involving participant
observation, it was important to choose a language that the primary
researcher (Hill) speaks and understands.
 A second consideration was the availability of suitable programs.

Design of the study
 Participants and programs
 One high school (Years 7 to 12) and one primary school (Preparatory to Year 6), both
government schools, were recruited for the study.
 The selected programs were both well established and could be considered ‘typical’
in terms of contact hours, resources, curriculum and student demographic. One
Indonesian class was recruited from each year level. The Year 6 class received one
hour of Indonesian instruction per week as compared to five 55-minute classes per
fortnight (or 125 minutes a week) in Year 7.
 One Year 6 (primary) teacher and two Year 7 (high school) teachers volunteered for
the study. Each had a high level of competence in language teaching (Duff & Uchida,
1997) and a genuine interest in the research. The Year 6 teacher taught Indonesian P-
6 and had taught the participating students for four consecutive years.

Design of the study
 Data collection and data analysis
 Data collection took place in the final 10 weeks of Year 6 (end of
primary school) and the first 10 weeks of Year 72 (beginning of
secondary school) (Figure 1).
 Data comprised approximately 80 hours of audio-recordings (including
classroom interactions and teacher and focus group interviews) as well as
participant observation, field notes and documents (reports, worksheets,
etc.) see Table 3.
 A summary of the data set from a single unit of instruction in Year 7 has
been provided as an example (Table 4).

Findings
 RQ1 What do teachers do?
 Four key assessment processes were identified in the data:
planning framing
conducting using

Findings
 Planning assessment
 The first category, ‘Planning’, arose from an analysis of internal syllabus
documents as well as from discussions about teaching and assessment
with, and between, the participating teachers.
 This category captures information about the type and nature of planned
assessment tasks and the relationship of assessment to instruction as well
as to the relevant external frameworks

Findings
 Framing assessment for students
 This category investigates how (or whether) learners in the respective
classrooms came to know that a given activity was ‘for assessment’,
which Rea-Dickins (2006) argues is important if learners are to adopt an
appropriate orientation to the task.
 This category became significant precisely because assessment activities
in Year 6 were embedded in teaching in a way that made the assessment
dimension essentially ‘covert’ from the learners’ perspective.

Findings
 Conducting assessment
 The processes captured by this category range from explicit, planned,
formal assessment activities to less visible, unplanned, instruction-
embedded assessment activities. Table 5 sets out the terms used to
distinguish between these types.

Findings
 Using assessment data
 This category classifies assessment as for ‘teaching’, ‘learning’, ‘reporting’, ‘management’ and
‘socialization’ purposes respectively. Again, while these are presented as distinct categories,
analytically it is not always possible to separate one purpose from another.
 Teaching. How the teachers’ informal observations of how the Year 7 class was progressing
(‘incidental’, ‘group-level’ assessment) was used to inform the pace of teaching.
 Learning (feedback). Assessment-related information was also used to promote immediate
learning, through the provision of feedback on performance.
 Reporting. the Year 7 teachers use assessment-related information to inform decisions about
students’ end-of-year reports.
 Management. The use of assessment for classroom management, such as controlling or reinforcing
behaviour, for encouragement or for creating a positive atmosphere.
 Socialization. The ‘socialization’ of learners into the local conventions of teaching and assessment.

Findings
 RQ2 What do teachers look for?
 Sources of information about the valued enterprises and qualities and
standards operating in the respective classrooms included syllabus
documents, written or verbal instructions, assessment rubrics, written or
verbal feedback, report-writing meetings and written reports.
 This category has been divided into:
 information provided ‘in advance’ of task performance,
information provided in written or spoken ‘feedback’ and
information provided through ‘reporting’ respectively.

Findings
 In advance
 Figure 5 provides a segment of the instructions to students for a Year 6
speaking and writing task. It provides advance information about the
obligatory components of the task (Parts A, B and C), the response
format (writing and speaking), performance conditions (‘with a partner’)
and weighting (15 from a total of 60 points).
 However, the specified criteria (‘organization’, ‘persistence’, ‘getting
along’ and ‘confidence’) refer to personal qualities rather than any
feature of written and spoken language.

Findings
 In feedback
 The importance of feedback in communicating criteria and standards as well as the
strategies for achieving them (e.g. Sadler, 1989; Torrance & Pryor, 1998; Tunstall &
Gipps, 1996).
 In the following example the Year 7 class had been given an aural discrimination
task, where they had to note how many times they heard a specific vocabulary item.
Here the teacher provides information about the acceptable standard (or range of
performance) for this task, that is, exactly seven times (the correct answer) or close to
that number (e.g. five times).
Ok, whether you heard it five times or seven times as long as you’re in the vicinity you
should be pretty happy with yourselves that you’ve heard it that many times. If you’ve
written down ‘once’or if you’ve written down ‘24’then there’s a bit of a problem. (T2,
Year 7)

Findings
 In reporting
 Another source of evidence about what teachers look for was provided
during the reporting process.
 Tunstall and Gipps (1996) suggest teachers have a ‘notion of excellence’
which they characterize as part of the teachers’ ‘guild knowledge’. This
‘guild knowledge’ informs what Wiliam (2001) has termed ‘construct-
referenced’ assessment, which ‘relies on the existence of a construct (of
what it means to be competent in a particular domain) being shared by a
community of practitioners’ (pp. 172–173).

Findings
 RQ3 What theory or standards do teachers use?
A number of researchers have postulated a close relationship between
 teachers’ representations of the subject or content area,
 their pedagogic principles, and
 their assessment practices (e.g. James, 2006; Leung, 2005, 2007; Thomas
& Oldfather, 1997; van Lier, 2004; Wiliam, 2001).
 Hence this category explores articulated views regarding the subject
(Indonesian), language and language learning, and assessment, which
may underlie classroom practice.

Findings
 Views of the subject or content area (Indonesian)
 Syllabus documents provided an important source of evidence regarding how the
discipline (i.e. Indonesian) was constructed in the respective classrooms. For
example, the Year 7 syllabus document (Figure 6) comprises a series of generic
(culturally neutral) topics.
 Beliefs about language and language learning
 This category investigates the theories of language and language learning that appear
to underpin teachers’ beliefs and practice.
 Beliefs about assessment
 The belief in the importance of intuition and knowledge of the students over time
rather than a reliance on formal assessment.

Findings
 RQ4 Learner understandings
 Rea-Dickins (2006) and others have highlighted the need for further
research on assessment from a learner perspective.
 Hence this category explored learner understandings regarding
the nature of the subject (Indonesian, and language and foreign language
learning more broadly), as well as
 their notions of assessment (criteria and standards).

Findings
 Understandings of language learning
 As with their teacher, the Year 7 students attribute their friend’s competence
in Indonesian to ‘innate’ variables (ethnicity and general intelligence),
rather than effort.
 Understandings of assessment
 Previous research has found that learners often draw on their own, possibly
incongruent, understandings of task, criteria and standards (Coughlan &
Duff, 1994; Moni, 1999; Torrance & Pryor, 1998).
 The Year 7 teacher anticipates and expressly discourages a known
propensity for students to focus on presentation at the expense of content in
their written work.

Discussion: A framework for CBA research
 The findings from the study can now be summarized as a framework for
CBA research:

Discussion: A framework for CBA research

Conclusion
 The aim of this empirical study was to understand rather than evaluate CBA practices
in the respective classrooms with the aim of expanding, rather than answering, the
questions that should be asked in CBA research.
 There is already a volume of research evidence regarding the effects of different
CBA practices on learning, not least of all that found in Black and Wiliam’s (1998)
influential meta-analysis. However, there is clearly a place for experimental studies
of how the different CBA processes outlined in this paper might impact on learning
outcomes.
 In conclusion, we anticipate that the framework will be useful for researchers
interested in understanding classroom-based assessment and for teachers wishing to
gain greater insight into the integration of assessment in their everyday teaching
practices and the impact of their assessment practices on learning.

Developing a comprehensive empirically based research framework for classroom based assessment

Developing a comprehensive empirically based research framework for classroom based assessment

More Related Content

What's hot (20)

Viewers also liked (18)

Similar to Developing a comprehensive empirically based research framework for classroom based assessment (20)

More from ahfameri (12)

Recently uploaded (20)

Developing a comprehensive empirically based research framework for classroom based assessment