Using Educational Effectiveness research to Design Theory-Driven Evaluation Aiming to Improve Quality of Education / B.P.M. Creemers & L. Kyriakides

USING EDUCATIONAL EFFECTIVENESS
RESEARCH TO DESIGN THEORY-DRIVEN
EVALUATION AIMING TO IMPROVE QUALITY
OF EDUCATION

B.P.M. CREEMERS 1 & L. KYRIAKIDES 2

Faculty of Behavioral and Social Sciences, University of
Groningen, the Netherlands 1
Department of Education, University of
Cyprus, Cyprus2

INTRODUCTION

Educational Effectiveness Research (EER) aims to identify
factors operating at different levels which can explain
variations in student achievement by taking into account the
influence of background characteristics.
EER attempts to establish and test theories which explain why
and how some schools and teachers are more effective than
others in promoting better outcomes for students.
The origins of EER stem from reactions to work on equality of
opportunity in education that was conducted in the USA and
undertaken by Coleman et al (1966), and Jencks et al (1972).
The first two effectiveness studies (Edmonds, 1979; Rutter et
al., 1979) were concerned with examining evidence and making
an argument about the potential power of schooling to make a
difference in the life chances of students.
These two studies were followed by numerous studies in
different countries on educational effectiveness.

OUTLINE

1) Phases of EER
2) The dynamic model of educational
effectiveness
3) Theory-driven evaluation studies
4) Evaluation studies: lessons drawn from
the dynamic model
5) A framework for conducting theory-
driven evaluations: the contribution of the
dynamic model

Phases of EER

First Phase: A focus on Size of School Effects (Establishing
the field by showing that ‘school matters’ )
Studies conducted during the early 1980s showed how
important it is for students to have effective teachers and
schools.
School and teacher effects tend to be larger for disadvantaged
groups.
Second Phase: A focus on the characteristics/correlates of
effectiveness (searching for factors associated with better
student outcomes)
In the late 1980s and early 1990s, EER was mainly concerned
with identifying factors associated with student outcomes.
These studies resulted in a list of factors which were treated
as characteristics of effective teachers and schools.

Phases of EER

Third Phase: Modelling Educational Effectiveness.
By the late 1990s several integrated models of
educational effectiveness were developed. These
models sought to explain why factors which operate
at different levels are associated with student
outcomes.
These models guided not only the theoretical
development of EER but also the design of empirical
studies within the field.

Phases of EER

Fourth Phase: Focus on Complexity (analysis of the complex nature
of educational effectiveness developed further links with the study
of school improvement).
Addressing issues such as consistency, stability, and differential
effectiveness.
The move away from seeing effectiveness as an essentially
stable characteristic to one that varies across years and may
differ for different student outcomes or in relation to
different groups of students places change at the heart of EER.
Researchers gave attention to the study of complexity in education
and pointed to the fact that the theoretical models of the third
phase had not emphasised the dynamic perspective of education.
The field became increasingly linked with the growth of large scale,
systematic evaluations of the long term effect of teachers and
schools and of local and national reform policies.

THE DYNAMIC MODEL OF EDUCATIONAL
EFFECTIVENESS: AN OVERVIEW
A) The Rationale of the Model
Takes into account the new goals of education and
related to this their implications for teaching and
learning.
The dynamic model is established in a way that helps
policy makers and practitioners to improve educational
practice by taking rational decisions concerning the
optimal fit of the factors within the model and the
present situation in the schools or educational
systems.
The model should not only be parsimonious but also be
able to describe the complex nature of educational
effectiveness.

B) The Essential Characteristics of the Dynamic Model
The model is multilevel in nature.
Teaching and learning situation is emphasised.
School-level factors influence the teaching-learning situation by
developing and evaluating the school policy on teaching and the
policy on creating a learning environment at the school.
The system level refers to the influence of the educational
system through a more formal way, especially through developing
and evaluating the educational policy at the national/regional level.
The teaching and learning situation is influenced by the wider
educational context.
The values of the society for learning and the importance
attached to education play an important role in shaping teacher
and student expectations and in the development of the
perceptions of stakeholders about effective teaching practice.

Factors at the school and system level have direct and indirect
effects on student achievement.
The impact of the school- and system- level factors is defined
and measured in a different way than the impact of classroom-
level factors.
Only changes in those factors for which schools face
significant problems are expected to be associated with the
improvement of school effectiveness.
The impact of school and system level factors depends on the
current situation of the objects under investigation.
The relation of some effectiveness factors with achievement may
not be linear.
Optimal points for the functioning of factors in relation to
student outcomes have to be identified.


There is a need to examine the relationships
between factors which operate at the same level.
Such approach to modelling school effectiveness
reveals grouping of factors that make teachers
and schools effective.
Different dimensions for measuring the functioning
of effectiveness factors are used.
Each factor is defined and measured using five
dimensions: frequency, focus, stage, quality, and
differentiation.

C) Classroom Factors of the Dynamic Model
Classroom factors refer to observable instructional
behaviour of teachers in the classroom rather than on
factors that may explain such behaviour.
The eight factors included in the model are as
follows: orientation, structuring, questioning,
teaching-modelling, applications, management of time,
teacher role in making classroom a learning
environment, and assessment.
These eight factors do not refer only to one approach
of teaching such as structured or direct teaching or
to approaches associated with constructivism.

Frequency is a quantitative way to measure the
functioning of each factor. The other four dimensions
examine qualitative characteristics of the functioning
of the factors.
Two aspects of the focus dimension are taken into
account. The first one refers to the specificity of the
activities associated with the functioning of the
factor whereas the second one to the number of
purposes for which an activity takes place.
The stage at which tasks associated with a factor
take place is also examined. Factors need to take
place over a long period of time to ensure that they
have a continuous direct or indirect effect on student
learning.


The quality refers to the properties of the factor
itself, as these are discussed in the literature.
Differentiation refers to the extent to which
activities associated with a factor are implemented
in the same way for all the subjects involved with it.
Adaptation to specific needs of each subject or
group of subjects will increase the successful
implementation of a factor and will maximize its
effect on learning outcomes.
Considering effectiveness factors as multidimensional
constructs may help us develop strategies for
improving educational practice.

D) School Factors of the Dynamic Model
School factors influence classroom-level factors.
The dynamic model gives emphasis to two main aspects of
the school policy which affect learning at both the level of
students and teachers:
a) school policy for teaching and
b) school policy for creating a learning environment at
school.
These two factors refer to the actions taken by the school
to help teachers and other stakeholders have a clear
understanding of what is expected from them to do.
Support offered to teachers and other stakeholders to
implement the school policy is also an aspect of these
factors.

The dynamic model is concerned with the processes and
the activities which take place in the school in order to
improve the teaching practice and its learning
environment.
The following four overarching factors at the school
level are included in the model:
school policy for teaching and actions taken for
improving teaching practice,
evaluation of school policy for teaching and of actions
taken to improve teaching,
policy for creating a SLE and actions taken for
improving the SLE, and
evaluation of the SLE

THEORY-DRIVEN EVALUATION STUDIES

Theory driven evaluation is a collection of different
methodological approaches that can be used by
evaluators in trying to understand the impact of a
reform policy.
Evaluation has the task of testing out the underlying
programme theories and identifying unintended
consequences, which may or may not be beneficial.
When one evaluates, he/she always returns to the core
theories about how a programme is supposed to work and
then interrogates it by asking whether the basic plan is
sound, plausible, durable, practical, and above all, valid.
Evaluation projects that are theory driven take into
account the needs and issues raised by the various
stakeholders associated with an innovation.

THEORY-DRIVEN EVALUATION STUDIES

The agenda is expanded in such a way as to allow
evaluators to not only provide answers to the questions
raised by stakeholders but also help them understand
the reasons why a reform is more or less effective.
Evaluators in education should make use of the growing
knowledge base of EER as it is concerned with the
correlates of effective practice and provides theories
about their relationships with each other and with
student outcomes.
EER can be seen as a theoretical foundation upon which
better evaluation studies in education can be designed.
Theories of educational effectiveness can help
evaluators identify factors most closely associated
with the effective implementation of a reform.

EVALUATION STUDIES: LESSONS DRAWN
FROM THE DYNAMIC MODEL
A) Criteria of Evaluation
Measuring the Impact of the Reform on Student
Achievement
Evaluators adopting the dynamic model should identify the
extent to which the organisation units (e.g., schools or
educational systems) which implement the reform managed to
improve their effectiveness status (by looking at the progress
that their students made during the implementation of the
reform).
Both direct and indirect effects of the reform on student
achievement should be measured.
Indirect effects: They should find out whether the reform
has any impact on the functioning of effectiveness factor(s).
Evaluators searching for the impact of a national reform in
using IT in teaching mathematics should find out whether
the reform had any effect on teacher behaviour in the
classroom.

B) Taking into account the Multilevel Nature of
Education
Use multilevel statistical modelling approaches to
identify the impact of reform on student
achievement.
Searching for indirect effects upon factors
operating at different levels
C) Take into account the skills of those associated
with the reform to implement it
D) Use data of formative evaluation to take
decisions on how to improve the impact of the
reform on effectiveness factors associated with
the nature of the reform.

E) Going Beyond Stakeholders’ Reactions towards
the Reform
When evaluating a reform, researches face
difficulties in reducing their scope of examination
since reforms constitute complex phenomena, and
numerous aspects of them should be examined.
There will be variation in the ability of teachers and
schools to implement any reform.
Evaluators should search for characteristics of
teachers and schools that make them more or less
effective in implementing the reform.

Teachers’ reactions towards the reform
Examining teachers’ attitudes, thoughts, and criticism
regarding a reform is judged imperative.
Teachers’ perspectives and responses to a reform should
not be considered independently of the context in which
they operate.
Some schools seem to implement reforms in
supportive ways while others resist more to the
introduction of the reform.
Knowing that colleagues in a school are implementing
the reform successfully creates a productive
atmosphere for teachers to experiment with the
reform.

Looking at changes in teacher behaviour in the classroom
Findings of evaluation studies looking at the perceptions of
stakeholders cannot help us explain the differences in the
effective implementation of the reform from classroom to
classroom and school to school.
A reform is seldom implemented as planned; teachers often
make adaptations to the proposed reforms either to fit
with their professional judgment and ideologies or to match
the realities of their experiences and meet their students’
needs.
What matters in an educational reform is not the
availability of supporting resources but the quality of
teachers themselves which determines the quality of
teaching and, consequently, the effectiveness of any reform
policy.

EER may provide a list of criteria that may help
researchers elaborate on and study teachers’
practices during the reform implementation.
EER could facilitate the investigation of teachers’
professional development, if there is any, during the
implementation of the reform.
The dynamic model may offer a blueprint for
conducting observations.
The dynamic model may help in providing a historical
perspective of teachers’ practice as regards the
reform rather than a motionless picture of teachers’
implementation of the reform at a specific time.

F) Using the Measurement Framework of the Dynamic Model
to Design Evaluation Studies and Build a Meta-evaluation
System
The framework of the dynamic model can be used in analysing
the characteristics of the reform.
Frequency: we may raise questions such as how many types
of software are recommended to be used or how many
lesson plans are offered to teachers in order to support
them in using IT in teaching mathematics.
Focus: Investigate whether the guidelines given and/or the
actions taken for improving teaching practice are too
specific (or too general) in terms of what each teacher is
expected to do. We can also examine the purpose(s) that
are expected to be achieved by the reform policy, and
especially whether the reform aims to achieve a single
(e.g., improving teaching) or more purposes.

The stage dimension is seen as a very important measurement
dimension since according to the dynamic model, the reform
policy should be flexible, as this is reflected in the fact that,
from time to time, changes in the reform policy take place.
Continuity of the reform policy is also examined. We measure
the extent to which changes in the reform policy emerge from
the results of a systematic evaluation of the reform.
Quality is measured by investigating the properties of the
guidelines on using IT in teaching mathematics, especially
whether these are clear, concrete, in line with the literature,
and provide support to teachers, students, and administrators
to implement the reform policy.
Differentiation is measured by investigating the extent to
which school policy is designed in such a way that more support
is given to teachers and students who have difficulties in
implementing the policy.


The measurement framework can be used in
investigating the implementation of a reform in
different organisations (e.g., schools or classrooms).
Frequency can be measured by looking at the extent
to which the reform policy is implemented in a
classroom or a school.
Focus: We can investigate the extent to which
teachers or schools implement a policy by following
exactly what they have been asked to do or whether
they are flexible in implementing the reform policy.


The stage dimension is measured by looking at the
period in which the reform policy is implemented.

Differentiation is measured by investigating the
extent to which the reform policy is implemented
in such a way that more emphasis on the
implementation of the reform is given to teachers
and students who need the reform more.

A FRAMEWORK FOR CONDUCTING THEORY-DRIVEN
EVALUATIONS: THE CONTRIBUTION OF THE
DYNAMIC MODEL
1) Reformulate the research questions of stakeholders
 Characteristics of the dynamic model
 The theory upon which the reform is based.
2) Design the evaluation plan
 Summative evaluation (short and the long term effect on
student outcomes)
 Formative evaluation (implementation of the reform)
 Use the model to examine variation in the
implementation
 Evaluate changes in behaviour of teachers and
other stakeholders
 Search for learning opportunities offered to
students and other stakeholders

A FRAMEWORK FOR CONDUCTING THEORY-DRIVEN
EVALUATIONS: THE CONTRIBUTION OF THE
DYNAMIC MODEL
3) Collecting and analysing data
Use multilevel approaches
Develop instruments based on the framework of
the dynamic model
4) Reporting to policy makers and other stakeholders
5) Draw implications for the development of the
evaluation framework
The dynamic model could contribute to the
improvement of quality in education by establishing
an evidence-based and theory-driven approach.

Thanks for being
attentive …

Using Educational Effectiveness research to Design Theory-Driven Evaluation Aiming to Improve Quality of Education / B.P.M. Creemers & L. Kyriakides

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Using Educational Effectiveness research to Design Theory-Driven Evaluation Aiming to Improve Quality of Education / B.P.M. Creemers & L. Kyriakides (20)

Recently uploaded (20)

Using Educational Effectiveness research to Design Theory-Driven Evaluation Aiming to Improve Quality of Education / B.P.M. Creemers & L. Kyriakides