Experimental threats

Experimental
Design and Threats
Del Siegle and Scott Brown.
Reference-
E-Content- Module 12-Research Methods and Statistics.
1
SXCE,Patna

Experimental Research Designs have
Two Purposes:
• …to provide
answers to
research
research
questions
• ...to control variance
(differences)
2
SXCE,Patna

The main function of the
experimental research
design is to control variance.
• Principle: maximize
systematic variance,
control extraneous
and minimize error
variance.
In other words control variance.
3
SXCE,Patna

Therefore the researcher attempts to:
• maximize the variance of the variable (s) of the research
hypothesis (i.e., maximize the difference in the dependent
variable [outcome] caused by maximizing the differences in the
independent variable [treatment]).
•
control the variance of extraneous or "unwanted "
•
control the variance of extraneous or "unwanted " variables
that may have an effect on the experimental outcomes , but
which he/she is not interested (limit factors other than the
treatment (IV) that could be causing differences in the
outcome (DV) .
• minimize the error or random variance (i.e., avoid unreliable
measurement instruments which have high errors of
measurement ).
4
SXCE,Patna

Maximization of Experimental
Variance
• experimental variance
– the variance due to the
manipulated (i.e., treatment)
or attribute (i.e., gender)
variables (IV)
variables (IV)
research precept:
– design, plan and conduct
research so that experimental
conditions are as different as
possible on the independent
variable.
5
SXCE,Patna

Control of Extraneous
Variables (EV)
• eliminate the variable (i.e., if you are worried
about gender, only include one gender in the study).
• randomization (i.e., if you randomly assign subjects to
groups, the extraneous variable should be equally
distributed among the groups)
• build it into the design – make it a moderator variable (i.e.,
• build it into the design – make it a moderator variable (i.e., if you
are worried about gender, build it into the analysis [2- way
ANOVA]—we’ll learn about this later)
• match subjects (i.e., match the characteristics of subjects and
put one of each matched pair in each group)
• statistically equate groups (i.e., use ANCOVA [Analysis of
Covariance] to analyze the data with the extraneous variable
used as a covariate—we’ll learn about this later)
6
SXCE,Patna

Minimizing Error Variance has
Two Principle Aspects:
• reduction of errors of
measurement through
controlled conditions (i.e.,
standardize testing
procedures)
• increase in the reliability of
measures (i.e., revise test
instruments or find more
reliable ones)
7
SXCE,Patna

Experimenta
l Designs
Should be
Developed to
Ensure
Ensure
Internal and
External
Validity of
the Study
8
SXCE,Patna

Internal Validity:
• Are the results of the
study (DV) caused by
the factors included in
the study (IV) or are
they caused by other
factors (EV) which
were not part of the
study?
There are 16 common
threats to internal validity.
9
SXCE,Patna

Subject
Characteristics
Threats to Internal Validity
(Selection Bias/Differential Selection ) -- The groups
may have been different from the start. If you were
testing instructional strategies to improve reading
and one group enjoyed reading more than the other
group , they may improve more in their reading
because they enjoy it, rather than the instructional
strategy you used.
Characteristics
10
SXCE,Patna

(Mortality) -- All of the high or low scoring
subject may have dropped out or were
missing from one of the groups. If we
collected posttest data on a day when the
Loss of Subjects
collected posttest data on a day when the
honor society was on field trip at the
treatment school, the mean for the
treatment group would probably be much
lower than it really should have been.
11
SXCE,Patna

Perhaps one group was at a
disadvantage because of their
location. The city may have
Location
been demolishing a building
next to one of the schools in
our study and there are
constant distractions which
interfere with our treatment.
12
SXCE,Patna

The testing instruments may not be
scores similarly. Perhaps the person
grading the posttest is fatigued and
pays less attention to the last set of
papers reviewed. It may be that those
papers are from one of our groups and
papers are from one of our groups and
will received different scores than the
earlier group's papers
Instrumentation
Instrument Decay
13
SXCE,Patna

The subjects of one group may react differently to the
data collector than the other group. A male
interviewing males and females about their attitudes
toward a type of math instruction may not receive the
same responses from females as a female
interviewing females would.
Data Collector
Characteristics
14
SXCE,Patna

The person collecting data my favors one group,
or some characteristic some subject possess,
over another. A principal who favors strict
classroom management may rate students'
attention under different teaching conditions with
a bias toward one of the teaching conditions.
Data
Collector
Bias
15
SXCE,Patna

The act of taking a pretest or posttest may influence the
results of the experiment. Suppose we were conducting
a unit to increase student sensitivity to prejudice. As a
pretest we have the control and treatment groups watch
Testing
pretest we have the control and treatment groups watch
Shindler's List and write a reaction essay. The pretest
may have actually increased both groups' sensitivity
and we find that our treatment groups didn't score any
higher on a posttest given later than the control group
did. If we hadn't given the pretest, we might have seen
differences in the groups at the end of the study.
16
SXCE,Patna

Something may happen at one site during our study that
influences the results. Perhaps a classmate dies in a car
accident at the control site for a study teaching children bike
safety. The control group may actually demonstrate more
concern about bike safety than the treatment group.
History
17
SXCE,Patna

There may be natural
changes in the subjects
that can account for the
changes found in a study.
A critical thinking unit may
A critical thinking unit may
appear more effective if it
taught during a time when
children are developing
abstract reasoning.
Maturation
18
SXCE,Patna

The subjects may respond differently just because they are
being studied. The name comes from a classic study in
Hawthorne Effect
which researchers were studying the effect of lighting on
worker productivity. As the intensity of the factory lights
increased, so did the worker productivity. One researcher
suggested that they reverse the treatment and lower the
lights. The productivity of the workers continued to increase.
It appears that being observed by the researchers was
increasing productivity, not the intensity of the lights.
19
SXCE,Patna

One group may view that it is in competition with the other
group and may work harder than they would under normal
circumstances. This generally is applied to the control group
"taking on" the treatment group. The terms refers to the
classic story of John Henry laying railroad track.
John
Henry
Effect
20
SXCE,Patna

The control group may become discouraged
because it is not receiving the special attention
that is given to the treatment group. They may
perform lower than usual because of this.
Resentful
Demoralization of
the Control Group
21
SXCE,Patna

(Statistical Regression) -- A class that scores
particularly low can be expected to score slightly
higher just by chance. Likewise, a class that
scores particularly high, will have a tendency to
Regression
scores particularly high, will have a tendency to
score slightly lower by chance. The change in
these scores may have nothing to do with the
treatment.
22
SXCE,Patna

The treatment may not be implemented as
intended. A study where teachers are
asked to use student modeling techniques
may not show positive results, not because
modeling techniques don't work, but
modeling techniques don't work, but
because the teacher didn't implement them
or didn't implement them as they were
designed.
Implementation
23
SXCE,Patna

Someone may feel sorry for the control
group because they are not receiving much
attention and give them special treatment.
For example, a researcher could be studying
the effect of laptop computers on students'
attitudes toward math. The teacher feels
sorry for the class that doesn't have
Compensatory
Equalization of Treatment
sorry for the class that doesn't have
computers and sponsors a popcorn party
during math class. The control group begins
to develop a more positive attitude about
mathematics.
24
SXCE,Patna

Experimental Treatment
Diffusion
Sometimes the control group
actually implements the
treatment. If two different
treatment. If two different
techniques are being tested in
two different third grades in the
same building, the teachers
may share what they are
doing. Unconsciously, the
control may use of the
techniques she or he learned
from the treatment teacher.
25
SXCE,Patna

Once the researchers are confident that
the outcome (dependent variable) of the
experiment they are designing is the
result of their treatment
(independent variable)
[internal validity],
[internal validity],
they determine for which
people or situations
the results of
their study apply
[external validity].
26
SXCE,Patna

External
Validity:
• Are the results of the study generalizable to
other populations and settings?
External validity comes in two forms: population and ecological.
27
SXCE,Patna

1. ...the extent to which one can generalize from the study
sample to a defined population--
If the sample is drawn from an accessible population, rather
Threats to External Validity
(Population)
Population Validity is the extent to which the results of a study
can be generalized from the specific sample that was studied to a
larger group of subjects. It involves...
If the sample is drawn from an accessible population, rather
than the target population, generalizing the research results
from the accessible population to the target population is
risky.
2. ...the extent to which personological variables interact with
treatment effects--
If the study is an experiment, it may be possible that different
results might be found with students at different grades (a
personological variable).
28
SXCE,Patna

Ecological Validity is the
extent to which the results of an experiment can be
generalized from the set of environmental conditions
created by the researcher to other environmental
conditions (settings and conditions).
(Ecological)
conditions (settings and conditions).
There are 10 common
threats to external
validity.
29
SXCE,Patna

(Ecological) Explicit
description of
the
experimental
treatment
(not sufficiently described for others to
replicate) If the researcher fails to
adequately describe how he or she
conducted a study, it is difficult to
determine whether the results are
applicable to other settings.
treatment
30
SXCE,Patna

(catalyst effect)
(Ecological)
Multiple-treatment
interference
(catalyst effect)
If a researcher were to apply
several treatments, it is difficult to
determine how well each of the
treatments would work individually.
It might be that only the combination
of the treatments is effective.
31
SXCE,Patna

(attention causes differences)
Subjects perform differently because they
know they are being studied. "...External
(Ecological)
Hawthorne effect
know they are being studied. "...External
validity of the experiment is jeopardized
because the findings might not generalize
to a situation in which researchers or
others who were involved in the research
are not present" (Gall, Borg, & Gall, 1996,
p. 475)
32
SXCE,Patna

(Ecological)
Novelty and
disruption effect
(anything different makes a difference)
A treatment may work because it is novel and the subjects
respond to the uniqueness, rather than the actual
treatment. The opposite may also occur, the treatment
may not work because it is unique, but given time for the
subjects to adjust to it, it might have worked.
33
SXCE,Patna

(it only works with this experimenter)
(Ecological)
Experimenter effect
(it only works with this experimenter)
The treatment might have worked
because of the person implementing it.
Given a different person, the treatment
might not work at all.
34
SXCE,Patna

(pretest sets the stage)
A treatment might only work if a
pretest is given. Because they
(Ecological)
Pretest sensitization
pretest is given. Because they
have taken a pretest, the subjects
may be more sensitive to the
treatment. Had they not taken a
pretest, the treatment would not
have worked.
35
SXCE,Patna

(posttest helps treatment "fall into place")
The posttest can become a learning
(Ecological)
Posttest sensitization
The posttest can become a learning
experience. "For example, the posttest might
cause certain ideas presented during the
treatment to 'fall into place' " (p. 477). If the
subjects had not taken a posttest, the
treatment would not have worked.
36
SXCE,Patna

Interaction of
history and
treatment effect
(Ecological)
treatment effect
(...to everything there is a time...)
Not only should researchers be cautious about
generalizing to other population, caution should be
taken to generalize to a different time period. As
time passes, the conditions under which treatments
work change.
37
SXCE,Patna

(Ecological)
Measurement of
the dependent
variable
(maybe only works with M/C tests)
A treatment may only be evident with
certain types of measurements. A teaching
method may produce superior results
when its effectiveness is tested with an
essay test, but show no differences when
the effectiveness is measured with a
multiple choice test.
variable
38
SXCE,Patna

Interaction of time
of measurement
and treatment
(Ecological)
and treatment
effect
(it takes a while for the treatment to kick in)
It may be that the treatment effect does not occur until several
weeks after the end of the treatment. In this situation, a
posttest at the end of the treatment would show no impact,
but a posttest a month later might show an impact.
39
SXCE,Patna

First, and foremost, an experiment
must have internal validity. If the
researchers cannot be certain that the
results of the experiment are
dependent on the treatment, it does
not matter to which people or
situations they wish to generalize
(apply) their findings. The importance
(apply) their findings. The importance
of external validity is reliant on
having internal validity in much the same way that the validity
of a measurement instrument is reliant on the instrument
being reliable.
However, the more tightly experimenters design their study,
the more they limit the populations and settings to whom they
can generalize their findings.
40
SXCE,Patna

41
SXCE,Patna

Suppose a researcher wants to
study the effect of a reading
program on reading achievement.
She might implement the reading program with a
group of students at the beginning of the school year
and measure their achievement at the end of
the year.
X
the year.
O
This simple design is known as a one-shot case
study design.
42
SXCE,Patna

Unfortunately, the students’ end of year reading scores
could be influenced by other instruction in school, the
students’ maturation, or the treatment.
We also do not know whether the students’ reading skills
actually changed from the start to end of the school year.
We could improve on this design by giving a pretest at the
start of the study.
X O
start of the study.
O
This is known as a one-group pretest-posttest design.
43
SXCE,Patna

Unfortunately, the students’ end of year reading scores still
could be influenced by other instruction in school, the
students’ maturation, or the treatment.
Our researcher may wish to have a comparison group.
X O
O
O
O
This is a static-group pretest-posttest design.
44
SXCE,Patna

If our researcher believes that the pretest has an impact
on the results of the study, she might not include it.
X O
O
O
O
This is a static-group comparison design.
45
SXCE,Patna

Because our researcher did not pretest, she might wish
to randomly assign subjects to treatment and control
group.
Random assignment of subject to groups should spread
the variety of extraneous characteristics that subjects
possess equally across both groups.
X O
O
O
O
R
R
This is a randomized posttest-only, control group design.
46
SXCE,Patna

Of course, our researcher could also include a pretest
with her random assignment.
X O
O
O
O
This is a randomized pretest-posttest control group design.
R
R
47
SXCE,Patna

Occasionally researchers combine the randomized
pretest-posttest control group design with the
randomized posttest-only, control group design.
X O
O
O
O
R
R
This is a randomized Solomon four-group design.
X O
O
R
R
O
O
R
48
SXCE,Patna

With the randomized Solomon four-group design, all
groups are randomly assigned
X O
O
O
O
R
R
and given the posttest.
Two of the groups are given pretests.
One of the pretest
groups is assigned to
treatmentand one of
the non-pretest groups
X O
O
O
O
R
R
R the non-pretest groups
is assigned to
treatment.
49
SXCE,Patna

Each of the designs described in this section has
advantages and disadvantages that influence the
studies internal and external validity.
50
SXCE,Patna

Experimental threats

More Related Content

What's hot (19)

Similar to Experimental threats (20)

More from Vikramjit Singh (20)

Recently uploaded (20)

Experimental threats