Adlt673 session 5_quantitative_experimental_nonexperimental

Virginia Commonwealth University
ADLT 673 –Teaching as Scholarship in Medical Education
Kelly Lockeman, PhD

EXPERIMENTAL NONEXPERIMENTAL
 There is an intervention that
the researcher manipulates.
 Researcher has direct control
over the variables.
 Some say this is the only way
to truly determine cause.
 “The way things could be.”
 The researcher does not
manipulate anything.
 Researcher does not have
direct control.
 Researcher can only describe
variables and relationships.
 “The way things are.”
What’s the difference?

 Why do we do experimental studies?
 To make causal inferences about the relationship between
the independent and dependent variables
 What do we do?
 Directly manipulate the independent variable
 Control for extraneous variables
▪ Eliminate the variable from the study
▪ Statistically adjust for the effect of the variable
 What are the difficulties?
 Difficult to carry out in educational settings
 Difficult to control all factors that might affect the outcome

Program ObservationCauses
What you do What you see
Intervention orTreatment
IndependentVariable
DependentVariable
In this study…
alternative
cause
alternative
cause
alternative
cause
alternative
cause

 Two types of experimental validity:
1. InternalValidity – the extent to which the
independent variable, and not other extraneous
variables, produce the observed effect on the
dependent variable
2. ExternalValidity – the extent to which the
results are generalizable
 Each types of experimental validity can be
threatened by certain factors.

D – Diffusion ofTreatment
I – Instrumentation
S – Selection
H – History
M – Maturation
A – Attrition
T – Testing
S – Statistical Regression
D I S H M AT S McMillan (2011), p. 222

Diffusion ofTreatment the treatment is [inadvertently] given to the control group
Instrumentation
poor technical quality (validity, reliability) or changes in
instrumentation
Selection
groups that are not equal due to differences in the
participants in those groups (e.g., positive and negative
attitudes, high and low achievers)
History
extraneous events (e.g., the crash of the stock market, 9/11)
have an effect on the participants' performance on the
dependent variable
Maturation participants' maturation over the course of the study
Attrition differential loss of participants from groups
Testing the effect of having taken a pretest
Statistical Regression the natural movement of extreme scores toward the mean

 Participants sometimes perform very well or very
poorly on a measure because of chance factors
(e.g., luck, lack of awareness)
 These chance factors are not likely to be present in
a second testing, so their scores will not be so
extreme– the scores “regress to the mean.”
 These regression effects, not the effect of
treatment, may account for changes in
participants’ performance over time.

1. Subjects
▪ Representativeness of the sample in comparison to the population
▪ Consistency of the results across subgroups within the sample
▪ Personal characteristics of the subjects
▪ Subject's awareness of being involved in a study
2. Situations - characteristics of the setting (e.g., specific
environment, special situation, particular school, etc.)
3. Time - explanations can change over time
4. Treatments - specific way in which an experimental
treatment is conceptualized, operationalized, and
administered
5. Measures
▪ Different instruments measure content or constructs differently
▪ Measures change across studies

 Pre-experimental
 Quasi-experimental
 True experimental
 Factorial

 R – indicates random selection or random
assignment
 O – indicates an observation
▪ Test
▪ Observation score
▪ Scale score
 X – indicates a treatment
 A, B, C, ... – indicates a group

 Pre-experimental designs do not control threats
to internal validity very well.
 Single group pretest only
▪ A X O
 Single group pretest posttest
▪ A O X O
 Non-equivalent groups posttest only
▪ A X O B O
Key
R = Random
O = Observation
X =Treatment
A, B, C,… = Groups

 Quasi-experimental designs do not control
threats to internal validity very well.
 Non-equivalent pretest-posttest,
experimental control groups
▪ A O X O B O O
 Non-equivalent pretest-posttest,
multiple treatment groups
▪ A O X1 O B O X2 O
Key
R = Random
O = Observation
X =Treatment

 Important components
 Random assignment
▪ Participants are placed into groups using a random procedure
▪ This ensures equivalency of the groups
 Random selection of subjects
▪ Participants are chosen from a population using random procedures
▪ This ensures generalizability to the population from which the
participants were selected (i.e., external validity)
 Threats to internal validity
 Controls for selection, maturation, and statistical regression
 Likely to control for most other threats
 SeeTable 8.3 (McMillan, p. 232) for specific threats related to
each design

 Types
 Randomized posttest only experimental control groups
▪ R A X O R B O
 Randomized posttest only multiple treatment groups
▪ R A X1 O R B X2 O
 Randomized pretest-posttest
experimental control groups
▪ R A O X O R B O O
 Randomized pretest-posttest
multiple treatment groups
▪ R A O X1 O R B O X2 O
Key
R = Random
O = Observation
X =Treatment

 A special type of experimental research design
containing two or more independent variables
 Types of effects
 Main effects
▪ There is a main effect for each independent
variable
 Interaction effect
▪ A different effect for the level of the first
independent variable across the levels of the
second independent variable

 A special form of experimental research
 Designs in which the effect of an experimental
treatment is studied for one participant
 Repeated measurement of the dependent variable
before, during, and after implementing treatment
 Not restricted to one (1) participant, but rarely
involves more than three (3) participants
 Used extensively in studies involving exceptional
children or counseling

 Characteristics
 Reliable measurement
 Repeated measurement
 Clear description of the conditions
 Baseline and treatment conditions
 One variable investigated
 Notation
 A indicates a baseline condition without treatment
 B indicates a treatment condition
 Two Common Designs…

1. A B A
 Procedure:
▪ Multiple observations during initial baseline time frame
▪ Multiple observations during treatment implementation
▪ Treatment withdrawn
▪ Multiple observations during the second baseline time frame
 Variation: A B A B (i.e., adding a second treatment phase)
 Limitations:
▪ Complicated statistical analysis of the data
▪ Interpretation of specific outcomes (e.g., a lasting effect of treatment that
does not diminish in the second baseline observations)
2. Multiple baseline designs
 Extension of the A B A design to include more than one subject,
behavior, or setting
 These designs enhance the generalizability of the results

Adlt673 session 5_quantitative_experimental_nonexperimental

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Adlt673 session 5_quantitative_experimental_nonexperimental (20)

More from tjcarter (20)

Recently uploaded (20)

Adlt673 session 5_quantitative_experimental_nonexperimental