Part II: Design

AN INTRODUCTION TO SECOND LANGUAGE
RESEARCH METHODS: DESIGN AND DATA
Mariuxi Arcos
Andrea Arcos

What is research design?
 Research design is a set of instructions for data collection and analysis. As
examples of research design, Vogt (1999) gives experiments and quasi-experiments.
 It can be defined as an operating model or blueprint for a research project, which
accounts for internal reasoning (causality) and external reasoning (generalizability).
 The research design stipulates the parts of the research project, how they are
arranged, and how they function. However, the research design does not determine
the type of data, how the data are collected, or how they are analyzed.

How many research designs are there?
 Experimental
 Survey
 Ethnographic
 Correlational
 Case Study
 Action Research
 Qualitative
 Ex post facto
 Descriptive
 Introspection
 Grounded Theory
 Narrative
 Historical
 Evaluation
 Causal-Comparative
 Interactional analysis
 Phenomenology
 Other designs that were mentioned but not given full chapter status
include:
 Critical Theory, Heppner, Kivlighan, and Wampold (1999, p. 240)
 Constructivism, Heppner, Kivlighan, and Wampold (1999, p. 238)
 Content analysis, Neuman (2000, p. 150)
 Ethnomethodology, Babbie (2004, p. 290)
 Feminist Research Design, Neuman (2000, p. 82); Heppner,
Kivlighan, and Wampold (1999, p. 240)
 Hermeneutics, Heppner, Kivlighan, and Wampold (1999, p. 239)

How to select a research design
First, to decide on a research design is to select one that addresses
your purpose and answers your research questions.
Second, you can use the design you already know. Perhaps most of
the research papers that you have read use a certain design, and all or
most of your professors, colleagues, or other research mentors use
this design.
A third and related approach is to use the design promoted by a senior
colleague. Many researchers believe that only one design,
Fourth, it might be that your academic specialty has a preference
toward doing a research using a certain design.
Fifth, you might be intrigued by a certain design and want to learn how
it works. Therefore, you choose this design to guide your research. By
doing so, you broaden your area of expertise.

SURVEY RESEARCH DESIGN (SRD)
Survey Research Design (SRD)
defined
 The design of a survey uses several data
collection procedures to allow the teacher-
researcher-educator-educator (TREE) to
investigate a construct by asking questions of any
fact (descriptive) or opinion (explanatory) of a
sample of a population with the purpose of
generalizing the population The term survey is a
general term that allows many procedures for data
collection, including questionnaires, interviews and
observations.
 Survey design studies purport to measure a
construct which may be theoretical (for example,
opinion, beliefs, attitudes) or practical (for
example, ownership of certain objects, time spent
on certain tasks). Survey design studies, especially
those providing opinions, gather data made by
respondents after the fact. As a result, the data is
in one sense subjective and unverified and in
another sense objective.
Key components of Survey Design
 Ary, Jacobs, & Razavieh (1990, p. 411)
list five steps in the survey process:
Planning the survey, sampling or
deciding whom to survey, constructing
the instrument, conducting the survey,
and analyzing the data. Brown (2001, p.
8) lists six similar steps: planning the
survey, developing the instrument,
gathering the data, analyzing the data
statistically, analyzing the data
qualitatively, and reporting the results.

How survey design might look as
a visual
• A survey design consists of a
construct, a population of
interest, a sample, and a data
collection instrument used to
measure the construct in the
sample.
Select respondents
• The type of sampling used in
a survey determines the level
of generalization that can be
claimed by the researcher.
We assume you have
constructed a sampling frame
that is a list of all known, and
therefore possible,
respondents.

Type of Sampling
Nonprobability sampling Subjects
selected by the researcher
1. Convenience A group already formed
and easy to use
2. Purposeful Knowledgeable and
available persons
3. Snowball Selected respondents
suggest other respondents
4. Quota Stratified sampling, but not
randomly chosen
Probability sampling Subjects selected by
a random mechanism
1. Simple random Pull names out of a
hat
2. Systematic random Computer
generated numbers to select
3. Stratified The sample divided into
groups called strata
4. Cluster Groups of strata

EXPERIMENTAL RESEARCH DESIGN (EXD)
Experimental design defined
Cook and Campbell (1979) offer a
description of experimental design that
can be taken as a working definition: “All
experiments involve at least a treatment,
an outcome measure, units of assignment,
and some comparison from which change
can be inferred and hopefully attributed
to the treatment”.

Key components of experimental design
Variables can be discussed within several categories:
The dependent variable is the major variable that will be
measured (Hatch & Lazaraton, 1991, p 63). The dependent
variable is the variable of focus--the central variable--on which
the other variables will act.
The researcher selects independent variables to determine their
effect on or relationship with the dependent variables (Brown,
1988, p. 10).
A moderator variable is an independent variable that the
researcher does not consider important in the investigation
(Hatch & Lazaraton, 1991, p. 65).
Control variables are kept constant, neutralized, or otherwise
eliminated so that they will not affect the study (Brown, 1988, p.
11).
Intervening variables are abstract theoretical labels applied to
the relationship that links the independent and dependent
variables.

Threats
Threat is “any
condition which
blinds or misleads
researchers when
they interpret their
results” (Griffee,
2004).Threats are
powerful because
they rely on implicit--
and thus hidden--
beliefs and desires
that can blind us to
other realities, both
internal and external.

 Threat 1: History. History can be defined as “events, other than the
experimental treatment, occurring between pretest and posttest and
thus providing alternate explanations of effects” (Campbell & Russo,
1999, p. 80).
 Threat 2. The Hawthorne effect. It states that the mere knowledge
that one is in a study may affect behavior. Isaac and Michael (1995)
describe causes of the Hawthorne effect as: novelty; awareness that
one is a participant in an experiment; a modified environment
involving observers; special procedures; new patterns of social
interaction; and, knowledge of results in the form of daily
productivity figures and other feedback, ordinarily not
systematically available.
 Threat 3. Maturation. Maturation or maturing is the idea that
participants may change over the time of the research, and this
maturing affects the results, as opposed to the treatment. The
change may be physical (age, fatigue) or psychological (interest or
lack of interest).
 Threat 4. Instability of data. This threat results from issues such as
reliability, fluctuations in sampling persons or components, and
instability of repeated or “equivalent” measures.
 Threat 5. Testing. refers to the effect on the scores of a subsequent
test after taking an initial test.
 Threat 6. Reactivity. This occurs when the data collection
instrument interacts with the treatment or even causes the
treatment effect because it relies on self-reporting.
 Threat 7. Instrumentation. This threat concerns flaws in the testing
instruments. One threat is that the instruments are not reliable or
valid, or more likely, the researcher has not presented any reliability
or validity evidence.
 Threat 8. Regression. Also known as statistical regression, this can be
defined as “[p]seudo-shifts occurring when persons or treatment units
have been selected on the basis of their extreme scores”
 Threat 9. Selection. Selection occurs when the TREE forms the
comparison groups. The problem may be that the groups are different to
begin with, and as a result produce different scores.
 Threat 10. Mortality. Mortality refers to the loss of students in either or
both the control or experimental groups. Students often drop out of a
program, sometimes in large numbers.
 Threat 11. Researcher expectancy. When a researcher expects certain
outcomes, and thereby causes them to occur (Brown, 1995), this is
referred to as researcher expectancy.
 Threat 12. Teacher effect. The teacher effect is not a threat if a TREE is
conducting an experiment and is teaching both the control and
experimental classes.
 Threat 13. Diffusion of treatments. This refers to a situation when the
control group is given the treatment intended for the experimental group.
This is typically done by a teacher who changes the regular course of
instruction, believing the innovation to be helpful (Lynch, 1996).
 14. Ecological representativeness. This is an external threat to validity,
meaning that it is a threat or hindrance to the ability of results to be
applicable or generalizable to other classrooms or situations. Most of
the threats mentioned so far are threats to internal validity in that they
hinder the internal argument for causality. According to Beretta (1986),

CASE STUDY DESIGN (CSD)
 Case study
research. - is a
strategy for doing
social inquiry
although what
constitutes the
strategy is a matter
of some debate.
 Case Study Design Defined
 Case study is occasionally confused with experimental
single-case design--an experiment using only one
group..
 A case study or ethnographic research project may
seek to answer specific questions about occurrences
and their explanations similar to those answered by
quantitative” oriented researchers .
 One way is that ethnography covers a larger scope;
CSD is more limited and more focused. Second, an
ethnography is more likely to attempt to define the
culture included in the study, while CSD is more likely
to investigate narrowly defined topic áreas such as
classroom problems or language development. The
third way Nunan (1992) posits a possible difference is
that while both use qualitative methods, CSD can also
use quantitative data collection methods. Hamel
(1993) suggests that “a case is an in-depth study of the
cases under consideration”.

What Is a Case Study?
A case study must have
three aspects, which
can serve as a
definition: it must have
data from multiple
sources, examine
something in a real-life
context, and use theory
to generalize results.

What are its historical roots and assumptions?
Case study has a long history in anthropology dating from Bronislaw Malinowski’s work
in early 20th century Melanesia. Case study entered American sociology by means of
the sociology department of the University of Chicago, where Robert Park, a former
journalist, developed research methods for direct investigation, including open
interviews and material collection (Hamel, 1993).
The beliefs and assumptions of CSD maintain that a large group, say a culture or
society, can be understood by studying a smaller unit of that society. For Malinowski,
the key unit was an isolated tribe or village in which culture could be studied in
isolation, free from the influences of European culture.

What are the key components of the design?
 Probably the most important single concept is that of the case. Stake (1995) says a case
comprises people or programs, but not a problem, a theme, or a relationship because
these are too abstract and lack boundaries. Yin (2003) describes five components he
considers crucial for CSD: questions, propositions, analysis, linking of data to
propositions, and criteria for interpreting the findings. Questions include the research
question or questions, especially how and why questions. Propositions are the object
that is to be studied in the case. They are, or are similar to, purpose or hypothesis or
even thesis. The unit of analysis is what the research questions study. For example, for
many language teachers, the unit of analysis is students. The fourth component of case
design is linking data to the proposition. The fifth component is the stating of criteria for
success

Practical steps in getting started
1. Define the object of study
2. Select the case that will be
the object of study
3. Decide on data collection
methods.
4. Write.
5. Construct the explanation.

Part II: Design

More Related Content

What's hot (20)

Similar to Part II: Design (20)

Recently uploaded (20)

Part II: Design