Experimental ProceduresThe specific experimental design procedur.docx
- 1. Experimental Procedures The specific experimental design procedures also need to be identified. This discussion involves indicating the overall experiment type, citing reasons for the design, and advancing a visual model to help the reader understand the procedures. • Identify the type of experimental design to be used in the proposed study. The types available in experiments are pre- experimental designs, quasi-experiments, true experiments, and single-subject designs. With pre-experimental designs, the researcher studies a single group and provides an intervention during the experiment. This design does not have a control group to compare with the experimental group. In quasi- experiments, the investigator uses control and experimental groups but does not randomly assign participants to groups (e.g., they may be intact groups available to the researcher). In a true experiment, the investigator randomly assigns the participants to treatment groups. A single-subject design or N of 1 design involves observing the behavior of a single individual (or a small number of individuals) over time. • Identify what is being compared in the experiment. In many experiments, those of a type called between-subject designs, the investigator compares two or more groups (Keppel & Wickens, 2003; Rosenthal & Rosnow, 1991). For example, a factorial design experiment, a variation on the betweengroup design, involves using two or more treatment variables to examine the independent and simultaneous effects of these treatment variables on an outcome (Vogt, 2011). This widely used behavioral research design explores the effects of each treatment separately and also the effects of variables used in combination, thereby providing a rich and revealing multidimensional view. In other experiments, the researcher studies only one group in what is called a within-group design. For example, in a repeated measures design, participants are assigned to different treatments at different times during the
- 2. experiment. Another example of a within-group design would be a study of the behavior of a single individual over time in which the experimenter provides and withholds a treatment at different times in the experiment to determine its impact. • Provide a diagram or a figure to illustrate the specific research design to be used. A standard notation system needs to be used in this figure. A research tip I recommend is to use a classic notation system provided by Campbell and Stanley (1963, p. 6): X represents an exposure of a group to an experimental variable or event, the effects of which are to be measured. O represents an observation or measurement recorded on an instrument. Xs and Os in a given row are applied to the same specific persons. Xs and Os in the same column, or placed vertically relative to each other, are simultaneous. The left-to-right dimension indicates the temporal order of procedures in the experiment (sometimes indicated with an arrow). The symbol R indicates random assignment. Separation of parallel rows by a horizontal line indicates that comparison groups are not equal (or equated) by random assignment. No horizontal line between the groups displays random assignment of individuals to treatment groups. There are several threats to validity that will raise questions about an experimenter’s ability to conclude that the intervention affects an outcome and not some other factor. Experimental researchers need to identify potential threats to the internal validity of their experiments and design them so that these threats will not likely arise or are minimized. There are two types of threats to validity: (a) internal threats and (b) external threats. Internal validity threats are experimental procedures, treatments, or experiences of the participants that threaten the researcher’s ability to draw correct inferences from the data about the population in an experiment. Table 8.5 displays these threats, provides a description of each one of them, and
- 3. suggests potential responses by the researcher so that the threat may not occur. There are those involving participants (i.e., history, maturation, regression, selection, and mortality), those related to the use of an experimental treatment that the researcher manipulates (i.e., diffusion, compensatory and resentful demoralization, and compensatory rivalry), and those involving procedures used in the experiment (i.e., testing and instruments). Table 8.5 Types of Threats to Internal Validity Type of Threat to Internal Validity Description of Threat In Response, Actions the Researcher Can Take -History Because time passes during an experiment, events can occur that unduly influence the outcome beyond the experimental treatment. The researcher can have both the experimental and control groups experience the same external events. -Maturation Participants in an experiment may mature or change during the experiment, thus influencing the results. The researcher can select participants who mature or change at the same rate (e.g., same age) during the experiment. -Regression Participants with extreme scores are selected for the experiment. Naturally, their scores will probably change during the experiment. Scores, over time, regress toward the mean. A researcher can select participants who do not have extreme scores as entering characteristics for the experiment. -Selection Participants can be selected who have certain characteristics that predispose them to have certain outcomes (e.g., they are brighter). The researcher can select participants randomly so that characteristics have the probability of being equally distributed among the experimental groups.
- 4. -Mortality Participants drop out during an experiment due to many possible reasons. The outcomes are thus unknown for these individuals. A researcher can recruit a large sample to account for dropouts or compare those who drop out with those who continue—in terms of the outcome. -Diffusion of treatment Participants in the control and experimental groups communicate with each other. This communication can influence how both groups score on the outcomes. The researcher can keep the two groups as separate as possible during the experiment. -Compensatory/Resentful demoralization The benefits of an experiment may be unequal or resented when only the experimental group receives the treatment (e.g., experimental group receives therapy and the control group receives nothing). The researcher can provide benefits to both groups, such as giving the control group the treatment after the experiment ends or giving the control group some different type of treatment during the experiment. -Compensatory rivalry Participants in the control group feel that they are being devalued, as compared to the experimental group, because they do not experience the treatment. The researcher can take steps to create equality between the two groups, such as reducing the expectations of the control group. -Testing Participants become familiar with the outcome measure and remember responses for later testing. The researcher can have a longer time interval between administrations of the outcome or use different items on a later test than were used in an earlier test. -Instrumentation The instrument changes between a pretest and posttest, thus impacting the scores on the outcome.
- 5. The researcher can use the same instrument for the pretest and posttest measures. SOURCE: Adapted from Creswell (2012). Potential threats to external validity also must be identified and designs created to minimize these threats. External validity threats arise when experimenters draw incorrect inferences from the sample data to other persons, other settings, and past or future situations. As shown in Table 8.6, these threats arise because of the characteristics of individuals selected for the sample, the uniqueness of the setting, and the timing of the experiment. For example, threats to external validity arise when the researcher generalizes beyond the groups in the experiment to other racial or social groups not under study, to settings not examined, or to past or future situations. Steps for addressing these potential issues are also presented in Table 8.6. Other threats that might be mentioned in the method section are the threats to statistical conclusion validity that arise when experimenters draw inaccurate inferences from the data because of inadequate statistical power or the violation of statistical assumptions. Threats to construct validity occur when investigators use inadequate definitions and measures of variables. Table 8.6 Types of Threats to External Validity Types of Threats to External Validity Description of Threat In Response, Actions the Researcher Can Take -Interaction of selection and treatment Because of the narrow characteristics of participants in the experiment, the researcher cannot generalize to individuals who do not have the characteristics of participants. The researcher restricts claims about groups to which the results cannot be generalized. The researcher conducts additional experiments with groups with different characteristics. -Interaction of setting and treatment Because of the characteristics of the setting of participants in an experiment, a researcher cannot generalize to individuals in
- 6. other settings. The researcher needs to conduct additional experiments in new settings to see if the same results occur as in the initial setting. -Interaction of history and treatment Because results of an experiment are time-bound, a researcher cannot generalize the results to past or future situations. The researcher needs to replicate the study at later times to determine if the same results occur as in the earlier time. SOURCE: Adapted from Creswell (2012). Practical research tips for proposal writers to address validity issues are as follows: • Identify the potential threats to validity that may arise in your study. A separate section in a proposal may be composed to advance this threat. • Define the exact type of threat and what potential issue it presents to your study. • Discuss how you plan to address the threat in the design of your experiment. • Cite references to books that discuss the issue of threats to validity, such as Cook and Campbell (1979); Shadish, Cook, & Campbell (2001); and Tuckman (1999). The Procedure A proposal developer needs to describe in detail the procedure for conducting the experiment. A reader should be able to understand the design being used, the observations, the treatment, and the timeline of activities. • Discuss a step-by-step approach for the procedure in the experiment. For example, Borg and Gall (2006) outlined steps typically used in the procedure for a pretest-posttest control group design with matching participants in the experimental and control groups: 1. Administer measures of the dependent variable or a variable closely correlated with the dependent variable to the research participants. 2. Assign participants to matched pairs on the basis of their scores on the measures described in Step 1.
- 7. 3. Randomly assign one member of each pair to the experimental group and the other member to the control group. 4. Expose the experimental group to the experimental treatment and administer no treatment or an alternative treatment to the control group. 5. Administer measures of the dependent variables to the experimental and control groups. 6. Compare the performance of the experimental and control groups on the posttest(s) using tests of statistical significance. Data Analysis Tell the reader about the types of statistical analysis that will be used during the experiment. • Report the descriptive statistics calculated for observations and measures at the pretest or posttest stage of experimental designs. This call for descriptive analysis is consistent with the recent APA Publication Manual (APA, 2010). These statistics are means, standard deviations, and ranges. • Indicate the inferential statistical tests used to examine the hypotheses in the study. For experimental designs with categorical information (groups) on the independent variable and continuous information on the dependent variable, researchers use t tests or univariate analysis of variance (ANOVA), analysis of covariance (ANCOVA), or multivariate analysis of variance (MANOVA—multiple dependent measures). (Several of these tests are mentioned in Table 8.3, which was presented earlier.) In factorial designs, both interaction and main effects of ANOVA are used. When data on a pretest or posttest show marked deviation from a normal distribution, use nonparametric statistical tests. Also, indicate the practical significance by reporting effect sizes and confidence intervals. • For single-subject research designs, use line graphs for baseline and treatment observations for abscissa (horizontal axis) units of time and the ordinate (vertical axis) target behavior. Researchers plot each data point separately on the graph, and connect the data points with lines (e.g., see Neuman & McCormick, 1995). Occasionally, tests of statistical significance, such as t tests, are used to compare the pooled
- 8. mean of the baseline and the treatment phases, although such procedures may violate the assumption of independent measures (Borg & Gall, 2006). Interpreting Results The final step in an experiment is to interpret the findings in light of the hypotheses or research questions set forth in the beginning. In this interpretation, address whether the hypotheses or questions were supported or whether they were refuted. Consider whether the treatment that was implemented actually made a difference for the participants who experienced them. Suggest why or why not the results were significant, drawing on past literature that you reviewed (Chapter 2), the theory used in the study (Chapter 3), or persuasive logic that might explain the results. Address whether the results might have occurred because of inadequate experimental procedures, such as threats to internal validity, and indicate how the results might be generalized to certain people, settings, and times. Finally, indicate the implications of the results for the population studied or for future research.