module 6 lesson 2
- 1. MASTER’S COMPREHENSIVE EXAMINATION AND DOCTORAL PRELIMINARY EXAMINATION SAMPLE RESPONSES – SPRING 2009 RESEARCH DESIGN 1. A university research group in Eastern Maryland is interested in investigating the relationship between two constructs in elderly populations, “Mental Agility” and “Physical Activity”. The researchers decide to conduct a study with 18 residents at a retirement community in Annapolis. The age range of the 18 residents is 72-79 years and four of the residents are male. To investigate the relationship between the two constructs of interest, the researchers administer an IQ-test to the residents and also ask them about their exercise habits. Specifically, they ask the subjects how they would rate their exercise activity on a five-point scale from 1 = “not very active” to 5 = “highly active”. The researchers correlate the scores from the IQ test with the activity ratings and find a correlation of .31, which is statistically significant at α = .10. They claim that their findings prove that physical activity causes the elderly to have a high degree of mental agility. As an independent researcher you were asked to comment on the validity of the conclusions drawn by the primary researchers. Sample answers a. Identify at least four threats to the inferential validity in this study and explain their underlying reasons. First, the retirement community is locally selected, which makes the sample of respondents a convenience sample. This may or may not be an issue depending on the degree to which the sample is representative in its composition of the general population. Given that the retirement community is in a rather wealthy location – Annapolis – it is questionable to what degree the sample shares key characteristics with the general population of elderly about which inferences are desired. Furthermore, the responses of the retirees are going to be dependent due to the shared context. This may or may not be a statistical issue depending on the degree of dependency. Second, the sample is rather small, which will mean that any model effects are going to be rather imprecisely estimated and non-parametric models might be better models of choice. Third, both measurements contain some degree of measurement error. The rating scale is an imprecise measurement of the degree of physical activity because “active” can be interpreted in a variety of different ways. The IQ measure also contains measurement error, because intelligence is a latent characteristic. Moreover, it is questionable whether mental agility should be equated with general intelligence. 1
- 2. Fourth, the study is an observational study without any controlled intervention by the researchers. Thus, any correlational effects reflect a composite of characteristics, only one of which is physical activity measured in a particular manner. It is not surprising, therefore, that the correlation is relatively low in this study given the nature of the study and the measurement error in the variables. b. Describe an alternative research design that would be more appropriate to investigate the link of mental agility and physical activity in the elderly. Provide a thorough justification for the structure of this alternative design. A designed experiment with a larger cohort of retirees would be desirable to investigate the research question of interest. From a sampling perspective, it would be important to stratify the population of retirees in an area of interest (e.g., the Mid-Atlantic region) according to relevant characteristics that impact their level of mental agility. These might include things like sex, type of job held, and some measure of the severity of medical illnesses in the past. In addition, it would seem necessary to develop particular exercise regimen – or design interventions based on existing exercise regimen – that vary systematically in their intensity and assign retirees to these programs. In practice, this would be somewhat challenging because they might be used to doing this with friends and the amount of physical activity has to be matched to the physical state of the retirees for health safety. Hence, if there are different interventions they should always be offered at the same retirement community and retirees should be matched on key characteristics of interest before being randomly assigned to the interventions. Rather than using a general intelligence measure to measure mental agility, it might seem more appropriate to use an age-appropriate measure of fluid intelligence and to supplement this measure with tasks that require a reliance on quick information processing of particular types. These tasks could be used to supplement the quantitative profile from the standardized measure and may include such things as crossword puzzles, Sudoku, or games that use geographical information. 2. Answer the following questions based on the research designs represented by the schematics below Design 1 Design 2 1 2 3 4 → → → → O X O O O 1 2 3 4 → → → → R O X O R O O Sample answer a. Name and briefly describe each design. Include all critical characteristics of each design in your explanation. 2
- 3. Design 1 is an example of a quasi-experimental design. More specifically, the schematic represents a non-equivalent control group, pre-test/post-test design (Campbell & Stanley, 1963). The groups here are intact – meaning that the assignment to groups is not under the experimenter’s control. The “O’s” in the schematic represent observations (pre-test and post-test, respectively) and the assignment of X (the treatment) is assumed to be random and under the control of the investigator. Design 2 is an example of a true experimental design. One could classify this design as an equivalent control group, pre-test/post-test design. The equivalency is achieved through randomization. That is, subjects are randomly assigned to one of the two groups – either the group that receives the treatment (with X) or the control group (without X). b. State any threats to internal validity that may exist for each of the two designs. Provide a brief explanation describing how these threats might produce effects confounded with the treatment effect if they are not controlled for in the design. Campbell and Stanley (1963) and Cook and Campbell (1979) suggest that there are 7 – 8 different classes of nuisance variables that, if not controlled, might produce effects confounded with the treatment effect. They are: history, maturation, testing, instrumentation, regression to the mean, selection, mortality (attrition), selection- maturation interaction. Design 2, the true experiment, controls these rival hypotheses listed above. However, this is not the case for the quasi-experimental design outlined in design 1. According to Campbell and Stanley (1963), there are two potential threats to internal validity with the use of design 1. Of primary concern is the interaction between selection and maturation. For example, if the experimental group consists of children with low reading ability, and the control group some other convenient population tested and retested, a gain specific to the experimental group (afterschool reading program) might be interpreted as a spontaneous gain in reading achievement specific to such an extreme group. That is, a gain would have occurred even without the presence of X, and therefore is confounded with X. Regression to the mean provides the other major threat to internal validity. This can occur when either of the comparison groups (treatment or control), has been selected on the basis of its extreme scores on O1. Then, the degree of difference between pre-test and post-test between the two groups could very well be the by-product of regression – groups selected on the basis of extreme scores – and not on the experimental condition, X. c. Provide a discussion of the concept of causality and how it relates to the selection or construction of a suitable research design. 3
- 4. Margenau (1950) pointed out that “the words cause and effect are among the most loosely used in our language” (p. 389), and declared that science and scientists cannot be of much help in clarifying their meaning as they “are not primarily scientific terms.” With that said, the type of design has important implications for the validity of conclusions, inferences, and generalizations from research. While this discussion could take on many paths, germane to the discussion here will be the link between theory, internal validity and causal inference. Theory. One important link in establishing causation is a theoretical model. Theory is aimed at organizing and explaining specific aspects of the environment. Underlying scientific theory is that testable hypotheses may be derived from it. Theory provides the researcher with a selective point of view – an orientation for what to look for, which variables are relevant and which are superfluous. Internal Validity. Internal validity refers to the validity of assertions regarding effects of the independent variable(s) on the dependent variable(s). In the form of a question, this can be summarized as: Is what as taken place (i.e., the phenomenon observed) due to the variables the researcher claims to be operating (e.g., the manipulated variables), or attributable to other extraneous variables? It is clear that the “validity” of answering this question relies on the plausibility of alternative explanations. And it is here where control plays such an integral part. In practice, two general approaches are used to control the effects of extraneous variables: (i) experimental control and (ii) statistical control. These relate to the aforementioned design types in the following manner. Statistical Control In quasi-experimental designs, like that corresponding to design 1 above, in which random assignment of subjects to treatments is not possible or potentially unethical, statistical control is achieved through adjusting the estimated treatment effect by means of controlling for pre-existing group differences on the extraneous (concomitant) variable(s). This adjustment can be striking especially when the difference on the concomitant variable(s) across intact treatment groups is dramatic. Using ANCOVA or other statistical methods to equate groups on important covariates should not be viewed as a substitute for randomization. Control of all potential concomitant variables is not possible in quasi-experimental designs; and therefore, is always subject to threats to internal validity from unidentified covariates. Experimental Control Random assignment to experimental conditions, like that represented in design 2 above, ensures that any idiosyncratic differences among the groups are not systematic at the outset of the experiment. Random assignment does not guarantee that the groups are equivalent; rather that any observed differences are due only to chance.Various authors have asserted that it is only through variable manipulation (e.g., random assignment) that one may hope to study causation (e.g., Holland, 1986). 4
- 5. In sum, causation is a nebulous term that is often misused in the social/behavioral sciences. Establishing causation necessitates among numerous other things, providing a theoretical model which show the relationships between relevant variables as well as excluding irrelevant ones. Choosing a research design which provides the strongest internal validity evidence is crucial. In terms of internal validity, an experiment where subjects are randomly assigned to treatment conditions is the “gold standard” against which other designs are measured. When random assignment is not possible, internal validity evidence can be accumulated through the use of statistical control. Assessment 1. Imagine that you are the district coordinator of three large school systems in Northern Virginia. You have been directed to develop an integrated assessment system for measuring third- and fourth-grade students’ learning progress in basic arithmetic skills (e.g., addition, subtraction, multiplication, division, and fractions). Each school system has adopted a different textbook for teaching the relevant content, and they are using the benchmark assessments (for unit tests and end-of-marking-period tests) that were provided by the publisher of the textbooks. In addition, each school system is using a standards-based standardized large-scale assessment administered to all students at the end of each grade. The results of the new integrated assessment system will be utilized by three different groups for instructional and policy decision-making: Teachers will use the results to make decisions about instruction; parents will use the results to determine how their children are doing in school; school principals and district representatives will use the results to make school accountability judgments. Sample Answers a. Describe the assessment components and associated feedback mechanisms that you believe should be put into place to provide accurate, reliable, and valid information that will meet the needs of these different stakeholder groups. In order to develop an integrated assessment system, one would have to, ideally, align the targeted curriculum, the enacted curriculum, the non-standardized assessment practices, and the standardized assessment practices. To do this efficiently and effectively, intensive amounts of professional development will be required for teachers and district specialists. Furthermore, parents will need to be informed about the structure of the assessment systems and its impact on their children’s learning progressions. A meaningful starting point would be to link and augment existing standards-based assessments, benchmark assessments, and key classroom assessments. One could call into life an interdisciplinary expert group with teachers, curriculum developers, textbook 5
- 6. developers, and district specialists to (a) review the key competencies targeted by each assessment, (b) the linkages between the assessments, and (c) the desired interpretations from each assessment. The next step could be to augment these assessments with additional tasks that strengthen existing linkages and to develop remedial materials that can be deployed into the state for use by the teachers. To make such a system effective it would seem that continual training of teachers through workshops would be required alongside the development and deployment of computer-assisted management systems for uploading data and downloading reports. b. Using two assessment examples in this integrated system, describe the key principles that you would use to design, implement, and score the assessments given their respective feedback functions in the system. The resulting assessment system could target more narrow competencies in particular units in more depth, via classroom and benchmark assessments, and broader competency domains across units in less depth, via large-scale standardized assessments. Reporting for all stakeholders would require that simple statistics such as sum-scores, percentages correct, or transformed scale scores be used, which are based mostly on classical test theory. More complicated models such as item response theory models would be used “behind-the-scenes” for the large-scale standardized assessments only. The interdisciplinary expert team, in collaboration with a measurement specialist, might use these models to create simple score reports. As an example of what teachers might do, consider having them use an on-line interface or an Excel spreadsheet. They could be trained to check of particular competencies that they measure with their classroom assessments. Only those assessments that provide reasonably reliable information about individual students should be tracked while more impressionistic data could be added and weighted accordingly. As stated above, the classroom assessments should target narrow competencies (e.g., reasoning with simple rational numbers) linked to state standards such that reliable interpretations about subskills could be obtained. The benchmark assessments should cover broader competency domains that arch over the content covered by a series of classroom assessments. They would probably contain a larger fraction of selected response items and simple constructed response questions than other assessments. Ideally, they would be administered and scored in a standardized manner and would be linked across the districts via common items. The results would provide information about the performance of individual students to the teachers, students, and parents, and information about the performance of student groups (e.g., by sex, school, or school district) for district specialists. 2. Two of the most critical concepts in measurement are reliability and validity. Despite the fact that these concepts are relatively simple to describe at a broad conceptual level, many people find it challenging to address their more practical applications. One context that is 6
- 7. often seen as particularly challenging is non-standardized educational classroom assessment. Sample Answers a. Explain the similarities and differences between the operationalization of reliability in large-scale standardized assessment contexts and its operationalization in classroom assessment contexts. Provide at least two examples to support your explanation of the similarities and differences. Choose examples that can be considered realistic, rather than contrived. Reliability is concerned with the consistency or replicability of scores, rather than the interpretations that are drawn from the scores. This meaning of score quality is similar to both large-scale and classroom assessment even though the means by which it is empirically assessed can differ. In a classical test theory framework, reliability is the ratio of true-score variance to observed-score variance or the amount of observed score variation that can be attributed to true inter-individual differences. In an item response theory framework, reliability is a marginal quantity that is less frequently used. More commonly, the information of a test at particular values of the latent score range is of most interest similar to the conditional standard error of measurement in classical test theory. In large-scale assessment, we typically distinguish between three types of reliability of consistency, (a) parallel forms, (b) test-retest, and (c) internal consistency. It is quite common in large-scale assessment to develop structurally similar test forms to minimize cheating or to re-administer the same test form to obtain an estimate of temporal stability. In classroom assessment this is not realistic, however. If teachers utilize assessments from a textbook or make up their own assessment, they may compute an internal consistency coefficient such as coefficient alpha for their scores. Realistically, however, this only happens when such activities are supported by professional development efforts, a commitment of the district to data-driven instruction, and easy-to-use interfaces for tracking the data. b. Explain the similarities and differences between the investigation of validity in large- scale standardized assessment contexts and its investigation in classroom assessment contexts. Provide at least two examples to support your explanation of the similarities and differences. Choose examples that can be considered realistic, rather than contrived. Validity is concerned with the defensibility of interpretations that are drawn from test scores, at least in the most common conceptualization. Thus, scores need to be reliable to support potentially valid interpretations, but even if they are reliable they may be interpreted in inappropriate ways by decision-makers. In large-scale assessment a lot of effort is devoted to comprehensively document evidence for different aspects of validity. For example, the test development process, from the construct definition in the target domain to its operationalization via test items, their administration and scoring, and the reporting need to be meticulously documented. In addition, secondary measures that are 7
- 8. assumed to positively correlate with test scores (e.g., general ability measures, working- memory measures, alternative achievement measures) and negatively correlate with test scores (e.g., measures of unrelated constructs) are typically administered alongside the test of interest to investigate criterion-related evidence for validity. In classroom assessments, the primary emphasis is on content validity, which is often documented by the design of the assessment and the sequence of assessments overall. Investigations of criterion-related evidence for validity are typically not administered even though school principals might correlate particular grades across particular domains for documentation or research purposes. If the assessment system that is in use in a particular district was thoughtfully designed, then the weighting of the individual assessments within the system might also be rationally used to argue that they structurally match the relative importance of the measured competencies in the target domain. 8