Research Design and Validity
5 likes6,144 views
This document summarizes a research study on complex problem solving using a computer simulation called Syntex. The study involved 54 students from Zhejiang University divided into 18 groups. The groups made management decisions for a simulated company each month. The researchers found differences between the "best" and "worst" groups based on company performance. Best groups increased capital and hired more employees over time compared to worst groups. The researchers also analyzed decision making processes and information gathering between the groups. They explored perspectives including the problem solving process, group interactions, and cross-cultural differences. The document discusses research design, validity, variables measured, and limitations of generalizing the results.
![8
The differences between “best” and “worst” groups
The Performance＝100*[2*Z（capital）+Z（number）+Z（satisfaction）]/4
Sorting the groups by score, selecting higher 8 groups as best groups, lower 8 groups as worst groups. ( t(14)=4.34, p=.001 )
Z=0Normal Distribution
The change of capital during 12 months:
Better groups can increase the capital
The size of the company : better groups hire
more people than worse
The change of employees’ satisfaction: both groups
can’t manage very well, worse groups are worse](https://image.slidesharecdn.com/ss02cpsdesignvalidtyfinal-100615022930-phpapp02/85/Research-Design-and-Validity-8-320.jpg)
Research Design and Validity
- 1. Quantitative Research Methodologies (2/6): Research Design and Validity Prof. Dr. Hora Tjitra & Dr. He Quan www.SinauOnline.com
- 2. 2 Research Team Research Program: Complex Problem Solving (CPS) Hora Tjitra, HE Quan, LIU Manxia, SHI Wei, ZHANG Qi, YU Zhaoyang, WANG Huiqi Project Team Leader Complex Problem Solving Research project Project Team Member The Police Bureau, Huzhou Project Team Member Zhongcai Group, Hangzhou Phd. Candidate, Zhejiang University Industry and Organization Psychology Bachelor Degree, Zhejiang University Industry and Organization Psychology
- 3. 3 Complex Problem Solving Stories Economy Environment Technology Education Improve The Well Being Of Citizens in One Nation Politics Social The Moral Issue aroused by the Biological Technology Overexploiting The Underground Water in Beijing City
- 5. CPS Research Design Observation System • Group Interaction • Decision Making Process Syntex System Data • Corporate Results • Group Knowledge • Group Decisions Questionnaire • Quantitative Data Interview Coaching • Group Reflection SYNTEX • Computer Business Simulation • Managing Complex Situations Multi-Methods Approach AmericaChina IndonesiaGermany France Japan Cross-Cultural Perspective
- 6. 6 Background Information and Experimental Process Participants Make decisions by themselves for company every month, for example the products order and the material purchasing. The Task 3-persons cooperate, 54 persons divided in 18 groups, the students coming from Zhejiang University Program Display
- 7. 7 Product Quality The system structure behind Syntex business simulation Material Storage Purchasing Materials Employee Satisfaction Market Demand HR Performance Product Storage Advertisement Salary Pricing Policy Production Order Recruitment Social Fund Product Manufacture Product Sales Company Income Season / Weather Competitor Price / Quality Purchasing + - - - + + + + + + + + + Employee Number Oval represents the control variable, rectangle: system variable, ‘+’means active influence,’ –’mean negative influence . Company Assets Maintenance Machine / Diesel Leadership HR Work Load Lay-off Exchange Resign + - + - + + + - Number and Assortment Sale PriorityMan. Priority and • Bank Account • Material Storage • Product Storage • Machine, Diesel
- 8. 8 The differences between “best” and “worst” groups The Performance＝100*[2*Z（capital）+Z（number）+Z（satisfaction）]/4 Sorting the groups by score, selecting higher 8 groups as best groups, lower 8 groups as worst groups. ( t(14)=4.34, p=.001 ) Z=0Normal Distribution The change of capital during 12 months: Better groups can increase the capital The size of the company : better groups hire more people than worse The change of employees’ satisfaction: both groups can’t manage very well, worse groups are worse
- 9. 9 Information and Decision Making Difference Best groupsBest groups Worst groupsWorst groups Variables M SD M SD Number of questions, total 124 29.76 123 20.71 Number of decisions, total 55 11.01 61 13.63 MANOVA( Repeated Measure) Information Decision Condition F(1/14)=.00 p=.95 F(1/14)=1.02 p=.33 Month F(5.69/79.72)=7.06 p=.00 F(8.51/69.18)=3.67 p=.00 Interaction F(5.69/79.72)=1.31 p=.26 F(8.51/69.18)=1.29 p=.25
- 10. 10 The Analyzing Perspectives from the Data AmericaChina IndonesiaGermany France Japan 3.Cross-Cultural Perspective 1.CPS Process Perspective How do groups explore and manage the system? What is the difference between the best and worst groups in cognitive perspectives 2.Group Interaction Perspective How do group member discuss and inquire the information? How do group member support each other?
- 11. 11 Variables Used in Experimentation Dependent Variable Independent Variable Variable …One of the antecedent conditions manipulated by the experimenter …The response of the organism; the variable that measures the influence the independent variable …Any characteristic or phenomenon that can vary across organisms, situations, or environments
- 12. 12 Three Main Research Designs Using Experimental Approach A design in which the influence of extraneous variables is controlled for while the influence of the independent variable is tested Research design: The outline, plan, or strategy used to investigate the research problem A research design in which an experimental procedure is applied but all extraneous variables are not controlled Experimental design Quasi- Experimental design Single-case design A research design in which a single groups of individuals is used to investigate the influence of a treatment condition
- 13. 13 Experimental Design • the experimental and the control groups‘ posttest scores are compared to assess the influence of the treatment condition X Y2 Y1 treatment measure Posttest only design Pretest-posttest design • the treatment effect is assesses by the difference between the experimental and control groups‘ pre- and posttest scores X Y2 Y1 treatment measurePremeasure Y Y
- 14. 14 Quasi-Experimental Design Nonequivalent comparison group design Interrupted time-series design • The results obtained from nonequivalent experimental and control groups are compared Premeasure treatment measure Y Y X Y1 Y2 • A treatment effect is assessed by comparing the pattern of pre- and posttest scores of one group of research participants X Y treatment measurePremeasure Y Y Y
- 15. 15 Basic Types of Designs for Case Studies CONTEXT Case CONTEXT Case CONTEXT Case CONTEXT Case CONTEXT Case single-case designs multiple-case designs holistic (single- unit of analysis) CONTEXT Case Embedded Unit of Analysis 1 Embedded Unit of Analysis 1 CONTEXT Case CONTEXT Case CONTEXT Case CONTEXT Case embedded (multiple units of analysis) SOURCE: COSMOS Corporation
- 16. 16 Research Validity ... the best available approximation to the truth or falsity of propositions, incl. propositions about cause 1. Is there a relationship between two variables ? 2. Given that there is a relationship, is it plausibly causal from one operational variables to the other or would the same relationship have been obtained in the absence of any treatments of any kind ? 3. Given that the relationship is plausibly causal and is reasonably known to be from one variable to another, what are the particular cause and effect constructs involved in the relationship ? 4. Given that there probably a causal relationship from construct A to construct B, how generalizable is this relationship across persons, settings, and times ?
- 17. 17 Four Types of Validities ... Validity of the inference made about whether the independent and dependent variables co-vary ... Validity of the inference that independent and dependent variables are causally related ... Validity of the inference about the higher-order constructs from the operations used to represent them ... Validity of the inference about whether the causal relationship holds over people, setting, treatment variable, measurement variables, and time Statistical Conclusion Validity Internal Validity Construct Validity External Validity
- 18. 18 Internal Validity ... a relationship between two variables is causal or that the absence of a relationship implies the absence of cause. A Causes / Treatments / Independent Variables B Effects / Outcomes / Dependent Variables C2 Comparison / Experimental / Control Units C1 Extraneous Forces / Controlled Setting 1. Is there a relationship between A & B ? 2. Is there a causal relationship from A to B ? 3. Is there any other reason that could also caused B apart from A ?
- 19. 19 Internal Validity - Threats ... Validity with which statements can be made about whether there is a causal relationship from one variable to another in the form in which the variables were manipulated or measured. • History ... an event which takes place between the pretest and posttest. • Maturation ... the respondent‘s growing older, wiser, stronger, more experienced, and the like between the pretest and posttest. • Instrumentation ... a change in the measuring instruments (e.g. observer learning effects) • Testing ... Changes in person’s score on the second administration of a test as a result of previously having taken the test ... is a threat when an observed effect might be due to ... • Regression artifact ... The tendency for extreme scores to become less extreme on a second assessment • Attrition ... Some people do not show up for the study or do not complete it • Selection ... The choice of participants for the various treatment groups is based on different criteria • Additive and interactive effects ... The combined effect of several threats to internal validity
- 20. 20 ... the approximate validity with which we can make generalizations about high-order constructs from research operations. Construct Validity of Causes and Effects A Independent Variables B Effects / Outcomes / Dependent Variables Causes / Treatments C2A Independent Variables C2B Effects / Outcomes / Dependent Variables Manipulated Group 1 Control Group C3A Independent Variables C3B Effects / Outcomes / Dependent Variables Placebo Manipulated Group 2
- 21. 21 Construct Validity - Threats • Inadequate Preoperational Explication of Constructs …the choice of operations should depend on the result of a conceptual analysis of the essential features of a construct (e.g. Aggression). • Mono-Operation/ Method Bias …many experiments are designed to have only one exemplar of a particular possible cause, and some have just one measure to represent each of the possible effect constructs. • Reactive Self-report Changes ... it is possible that the treatments effects are due to the respondents being willing to present themselves to experimenters in ways that would lead to favorable evaluation. • Experimenter Effects ... which indicates that experimenter‘s attitudes/expectancies can bias the data obtained. • Treatment Diffusion …individuals in one treatment group receive some or all of another group’s treatment • Restricted Generalizability Across Constructs …sometimes treatments will affect dependent variables quiet differently, implying a positive effect on some construct and an unintended negative effect on another
- 22. 22 ... the presumed causal relationship can be generalized to and across alternate measures of the cause and effect and across different types of persons, settings, and times. (1) External Validity ... the approximate validity with which conclusions are drawn about the generalizability of a causal relationship to and across populations of persons, settings, and times. (2) 1. Definition of the Target Populations (persons, settings, or times) 2. Samples to Represent These Populations Types of Samples / Populations: 1. Target Populations 2. Formally Representative samples that correspondent to known populations 3. Sample Actually Achieved (in Field Research) 4. Achieved Populations
- 23. 23 The categories of the external validity ... The extent to which the results of a study can be generalized to the larger population Population Validity ... The extent to which the results of a study can be generalized across settings or environmental conditions Ecological Validity ... The extent to which the results of a study can be generalized to the larger population Temporal Validity ... The generallizability of results across different but related dependent variables Outcome Validity Treatment Variation Validity ... The generalizability of results across variation of the treatment
- 24. 24 External Validity - Threats • Interaction of Selection and Treatment In which categories of persons can a cause effect relationship be generalized? • Interactions of Setting and Treatment Can a causal relationship obtained in a factory be obtained in a military camp or an university campus? • Interaction of History and Treatment To which period in the past and future can a particular causal relationship be generalized?
- 25. 25 The literatures Altschuler, L., Sussman,N.M., & Kachur, E. (2003). Assessing changes in intercultural sensitivity among physician trainees using the intercultural development inventory. International Journal of Intercultural Relations, 27(4), 387–401. Leclerc, D., & Martin, J.N. (2004).Tour guide communication competence: French, German and American tourists' perceptions. International Journal of Intercultural Relations(4), 28,181-200. Palthe, J. (2004).The relative importance of antecedents to cross-cultural adjustment: implications for managing a global workforce. International Journal of Intercultural Relations, 28(1), 37-59. Strohschneider, S., & Guess, D. (1999). The Fate of the Moros: A Cross-cultural Exploration of Strategies in Complex and Dynamic Decision Making. International Journal of Psychology,34(4), 235-252
- 26. @ Tjitra,2010 Thanks You Any comments & questions are welcome Contact me at hora_t@sianuonline.com 26 www.SinauOnline.com