Chapter 2Theories & Models Week II – Slides 1
- 1. Chapter 2: Theories & Models Week II – Slides 1 Chapter 2: Theories & Models Week II – Slides 2 What we are Covering: What is a theory? Where do theories come from? What is a model? Unit of analysis Logic models Usefulness of a logic model Additional issues in theory building Finding and focusing a research question 3 3
- 2. What is a theory? Theories are nets cast to catch what we call the world, to rationalize, to master, and to explain it. We endeavor to make the mesh ever finer & finer.“ Karl Popper, The Logic of Scientific Discovery 4 4 But, before theory . . . Ask a Question or Observe/Identify a Puzzle Goal: General Explanation What is the general phenomenon you are seeking to explain? Think in terms of concepts, not specific examples Primary interest: Explain change (a.k.a. variation) in the phenomenon of interest (a.k.a. dependent variable) 5 5 What is a theory? Theories identify key variables So we know what concepts to measure and observe Theories tell causal stories Often focusing on just one cause at a time Example: broken windows theory, which looks at the variable
- 3. disorder as a possible factor in crime Theories explain variation 6 6 Dimensions & Variation Does the variation we are interested in occur over time, across units, or both? Spatial Variation Multiple units are measured at one moment in time Cross Sectional (CS) (e.g., # of giving campaigns by each non- profit in Broome County, in 2020) Temporal Variation Repeated measurement of one unit at different moments in time Time-Series (TS) (e.g., # Broome county residents diagnosed each day with Covid19 from March 15th – September 1st) 7 7 Cross-sectional variation example 8 8
- 4. Longitudinal variation example 9 9 Dimensions & Variation Can look at both space and time variation Time-series cross-sectional (TSCS) (e.g. Binghamton University mean GPAs across majors and semesters) 10 10 TSCS example 11 11 What is a theory? Theories generate testable hypotheses Hypotheses are predictions of what will happen if a theory is correct Hypotheses can be compared with the facts, and can potentially falsify a theory
- 5. 12 12 What is theory? Theories focus on modifiable variables (Note: this is a more PA/PP specific concern) Social and policy research tends to focus upon modifiable variables as a way to offer guidance in policy and practice Modifiable and nonmodifiable variables Applied theories focus on modifiable variables—causes of an outcome that we can influence Nonmodifiable variables cannot be changed by policy or practice (example: policymaking in the US will be done under democratic process & norms) 13 13 Where do theories come from? Grand social theories Sometimes referred to as theoretical paradigms, which shape a researcher’s view of the variables and mechanisms involved in explaining human behavior Example: Rational-choice theory Individuals know all potential action that they can take Will select decision which maximizes their benefits (utility)
- 6. 14 14 Where do theories come from? Academic disciplines Such as political science, psychology, economics, etc. Induction Building up theory from empirical evidence and observation Important caution about induction: You cannot test an inductive theory with the same set of facts used to create the theory Deduction Starting from initial ideas or logical principles Often theory comes from both thought processes 15 15 Where do theories come from? Exploratory and qualitative research Linking threads of empirical evidence from exploratory studies in a field Qualitative research often used to generate theory This is very hard to do! But, when done well, is usually the most valuable research in the discipline.
- 7. 16 16 Where do theories come from? Theories, norms, and values Scientific theories are positive—about how things are Not normative—about how things should be Still theories reflect values, beliefs, and interests (Example: we study human rights violation because we normatively care about curtailing them in the future) 17 17 What is a model? A graphical or mathematical representation of two items Variables That can take on different values or assume different attributes—they vary Relationships That show how change in one variable produces change in another variable
- 8. 18 18 Why do we do this? It forces you to make your assumptions explicit Establishes that implications follow logically from assumptions 19 19 More on assumptions Explicit statement of our assumptions leads us to think precisely about our concepts What are the precise definitions? Thinking about the assumptions could lead to promising lines of research Are the assumptions in a well-known theory flawed? Assumptions do not always hold in all cases. What are the implications if the assumptions do not hold? 20 20
- 9. Why do we do this? Empirical tests of hypotheses are not the only way in which we evaluate theories: we also evaluate them on logical and other grounds. It is worth our time to “kick the tires” before we invest a lot of time collecting data 21 21 What question should we ask during this stage? Is your theory causal? It should explain how and why change in the values of the independent variable change the values of the dependent variable. Does your theory generate testable hypotheses? For a theory to be testable, it must be falsifiable You should justify how your measurements match your concepts 22 22 Hallmarks of a good model Keep it simple Connect x and y via the shortest explanatory route
- 10. Parsimonious models are better models Occam's Razor Is your model novel and interesting? Your model should make new predictions Your model should not propose explanations that are obvious to all 23 23 What is a model? A quick refresher Independent and dependent variables X Y “Cause” “Effect” Independent Dependent Causal mechanisms The process by which change in X is presumed to cause change Y 24 24 Note: we use a ton of different terms here
- 11. Left-hand right-hand They all mean the same thing 25 25 Path diagram – a basic bivariate design 26 26 Path diagram – a basic bivariate design Q: Indentify the independent and dependent variables? Q: What is the presumed causal mechanism? 27 27 What is a model? Direction of a relationship Positive (+) relationship High values of X tend to occur with high values of Y X and Y vary in the same direction
- 12. Negative (−) relationship High values of X tend to occur with low values of Y X and Y vary in the opposite direction 28 28 Positive relationship 29 29 Negative relationship 30 30 Positive or negative? Are these examples of relationships positive, or negative? Air po 31
- 13. 31 Unit of analysis Unit of analysis The objects or things described by the variables in a model Same theory may use different unit of analysis A good theory should – more often than not – explain patterns across many different units In longitudinal research, the unit of analysis includes the time period Days, months, quarters, years 32 32 Unit of analysis (income) 33 33 Unit of analysis Broken Windows Theory 34
- 14. 34 Logic models Also referred to as Program theories Outcome-sequence charts Theories of change Graphical models showing how a program produces desired outcomes 35 35 Logic models The simple bivariate model A direct effect from the IV on the DV 36 36 Logic models The simple causal model A direct effect from the IV on the DV Usually we will have variable names and give an expected
- 15. direction on the arrow - 37 37 Logic models The simple causal model A direct effect from the IV on the DV Usually we will have variable names and give an expected direction on the arrow Models are often bivariate, but reality is multivariate 38 38 Adding intervening variables Intervening variables Variables that intervene between the independent variables and dependent variables Also known as mediators in some disciplines, or intermediate outcomes in program evaluation They help articulate the causal process(es) – sometimes termed
- 16. causal chains – through which X produces Y 39 39 Adding intervening variables 40 40 Adding intervening variables – An example 41 41 Adding intervening variables – An example 42 42 Small Class Sizes + One-on-one Attention +
- 17. Higher Test Scores Problems with poorly thought out path models Stop and think considerations while developing Is there reverse causation? Is there spuriousness? More on this in week 4 43 43 You can have more than one intervening variable 44 44 Why we use logic models Helps identify previously unrecognized variables to track as performance indicators Helps in planning the design of a program evaluation Suggests logical weak links a program
- 18. 45 45 A DIY guide to logic models Start with a single outcome or Y variable Add a single X variable representing the program Put the program (X) on the left and the outcome (Y) on the right Add intervening variables between X and Y Distinguish causal “chains” from separate “pathways” Look for links that need explanation—consider additional intervening variables Give nondirectional names to variables and add “+” or “–” signs to the relationships (arrows) Make sure there is not too much, or too little, detail for your audience 46 46 Let’s practice
- 19. 47 47 Logic models in program implementation Often logic models are used to represent implementation of a program and include Inputs Financial, human, and material resources Activities Training, counseling, marketing, and other tasks Outputs The immediate products of activities (people trained, vaccinations given, etc.) Outcomes The results, including short-term, intermediate, and long-term outcomes 48 48 49 Additional issues in theory building
- 20. Moderator A variable that influence (strengthens or weakens) the relationship between two other variables 50 50 Additional issues in theory building Aggregation problem and ecological fallacy Relationships that hold at one unit of analysis may not hold at more aggregated levels Key takeaway – often our unit of analysis matters Think carefully 51 51 Additional issues in theory building Hierarchical (multilevel) models and contextual variables It’s weird the book places this here. Revisit this after we have covered regression for this to make more sense. Key takeaways: Many times we will have questions about society where
- 21. observations exist within groups: Students within classrooms Patients within wards Citizens within counties 52 52 Additional issues in theory building Variables that influence DVs that care about may be influenced by either individual-level or group-level characteristics Example: A student's success in the classroom may be a function of both their socio-economic background (individual- level variable) and the experience of the teacher (group-level variable) The group-level variables affect all students in the classroom We have to use special modeling strategies to accurately capture these affects in the real world 53 53 Additional issues in theory building Theoretical research
- 22. Theoretical research uses existing facts to gain insight, make valuable predictions and recommendations. 54 54 How to find and focus research questions Research question The question that motivated the researcher to do the study Applied research questions Arise from the practical concerns of policymaker s and practitioners A good research question . . . should be answerable may be descriptive or causal should be positive, not normative 55 55 Logic Model Assignment (25 points – 10 for memo & 15 for logic model) Consider a policy or social program that actually exists, that you would like to propose or that someone else has proposed.
- 23. Choose an area that interests you and that you know something about. Prepare a description of the theory of the mechanism of how your program works to affect the outcome. If you are looking for a program – I recommend reading through this report from UCLA on the impact of Washington DC’s voucher program. Write this up as memo (~ 400 words) to a boss or collaborator who is working with you to develop the program. This is not someone you need to convince about the importance of the outcomes or the program. Make sure that you including the following: (1) What is (are) the outcome(s) (dependent variable(s)) the program is designed to affect? If there are many outcomes, restrict your analysis to one outcome or two closely related outcomes. (For example, your program program’s goal might be to raise high school graduation rates in urban areas and so the outcome is graduation rate.) Make sure that you state the outcome(s) explicitly. (2) Describe your program—what it is. This should be as explicit as possible, not vague generalities. This section should be brief: a half double-spaced page at the most. Do not include marketing or promotion: your reader does not need to be convinced of the importance of the project. Write an objective and concrete statement of what the program literally does but do include implementation details. (3) Using a path diagram and a narrative description, describe your theory of how the program is supposed to work. Both the path diagram and the narrative description should make clear the mechanism(s) through which the program will affect the outcome. So, if a link is not obvious, break it down into the
- 24. steps along the way, illustrating the intervening variables. This section should illustrate to your readers why they should believe that the program will work—will affect the outcome(s). It should also make clear what the weak linkages are. This part (3) is the main focus of the assignment. Notes and advice: · The circles represent variables and the arrows represent causal effects. Make sure that you understand clearly the unit of analysis in your theory—the individuals to whom the variable applies: For example, is the program working on students, on schools, on cities? · There can be many mechanisms through which a program works. If so, pick only a couple and just note that there are other mechanisms. These should be more detailed than the logic models you see in many grant proposals and papers. Each link should be spelled out and made believable. · Draw your logic model by hand. Using software takes unnecessary time and (much more importantly) makes it hard to insert extra variables and arrows or to change what is there. If you absolutely must draw this in software, do not do that until the last possible point when you have already finalized your logic model by hand. To submit the logic model by email scan just the diagram part. · Do not include introductions, motivations, background, marketing and so on. · Do not include inputs, resources, or (detailed) activities. This logic model is not an implementation-oriented one: It focuses on mechanism. Implementation is done more effectively after
- 25. you understand clearly the mechanism. So, this logic model should not look like Figure 2.9. It should look more like Figure 2.8 or the top of p. 45 but with more details (e.g., more branches, more intervening variables). · Do use the tips on pp. 43-45. These tips were developed based on commonly made errors! · Check that each separate causal link makes sense isolated. Check that you are not missing causal links between variables on the page. Rubric for Logic Model Assignment Issue A level work B level work C level work F level work Independent variable (program) Clearly defined indep var in narrative and path diagram All effects on outcome (except contextual vars) lead ultimately from indep var Clearly defined indep var in narrative and path diagram Some effects on outcome lead ultimately from indep var Some definition of indep var in either narrative or path diagram No clearly defined indep var Dependent variable (outcome) Clearly defined Dep var in narrative and path diagram No other unspecified outcomes are de facto outcomes
- 26. Clearly defined dep var in narrative and path diagram Almost no other unspecified outcomes are de facto outcomes Some definition of dep var in either narrative or path diagram Other unspecified outcomes are de facto outcomes No clearly defined dep var Intervening variables Clearly defined Interv variables in narrative and path diagram The bulk of preceding variables are logical causes and following variables are logical consequences Does not confuse process and mechanism Mostly clearly defined Interv variables in narrative and path diagram Most of preceding variables are logical causes and following variables are logical consequences Does not confuse process and mechanism Some definition of interv variables in either narrative of path diagram Some of preceding variables are logical causes, and some following variables are logical consequences Confuses process and mechanism Intervening variables are not variables Preceding variables are not logical causes and following variables are not logical consequences
- 27. Completely substitutes process for mechanism Issue A level work B level work C level work F level work Mechanics of Assignment Narrative of mechanism is clear, concise and avoids marketing in favor of program description. Clearly describes variables and mechanisms. The bulk of path diagram has clearly drawn variables, explicit relationship arrows, and includes signs to show positive or inverse causal effects Narrative portion is clear, concise and contains little marketing in favor of program description. Describes variables and mechanisms, Most of the path diagram has clearly drawn variables, explicit relationship arrows, and includes signs to show positive or inverse causal effects Narrative portion is confusing and/or contains mostly program marketing language. Path diagram is missing some variables and/or some arrows and signs Narrative does not explain the program or variables. Path diagram missing many arrows and signs. Writing quality
- 28. Writing is very clear Arguments are cogent &persuasive Organization is sensible & clear Language is correct and concise No repetition Writing is fairly clear Arguments are fairly cogent & persuasive Organization is mostly sensible & clear Language is mostly correct Some unnecessary repetition Writing is unclear Arguments are not cogent & persuasive Poor organization Language has mistakes Much unnecessary repetition Writing is unclear No arguments are made
- 29. No organization Language has many mistakes Much repetition