Mixed Effects Models - Introduction
- 1. Using Mixed Effects Models in Psychology Scott Fraundorf sfraundo@pitt.edu Zoom office hours: Wed 11-12, or by appointment
- 2. Mixed Effects Models Intro ! Course goals & requirements ! Flex @ Pitt considerations ! Motivation for mixed effects models - Multiple random effects - Nested random effects - Crossed random effects - Categorical data - Continuous predictors ! Big picture view of mixed effects models ! Terminology ! R
- 3. Course Goals ! We will: - Understand form of mixed effects models - Apply mixed effects models to common designs in psychology and related fields (e.g., factorial experiments, educational interventions, longitudinal studies) - Fit mixed effects models in R using lme4 - Diagnose and address common issues in using mixed effects models ! We won’t: - Cover algorithms used by software to compute mixed effects models
- 4. Course Requirements ! We’ll be fitting models in R ! Free & runs on basically any computer ! Next week, will cover basics of using R
- 5. Mixed Effects Models Intro ! Course goals & requirements ! Flex @ Pitt considerations ! Motivation for mixed effects models - Multiple random effects - Nested random effects - Crossed random effects - Categorical data - Continuous predictors ! Big picture view of mixed effects models ! Terminology ! R
- 6. Why Mixed Effects Models?
- 7. Mixed Effects Models Intro ! Course goals & requirements ! Flex @ Pitt considerations ! Motivation for mixed effects models - Multiple random effects - Nested random effects - Crossed random effects - Categorical data - Continuous predictors ! Big picture view of mixed effects models ! Terminology ! R
- 8. • Inferential statistics you may be familiar with: • ANOVA • Regression • Correlation • All of these methods involve random sampling out of a larger population • To which we hope to generalize Problem 1: Multiple Random Effects Subject 1 Subject 2 Subject 3
- 9. • Inferential statistics you may be familiar with: • ANOVA • Regression • Correlation • Standard assumption: All observations are independent • Subject 1’s score doesn’t tell us anything about Subject 2’s Problem 1: Multiple Random Effects Subject 1 Subject 2 Subject 3
- 10. • Independence assumption is fair if we randomly sample 1 person at a time • e.g., you recruit 40 undergrads from the Psychology Subject Pool • But maybe this isn’t all we should be doing… (Henrich et al., 2010, Nature) Problem 1: Multiple Random Effects
- 11. • Important assumption! • Does testing benefit learning of biology? Problem 1: Multiple Random Effects Section A n=50 Practice quizzes Section C n=50 Practice quizzes Section B n=50 Restudy Section D n=50 Restudy
- 12. • Impressive if 100 separate people who did practice tests learned better than 100 people who reread the textbook • Not so impressive when we realize these aren’t fully independent • Need to account for differences in instructor, time of day • Problem 1: Multiple Random Effects
- 13. Mixed Effects Models Intro ! Course goals & requirements ! Flex @ Pitt considerations ! Motivation for mixed effects models - Multiple random effects - Nested random effects - Crossed random effects - Categorical data - Continuous predictors ! Big picture view of mixed effects models ! Terminology ! R
- 14. • But many sensible, informative research designs involve more complex sampling procedures • Example: Sampling multiple children from the same family • Kids from the same family will be more similar • a/k/a clustering Child 2 FAMILY 1 FAMILY 2 Child 3 Child 1 Problem 1A: Nested Random Effects
- 15. • But many sensible, informative research designs involve more complex sampling procedures • Or: Two people in a relationship • Or any other small group (work teams, experimentally formed groups, etc.) COUPLE 1 Problem 1A: Nested Random Effects
- 16. • But many sensible, informative research designs involve more complex sampling procedures • Or: A longitudinal study • Each person is sampled multiple times • Some characteristics don’t change over time—or don’t change much Problem 1A: Nested Random Effects
- 17. • But many sensible, informative research designs involve more complex sampling procedures • Or: Kids in classrooms in schools • Kids from the same school will be more similar • Kids in same classroom will be even more similar! Problem 1A: Nested Random Effects Student 2 CLASS- ROOM 1 Student 3 Student 1 CLASS- ROOM 2 SCHOOL 1
- 18. • One way to describe what’s going on here is that there are several levels of sampling, each nested inside each other • Each level is what we’ll call a random effect (a thing we sampled) Problem 1A: Nested Random Effects SAMPLED SCHOOLS SAMPLED CLASSROOMS in those schools SAMPLED STUDENTS in those classrooms Student 2 CLASS- ROOM 1 Student 3 Student 1 CLASS- ROOM 2 SCHOOL 1
- 19. • Two challenges: • Statistically, we need to take account for this non- independence (similarity) • Even a small amount of non-independence can lead to spurious findings (Quené & van den Bergh, 2008) • We might want to characterize differences at each level! • Are classroom differences or school differences bigger? Problem 1A: Nested Random Effects Student 2 CLASS- ROOM 1 Student 3 Student 1 CLASS- ROOM 2 SCHOOL 1
- 20. Mixed Effects Models Intro ! Course goals & requirements ! Flex @ Pitt considerations ! Motivation for mixed effects models - Multiple random effects - Nested random effects - Crossed random effects - Categorical data - Continuous predictors ! Big picture view of mixed effects models ! Terminology ! R
- 21. Problem 1B: Crossed Random Effects • A closely related problem shows up in many experimental studies • Experimental / research materials are often sampled out of population of possible items • Words or sentences • Educational materials • Hypothetical scenarios • Survey items • Faces
- 22. Problem 1B: Crossed Random Effects • We might ask: • Do differences in stimuli used account for group / condition differences? • Study word processing with a lexical decision task EAGLE PLERN
- 23. Problem 1B: Crossed Random Effects • We might ask: • Do differences in stimuli used account for group / condition differences? • e.g., Maybe easier vocab words used in one condition Unprimed Primed EAGLE PENGUIN EAGLE MONKEY
- 24. Problem 1B: Crossed Random Effects • We might ask: • Do differences in stimuli used account for group / condition differences? • Do our results generalize to the population of all relevant items? • All vocab words • All fictional resumes • All questionnaire items that measure extraversion • All faces
- 25. Problem 1B: Crossed Random Effects • Again, we are sampling two things—subjects and items • Arrangement is slightly different because each subject gets each item • Crossed random effects • Still, problem is that we have multiple random effects (things being sampled) Subject 1 Subject 2 EAGLE MONKEY
- 26. • Robustness across stimuli has been a major concern in psycholinguistics for a long time Problem 1B: Crossed Random Effects
- 27. • Robustness across stimuli has been a major concern in psycholinguistics for a long time • If you are doing research related to language processing, you’ll be expected to address this • (But stats classes don’t always teach you how) • Now growing interest in other fields, too Problem 1B: Crossed Random Effects
- 28. • Robustness across stimuli has been a major concern in psycholinguistics for a long time • If you are doing research related to language processing, you’ll be expected to address this • (But stats classes don’t always teach you how) • Now growing interest in other fields, too • Be a statistical pioneer! • Test generalization across items • Characterize variability • Increase power (more likely to detect a significant effect) Problem 1B: Crossed Random Effects
- 29. F1(1,3) = 18.31, p < .05 F2(1,4) = 22.45, p < .01 Problem 1B: Crossed Random Effects ! OLD ANOVA solution: Do 2 analyses ! Subjects analysis: Compare each subject (averaging over all of the items) ! Does the effect generalize across subjects? ! Items analysis: Compare each item (averaging over all of the subjects) ! Does the effect generalize across items? SUBJECT ANALYSIS ITEM ANALYSIS
- 30. Problem 1B: Crossed Random Effects ! OLD ANOVA solution: Do 2 analyses ! Subjects analysis ! Items analysis ! Problem: We now have 2 different sets of results. Might conflict! ! Possible to combine them with min F’, but not widely used F1(1,3) = 18.31, p < .05 F2(1,4) = 22.45, p < .01 SUBJECT ANALYSIS ITEM ANALYSIS
- 31. Mixed Effects Models Intro ! Course goals & requirements ! Flex @ Pitt considerations ! Motivation for mixed effects models - Multiple random effects - Nested random effects - Crossed random effects - Categorical data - Continuous predictors ! Big picture view of mixed effects models ! Terminology ! R
- 32. Problem 2: Categorical Data ! Basic linear model assumes our response (dependent) measure is continuous ! But, we often want to look at categorical data RT: 833 ms Does student graduate high school or not? Item recalled or not? What predicts DSM diagnosis of ASD? Income: $30,000
- 33. Problem One: Categorical Data ! Traditional solution: Analyze percentages ! Maybe with some transformation (e.g., arcsine, logit) ! Violates assumptions of linear model - Among other issues: Linear model assumes normal DV, which has infinite tails - But percentages are clearly bounded - Model could predict impossible values like 110% Problem 2: Categorical Data But 0% ≤ percentages ≤ 100% ∞ - ∞ 100% 0%
- 34. Problem One: Categorical Data ! Traditional solution: Analyze percentages ! Violates assumptions of ANOVA - Among other issues: ANOVA assumes normal distribution, which has infinite tails - But proportions are clearly - Model could predict impossible values like 110% Problem 2: Categorical Data ∞ - ∞ But 0% ≤ percentages ≤ 100% 100% 0%
- 35. Problem One: Categorical Data ! Traditional solution: Analyze percentages ! Maybe with some transformation (e.g., arcsine, logit) ! Violates assumptions of linear model ! Can lead to: ! Spurious effects (false positives / Type I error) ! Missing real effects (false negatives / Type II error) Problem 2: Categorical Data
- 36. Mixed Effects Models Intro ! Course goals & requirements ! Flex @ Pitt considerations ! Motivation for mixed effects models - Multiple random effects - Nested random effects - Crossed random effects - Categorical data - Continuous predictors ! Big picture view of mixed effects models ! Terminology ! R
- 37. ! Many interesting independent variables vary continuously ! e.g., Word frequency Problem 3: Continuous Predictors pitohui eagle penguin
- 38. ! Many interesting independent variables vary continuously ! Or: Second language proficiency or reading skill ! ANOVAs require division into categories ! e.g., median split Problem 3: Continuous Predictors
- 39. ! Many interesting independent variables vary continuously ! Or: Second language proficiency or reading skill ! ANOVAs require division into categories ! e.g., median split ! Or: extreme groups design Problem 3: Continuous Predictors
- 40. ! Many interesting independent variables vary continuously ! Or: Second language proficiency or reading skill ! ANOVAs require division into categories ! Problem: Can only ask “is there a difference?”, not form of relationship ! Loss of statistical power (Cohen, 1983) Problem 3: Continuous Predictors
- 42. Mixed Effects Models Intro ! Course goals & requirements ! Flex @ Pitt considerations ! Motivation for mixed effects models - Multiple random effects - Nested random effects - Crossed random effects - Categorical data - Continuous predictors ! Big picture view of mixed effects models ! Terminology ! R Power of subjects analysis! Power of items analysis! Captain Mixed Effects to the rescue!
- 43. ! Biggest contribution of mixed-effects models is to incorporate multiple random effects into the same analysis Mixed Effects Models to the Rescue! How does the effect of parental stress on screen time generalize across children and families? How does the effect of aphasia on sentence processing generalize across subjects and sentences?
- 44. Mixed Models to the Rescue! ! Extension of linear model (regression) = School + + GPA Motiv- ational intervention Subject Outcome Problem 1A solved! 3.07
- 45. ! For many experimental designs, this means a change in what gets analyzed ! ANOVA: Unit of analysis is cell mean ! Mixed effects models: Unit of analysis is individual trial! Mixed Effects Models to the Rescue!
- 46. Mixed Models to the Rescue! ! Extension of linear model (regression) ! Look at individual trials/observations (not means) = Item + + RT Prime? Subject Lexical decision RT Problem 1B solved!
- 47. Mixed Models to the Rescue! ! In a linear model (regression), easy to include independent variables that are continuous = Item + + RT Lexical Freq. Subject Lexical decision RT Problem 3 solved! 2.32
- 48. Mixed Models to the Rescue! ! Link functions allow us to relate model to DV that isn’t normally distributed = Item + + Lexical Freq. Subject Odds of correct response on this trial (yes or no?) Accuracy Problem 2 solved!
- 49. Mixed Models to the Rescue! ! Link functions allow us to relate model to DV that isn’t normally distributed = Odds of graduating college Yes or No? Problem 2 solved! School + + Motiv- ational intervention Subject
- 50. Mixed Effects Models Intro ! Course goals & requirements ! Flex @ Pitt considerations ! Motivation for mixed effects models - Multiple random effects - Nested random effects - Crossed random effects - Categorical data - Continuous predictors ! Big picture view of mixed effects models ! Terminology ! R
- 51. A Terminological Note… ! “Mixed effects models” is a more general term ! Any model that includes sampled subjects, classrooms, or items (a “random effect”) plus predictor variables of interest (“fixed effects”) ! “Mixing” these two types of effects ! One specific type that we’ll be discussing are “hierarchical linear models”
- 52. Mixed Effects Models Intro ! Course goals & requirements ! Flex @ Pitt considerations ! Motivation for mixed effects models - Multiple random effects - Nested random effects - Crossed random effects - Categorical data - Continuous predictors ! Big picture view of mixed effects models ! Terminology ! R
- 53. R ! How do we run mixed effects models? ! Multiple software packages could be used to fit the same conceptual model ! Most popular solution: R with lme4
- 54. R Pros R Cons ! Free & open-source! ! Runs on any computer ! Lots of add-ons— can do just about any type of model ! Widespread use ! Makes analyses clear & more reproducible ! Documentation / help files not the best ! Other online resources ! Requires some programming, not just menus
- 55. Two Ways to Use R ! Regular R < www.r-project.org > ! RStudio < www.rstudio.com > ! Different interface ! Some additional windows/tools to help you keep track of what you’re doing ! Same commands, same results ! Also available for just about any platform ! Recommended (but not required) ! Requires you to download regular R first ! Homework for next class: Download them!
- 56. Mixed Effects Models Intro ! Course goals & requirements ! Flex @ Pitt considerations ! Motivation for mixed effects models - Multiple random effects - Nested random effects - Crossed random effects - Categorical data - Continuous predictors ! Big picture view of mixed effects models ! Terminology ! R
- 57. Wrap-Up ! Mixed effects models solve three common problems with linear model ! Multiple random effects (subjects, items, classrooms, schools) ! Categorical outcomes ! Continuous predictors ! For next week: Download R! ! Next class, we’ll get started using R ! Canvas survey about your research background & Flex@Pitt implementation