SlideShare a Scribd company logo

Mixed Effects Models - Effect Size

S
Scott Fraundorf

This document discusses effect size and how to establish causality in longitudinal data analysis. It begins by reviewing autocorrelation and how to account for it in mixed effects models. It then discusses cross-lagged models, where including lags of variables can help determine whether A causes B or B causes A. Establishing causality further requires showing the relationship is directional by testing models in both directions. The document provides an example using a diary study to show that emotional support attempts cause increased warmth, not vice versa. It also discusses how effect size compares to statistical significance and how to interpret the size of different effects.

Education
1 of 60
Downloaded 16 times
1
2
Most read
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
Follow-Up on Autocorrelation • Autocorrelation: An empirical phenomenon in longitudinal data where the value of a variable at time t correlates with its value at time t+1 • Could test this with a Pearson correlation
Correlation of time t with time t+1 (lag 1) Correlation of time t with time t+2 (lag 2) Correlation of time t with time t+3 (lag 3) • acf.fnc() plot shows pairwise correlations Correlation of time t with itself (always 1)
Follow-Up on Autocorrelation • Autocorrelation: An empirical phenomenon in longitudinal data where the value of a variable at time t correlates with its value at time t+1 • We can incorporate into a mixed-effects model • Unlike a pairwise correlation, accounts for nested structure and other variables • model.auto <- lmer(WarmthToday ~ 1 + Day + WarmthYesterday + (1 + Day + WarmthYesterday|Couple), data=relationship)
Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
Cross-Lagged Models • Our diary study also includes records of emotional support attempts from the partner • Do these cause increased warmth towards the partner?
Cross-Lagged Models • Our diary study also includes records of emotional support attempts from the partner • Do these cause increased warmth towards the partner? Wait just a darn minute! Correlation does not imply causation! You didn’t experimentally manipulate these support attempts, so you don’t know which caused which! I’VE FINALLY GOT YOU, FRAUNDORF!!
Cross-Lagged Models • Problem: Relation between support attempts & warmth is ambiguous • What could cause this? • Support attempts could increase warmth towards partner • Warmth towards partner could motivate support attempts • A third variable could explain both Perceived warmth Support attempt TIME t ? Relationship commitment
Cross-Lagged Models • Problem: Relation between support attempts & warmth is ambiguous • But: Causes precede effects in time • Support attempt on a previous day should influence warmth now Perceived warmth Support attempt TIME t TIME t-1 Support attempt Duckworth, Tsukayama, & May, 2010
Cross-Lagged Models • Use lags.fnc() to create a SupportYesterday variable • relationship %>% mutate(SupportYesterday= lags.fnc(relationship, time='Day', group='Couple', depvar='PartnerSupport', lag=1)) -> relationship • Then, use that in a model: • model.lagged <- lmer(WarmthToday ~ 1 + Day + SupportYesterday + (1|Couple), data=relationship) Perceived warmth Support attempt TIME t TIME t-1 Support attempt Duckworth, Tsukayama, & May, 2010
Cross-Lagged Models • model.lagged <- lmer(WarmthToday ~ 1 + Day + SupportYesterday + (1|Couple), data=relationship) Perceived warmth Support attempt TIME t TIME t-1 Support attempt Duckworth, Tsukayama, & May, 2010
Cross-Lagged Models • Warmth at t can’t be the cause of support at t-1 • Helps clarify which is the cause and which is the effect Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Duckworth, Tsukayama, & May, 2010
Cross-Lagged Models • Warmth at t can’t be the cause of support at t-1 • But, warmth at time t-1 could still function as a 3rd variable • Causes support attempts at time t-1 • Leads to greater warmth at time t (autocorrelation) Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth Duckworth, Tsukayama, & May, 2010 X
Cross-Lagged Models • To rule this out, we need to include the autocorrelative effect of perceived warmth (our DV) • model.lagged2 <- lmer(WarmthToday ~ 1 + Day + SupportYesterday + WarmthYesterday + (1|Couple), data=relationship) Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth Duckworth, Tsukayama, & May, 2010
Cross-Lagged Models • model.lagged2 <- lmer(WarmthToday ~ 1 + Day + SupportYesterday + WarmthYesterday + (1|Couple), data=relationship) Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth Duckworth, Tsukayama, & May, 2010
Cross-Lagged Models • Now, we are seeing a time-lagged effect of support attempts over and above what can predicted by previous warmth • No way to explain this in a model where the causation only works in reverse • Strong evidence against the reverse direct of causation Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X Duckworth, Tsukayama, & May, 2010
Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
Establishing Causality • Between-person variation in support attempts predicts within-couple change in warmth Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X Duckworth, Tsukayama, & May, 2010
Establishing Causality • But, there’s still the possibility of a third variable that really drives this between-person difference • e.g., relationship commitment could explain variation in previous support attempts and increase in warmth Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X Relationship commitment Duckworth, Tsukayama, & May, 2010
Establishing Causality • If relationship is driven by an underlying 3rd variable, then warmth & support don’t have a cause/effect relation • Should see the same relation regardless of their order Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X Relationship commitment Duckworth, Tsukayama, & May, 2010
Establishing Causality • To establish causality, show that the direction of the relationship matters • Run the inverse model where support attempts are the DV and previous warmth is the predictor • model.lagged3 <- glmer(PartnerSupport ~ 1 + Day + WarmthYesterday + SupportYesterday + (1|Couple), data=relationship, family=binomial) Perceived warmth Support attempt TIME t TIME t-1 Support attempt Perceived warmth X X
Establishing Causality • model.lagged3 <- glmer(PartnerSupport ~ 1 + Day + WarmthYesterday + SupportYesterday + (1|Couple), data=relationship, family=binomial) • No significant effects • Earlier support attempts predict later warmth (model.lagged2) • But earlier warmth doesn’t predict later support attempts (model.lagged3) • Evidence for directionality of effect
Establishing Causality • This kind of evidence is called Granger causality • Still one kind of 3rd variable not ruled out: One with immediate effect on support attempts & a delayed effect on warmth • However, much less likely • So, not quite as good as randomized experiment • But, effective when experimental control not possible (e.g., economics, neuroscience) Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X ???
Establishing Causality • This kind of evidence is called Granger causality • Still one kind of 3rd variable not ruled out: One with immediate effect on support attempts & a delayed effect on warmth • However, much less likely • So, not quite as good as randomized experiment • But, effective when experimental control not possible (e.g., economics, neuroscience) Adapted from Kaminski et al., 2011
Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
Effect Size • Revisiting lifexpectancy.csv, let’s run a model predicting Lifespan from fixed effects of YrsEducation and IncomeThousands, and a random intercept of Family • Two variables related to socioeconomic status • Does each matter, when controlling for the other? Which is the most important?
Effect Size • Revisiting lifexpectancy.csv, let’s run a model predicting Lifespan from fixed effects of YrsEducation and IncomeThousands, and a random intercept of Family • Two variables related to socioeconomic status • Which significantly predict the number of years that people live? • model.life <- lmer(Lifespan ~ 1 + YrsEducation + IncomeThousands + (1|Family), data=lifeexpectancy) They both do! Which is bigger?
Effect Size • Remember that t statistics and p-values tell us about whether there’s an effect in the population • Is the effect statistically reliable? • A separate question is how big the effect is • Effect size
Bigfoot: Little evidence he exists, but he’d be large if he did exist Pygmy hippo: We know it exists and it’s small LARGE EFFECT SIZE, LOW RELIABILITY [-.20, 1.80] SMALL EFFECT SIZE, HIGH RELIABILITY [.15, .35]
• Is bacon really this bad for you?? October 26, 2015
• Is bacon really this bad for you?? • True that we have as much evidence that bacon causes cancer as smoking causes cancer! • Same level of statistical reliability
• Is bacon really this bad for you?? • True that we have as much evidence that bacon causes cancer as smoking causes cancer! • Same level of statistical reliability • But, effect size is much smaller for bacon
Effect Size • Our model results tell us both Parameter estimate tells us about effect size t statistic and p-value tell us about statistical reliability
Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
Effect Size: Parameter Estimate • Simplest measure: Parameter estimates • Effect of 1-unit change in predictor on outcome variable • “Each additional $1,000 of annual income predicts another 0.25 years of life” • “Each minute of exercise increases life expectancy by about 7 minutes.” (Moore et al., 2012, PLOS ONE) • “People with a college diploma earn around $24,000 more per year.” (Bureau of Labor Statistics, 2018) • Concrete! Good for “real-world” outcomes
Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
Effect Size: Standardization • Which is the bigger effect? • 1 year of education = 0.57 years of life expectancy • $1,000 of annual income = 0.25 years of life expectancy • Problem: These are measured in different, non-comparable units • Years of education vs. (thousands of) US dollars
Effect Size: Standardization • Which is the bigger effect? • 1 year of education = 0.57 years of life expectancy • $1,000 of annual income = 0.25 years of life expectancy • Problem: These are measured in different, non-comparable units • Years of education vs. (thousands of) US dollars • Convert to z-scores: # of standard deviations from the mean • This scale applies to anything! • Standardized scores
Effect Size: Standardization • scale() puts things in terms of z-scores • New z-scored version of our predictors: • lifeexpectancy %>% mutate( YrsEducation.z = scale(YrsEducation)[,1], IncomeThousands.z = scale(IncomeThousands)[,1]) -> lifeexpectancy • # of standard deviations above/below mean income
Effect Size: Standardization • scale() puts things in terms of z-scores • New z-scored version of our predictors: • lifeexpectancy %>% mutate( YrsEducation.z = scale(YrsEducation)[,1], IncomeThousands.z = scale(IncomeThousands)[,1]) -> lifeexpectancy • # of standard deviations above/below mean income • Then use these in a new model • model.life <- lmer(Lifespan ~ 1 + YrsEducation.z + IncomeThousands.z + (1|Subject), data=lifeexpectancy)
Effect Size: Standardization • New results: • 1 SD increase in education = +2.1 years of life expectancy • 1 SD increase in income = +2.4 years of life expectancy • Income effect is bigger in this dataset
Effect Size: Standardization • Old results: • New results: No change in statistical reliability Effect size is now estimated differently
Effect Size: Standardization • Standardized effects make effect sizes more reliant on our data • Effect of 1 SD relative to the mean depends on what the M and SD are! • e.g., Effect of cigarette smoking on life • Smoking rates vary a lot from country to country! • Might get different standardized effects even if unstandardized is the same
Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
Overall Variance Explained • How well can we explain this DV? • In the linear model context: • R2= • But in mixed-effect context: Which variance are we talking about? Model-explained variance Model-explained variance + Error variance Level-1 error variance Variance from level-2 clustering Jaeger, Edwards, Das, & Sen, 2017; Rights & Sterba, 2019
• One R2 we can compute in mixed-effects context: • R2 c= • Obtain with the squared correlation between model-predicted and observed values • cor(fitted(model.life), lifeexpectancy$Lifespan)^2 • Here, 46%! Variance explained by fixed & random effects Conditional R2 All variance Jaeger, Edwards, Das, & Sen, 2017; Rights & Sterba, 2019
70 75 80 85 50 60 70 80 90 100 PREDICTED lifespan ACTUAL lifespan • One R2 we can compute in mixed-effects context: • R2 c= Variance explained by fixed & random effects Conditional R2 All variance
• One R2 we can compute in mixed-effects context: • R2 c= • But, are random effects really “explaining” the variance? Variance explained by fixed & random effects Conditional R2 All variance Some families have longer average lifespans Some subjects have faster RTs than others
• One R2 we can compute in mixed-effects context: • R2 c= • Summary: • Conditional R2 counts both fixed and random effects as explained variance • Preserves R2 as square of the correlation between observed & predicted data • Evaluates model’s ability to make good predictions • But, may overstate scientific/theoretical explanatory power Variance explained by fixed & random effects Conditional R2 All variance
Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
• Another R2 that may be more helpful: • R2 β*= • library(r2glmm) • r2beta(model.life) Variance explained by fixed effects only Marginal R2 All variance Variance explained by all fixed effects combined (14%) Partial R2 for each fixed effect Jaeger, Edwards, Das, & Sen, 2017
• Another R2 that may be more helpful: • R2 β*= • library(r2glmm) • r2beta(model.life) • r2dt(model1, model2) to test R2 difference between two models Variance explained by fixed effects only Marginal R2 All variance Jaeger, Edwards, Das, & Sen, 2017
• plot(r2beta(model.life)) Marginal R2
• Another R2 that may be more helpful: • R2 β*= • Summary: • Marginal R2 counts only fixed effects as explained variance • Evaluates model’s scientific/theoretical explanatory ability • Probably more useful in most purposes Variance explained by fixed effects only Marginal R2 All variance
Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
• Some conventional interpretations of R2 and partial R2: • But, take these with several grains of salt • Cohen (1988) just made them up • Unclear why we care about variance explained (R2) rather than standard deviations (r), in original units • Even small effects can accumulate over time (Funder & Ozer, 2019) Cohen (1988) “Small” .01 “Medium” .06 “Large” .15 Interpreting Effect Size
• Some conventional interpretations of R2 and partial R2: Cohen (1988) Funder & Ozer (2019) “Small” .01 “Medium” .06 “Large” .15 Interpreting Effect Size .001 .04 .09 .0025 “Very Small” .16 “Very Large”
• Consider in context of other effect sizes in this domain: • vs: • For interventions: Consider cost, difficulty of implementation, etc. • Aspirin’s effect in reducing heart attacks: r = .03, R2 < .01, but cheap! (Rosenthal, 1990) Our effect: .10 Other effect 1: .20 Other effect 2: .30 Our effect: .10 Other effect 1: .01 Other effect 2: .05 Effect Size: Interpretation
• For theoretically guided research, compare to predictions of competing theories • The lag effect in memory: • Is this about intervening items or time? Study RACCOON 5 sec. Study WITCH 5 sec. Study VIKING 5 sec. Study RACCOON 5 sec. 1 sec 1 sec 1 sec 1 day Study RACCOON 5 sec. Study WITCH 5 sec. Study VIKING 5 sec. Study RACCOON 5 sec. 1 sec 1 sec 1 sec 1 day POOR recall of RACCOON GOOD recall of RACCOON Effect Size: Interpretation
Effect Size: Interpretation • Is lag effect about intervening items or time? • Intervening items hypothesis predicts A > B • Time hypothesis predicts B > A • Goal here is to use direction of the effect to adjudicate between competing hypotheses • Not whether the lag effect is “small” or “large” Study RACCOON 5 sec. Study WITCH 5 sec. Study VIKING 5 sec. Study RACCOON 5 sec. 1 sec 1 sec 1 sec 1 day TEST A: Study RACCOON 5 sec. Study WITCH 5 sec. Study RACCOON 5 sec. 10 sec 10 sec 1 day TEST B:

More Related Content

PDF
Mixed Effects Models - Fixed Effect Interactions
Scott Fraundorf
 
PDF
Mixed Effects Models - Introduction
Scott Fraundorf
 
PDF
Mixed Effects Models - Data Processing
Scott Fraundorf
 
PDF
Mixed Effects Models - Random Intercepts
Scott Fraundorf
 
PDF
Mixed Effects Models - Model Comparison
Scott Fraundorf
 
PDF
Mixed Effects Models - Random Slopes
Scott Fraundorf
 
PDF
Mixed Effects Models - Autocorrelation
Scott Fraundorf
 
PDF
Mixed Effects Models - Simple and Main Effects
Scott Fraundorf
 
Mixed Effects Models - Fixed Effect Interactions
Scott Fraundorf
 
Mixed Effects Models - Introduction
Scott Fraundorf
 
Mixed Effects Models - Data Processing
Scott Fraundorf
 
Mixed Effects Models - Random Intercepts
Scott Fraundorf
 
Mixed Effects Models - Model Comparison
Scott Fraundorf
 
Mixed Effects Models - Random Slopes
Scott Fraundorf
 
Mixed Effects Models - Autocorrelation
Scott Fraundorf
 
Mixed Effects Models - Simple and Main Effects
Scott Fraundorf
 

What's hot (20)

PDF
Mixed Effects Models - Descriptive Statistics
Scott Fraundorf
 
PDF
Mixed Effects Models - Fixed Effects
Scott Fraundorf
 
PDF
Mixed Effects Models - Growth Curve Analysis
Scott Fraundorf
 
PDF
Mixed Effects Models - Crossed Random Effects
Scott Fraundorf
 
PDF
Time series forecasting
Firas Kastantin
 
PDF
Hibernate ORM: Tips, Tricks, and Performance Techniques
Brett Meyer
 
PPTX
Reporting Phi Coefficient test in APA
Ken Plummer
 
PPTX
T test
ashishjaswal
 
PDF
Delta Lake: Optimizing Merge
Databricks
 
PDF
Pandas UDF and Python Type Hint in Apache Spark 3.0
Databricks
 
PDF
Logistic Regression Analysis
COSTARCH Analytical Consulting (P) Ltd.
 
PPTX
The Chi-Square Statistic: Tests for Goodness of Fit and Independence
jasondroesch
 
PPTX
Time series analysis
Utkarsh Sharma
 
PPTX
Introduction to Statistics
jasondroesch
 
PPTX
Comparing means
University of Jaffna
 
PDF
Apache Flink Internals: Stream & Batch Processing in One System – Apache Flin...
ucelebi
 
PPT
Key ideas, terms and concepts in SEM
University of Southampton
 
PPTX
BASIC RELIABILITY MATHEMATICS.pptx
MohdNajibAliMokhtar
 
PPTX
Repeated-Measures and Two-Factor Analysis of Variance
jasondroesch
 
PPTX
Introduction to Hypothesis Testing
jasondroesch
 
Mixed Effects Models - Descriptive Statistics
Scott Fraundorf
 
Mixed Effects Models - Fixed Effects
Scott Fraundorf
 
Mixed Effects Models - Growth Curve Analysis
Scott Fraundorf
 
Mixed Effects Models - Crossed Random Effects
Scott Fraundorf
 
Time series forecasting
Firas Kastantin
 
Hibernate ORM: Tips, Tricks, and Performance Techniques
Brett Meyer
 
Reporting Phi Coefficient test in APA
Ken Plummer
 
T test
ashishjaswal
 
Delta Lake: Optimizing Merge
Databricks
 
Pandas UDF and Python Type Hint in Apache Spark 3.0
Databricks
 
Logistic Regression Analysis
COSTARCH Analytical Consulting (P) Ltd.
 
The Chi-Square Statistic: Tests for Goodness of Fit and Independence
jasondroesch
 
Time series analysis
Utkarsh Sharma
 
Introduction to Statistics
jasondroesch
 
Comparing means
University of Jaffna
 
Apache Flink Internals: Stream & Batch Processing in One System – Apache Flin...
ucelebi
 
Key ideas, terms and concepts in SEM
University of Southampton
 
BASIC RELIABILITY MATHEMATICS.pptx
MohdNajibAliMokhtar
 
Repeated-Measures and Two-Factor Analysis of Variance
jasondroesch
 
Introduction to Hypothesis Testing
jasondroesch
 
Ad

Similar to Mixed Effects Models - Effect Size (20)

PDF
slides-correlations.pdf
FlorentBersani
 
PPT
Chapter 2 methods and statistics
Psych Soon
 
PPTX
Correlation causality
veesingh
 
PDF
Perspective of feature selection in bioinformatics
Gianluca Bontempi
 
PPTX
Ucc one sided coin actual
Brian Hughes
 
PPT
PSY 239 401 CHAPTER 3 SLIDES
kimappel
 
PPTX
COORELATION
SHALINIrajput7
 
DOCX
1. A small accounting firm pays each of its five clerks $35,000, t.docx
SONU61709
 
PDF
Logistic regression vs. logistic classifier. History of the confusion and the...
Adrian Olszewski
 
PPTX
Modeling Social Data, Lecture 11: Causality and Experiments, Part 1
jakehofman
 
PDF
Final presentation (yw5178)
ssuser2d9f321
 
PPT
surveys non experimental
ROBERTOENRIQUEGARCAA1
 
PPT
quantitative research
caseolaris
 
PPTX
2. The scientific method - Chap 2.pptx
SaleemAbbas42
 
PPTX
Introduction to simulating data to improve your research
Dorothy Bishop
 
PPTX
Lesson 3 Exploring the Regression Analysis.pptx
JeanEndino
 
PPTX
Marketing Experimentation - Part I
Minha Hwang
 
PPT
Causal comparative research
Mira K Desai
 
PPTX
Matching Methods and Natural Experiments - Examples of Causal Inference from ...
Ingmar Weber
 
PPT
Ch07 Experimental & Quasi-Experimental Designs
yxl007
 
slides-correlations.pdf
FlorentBersani
 
Chapter 2 methods and statistics
Psych Soon
 
Correlation causality
veesingh
 
Perspective of feature selection in bioinformatics
Gianluca Bontempi
 
Ucc one sided coin actual
Brian Hughes
 
PSY 239 401 CHAPTER 3 SLIDES
kimappel
 
COORELATION
SHALINIrajput7
 
1. A small accounting firm pays each of its five clerks $35,000, t.docx
SONU61709
 
Logistic regression vs. logistic classifier. History of the confusion and the...
Adrian Olszewski
 
Modeling Social Data, Lecture 11: Causality and Experiments, Part 1
jakehofman
 
Final presentation (yw5178)
ssuser2d9f321
 
surveys non experimental
ROBERTOENRIQUEGARCAA1
 
quantitative research
caseolaris
 
2. The scientific method - Chap 2.pptx
SaleemAbbas42
 
Introduction to simulating data to improve your research
Dorothy Bishop
 
Lesson 3 Exploring the Regression Analysis.pptx
JeanEndino
 
Marketing Experimentation - Part I
Minha Hwang
 
Causal comparative research
Mira K Desai
 
Matching Methods and Natural Experiments - Examples of Causal Inference from ...
Ingmar Weber
 
Ch07 Experimental & Quasi-Experimental Designs
yxl007
 
Ad

More from Scott Fraundorf (10)

PDF
Mixed Effects Models - Signal Detection Theory
Scott Fraundorf
 
PDF
Mixed Effects Models - Power
Scott Fraundorf
 
PDF
Mixed Effects Models - Missing Data
Scott Fraundorf
 
PDF
Mixed Effects Models - Empirical Logit
Scott Fraundorf
 
PDF
Mixed Effects Models - Logit Models
Scott Fraundorf
 
PDF
Mixed Effects Models - Orthogonal Contrasts
Scott Fraundorf
 
PDF
Mixed Effects Models - Post-Hoc Comparisons
Scott Fraundorf
 
PDF
Mixed Effects Models - Centering and Transformations
Scott Fraundorf
 
PDF
Mixed Effects Models - Level-2 Variables
Scott Fraundorf
 
PDF
Scott_Fraundorf_Resume
Scott Fraundorf
 
Mixed Effects Models - Signal Detection Theory
Scott Fraundorf
 
Mixed Effects Models - Power
Scott Fraundorf
 
Mixed Effects Models - Missing Data
Scott Fraundorf
 
Mixed Effects Models - Empirical Logit
Scott Fraundorf
 
Mixed Effects Models - Logit Models
Scott Fraundorf
 
Mixed Effects Models - Orthogonal Contrasts
Scott Fraundorf
 
Mixed Effects Models - Post-Hoc Comparisons
Scott Fraundorf
 
Mixed Effects Models - Centering and Transformations
Scott Fraundorf
 
Mixed Effects Models - Level-2 Variables
Scott Fraundorf
 
Scott_Fraundorf_Resume
Scott Fraundorf
 

Recently uploaded (20)

PPTX
Cell Types and Its function , kingdom of life
ClaireMangundayao1
 
PDF
Supply Chain Operations Speaking Notes -ICLT Program
labyankof
 
PPTX
1st Inaugural Professorial Lecture held on 19th February 2020 (Governance and...
kawisojames10
 
PDF
Saundersa Comprehensive Review for the NCLEX-RN Examination.pdf
sreejithcareers
 
PDF
Physiotherapy_for_Respiratory_and_Cardiac_Problems WEBBER.pdf
DrMONISHAphysio
 
PPTX
PPT- ENG7_QUARTER1_LESSON1_WEEK1. IMAGERY -DESCRIPTIONS pptx.pptx
KMDee
 
PPTX
master seminar digital applications in india
ananyaray35
 
PDF
Complications of Minimal Access Surgery at WLH
mohitsuren827
 
PDF
102 student loan defaulters named and shamed – Is someone you know on the list?
Kweku Zurek
 
PDF
Sports Quiz easy sports quiz sports quiz
nikunj2757
 
PDF
FourierSeries-QuestionsWithAnswers(Part-A).pdf
Raji Murugan
 
PDF
The Lost Whites of Pakistan by Jahanzaib Mughal.pdf
jahanzaibmughal6
 
PDF
Microbial disease of the cardiovascular and lymphatic systems
KeyaSharma
 
PDF
Abdominal Access Techniques with Prof. Dr. R K Mishra
mohitsuren827
 
PDF
Module 4: Burden of Disease Tutorial Slides S2 2025
Jonathan Hallett
 
PPTX
human mycosis Human fungal infections are called human mycosis..pptx
shilpa939953
 
PPTX
Pharmacology of Heart Failure /Pharmacotherapy of CHF
Rajshri Ghogare
 
PDF
3rd Neelam Sanjeevareddy Memorial Lecture.pdf
Academy of Grassroots Studies and Research of India (AGRASRI)
 
PDF
RMMM.pdf make it easy to upload and study
sajedul2
 
PPTX
Cell Structure & Organelles in detailed.
DHANASHREESIVAKUMAR
 
Cell Types and Its function , kingdom of life
ClaireMangundayao1
 
Supply Chain Operations Speaking Notes -ICLT Program
labyankof
 
1st Inaugural Professorial Lecture held on 19th February 2020 (Governance and...
kawisojames10
 
Saundersa Comprehensive Review for the NCLEX-RN Examination.pdf
sreejithcareers
 
Physiotherapy_for_Respiratory_and_Cardiac_Problems WEBBER.pdf
DrMONISHAphysio
 
PPT- ENG7_QUARTER1_LESSON1_WEEK1. IMAGERY -DESCRIPTIONS pptx.pptx
KMDee
 
master seminar digital applications in india
ananyaray35
 
Complications of Minimal Access Surgery at WLH
mohitsuren827
 
102 student loan defaulters named and shamed – Is someone you know on the list?
Kweku Zurek
 
Sports Quiz easy sports quiz sports quiz
nikunj2757
 
FourierSeries-QuestionsWithAnswers(Part-A).pdf
Raji Murugan
 
The Lost Whites of Pakistan by Jahanzaib Mughal.pdf
jahanzaibmughal6
 
Microbial disease of the cardiovascular and lymphatic systems
KeyaSharma
 
Abdominal Access Techniques with Prof. Dr. R K Mishra
mohitsuren827
 
Module 4: Burden of Disease Tutorial Slides S2 2025
Jonathan Hallett
 
human mycosis Human fungal infections are called human mycosis..pptx
shilpa939953
 
Pharmacology of Heart Failure /Pharmacotherapy of CHF
Rajshri Ghogare
 
3rd Neelam Sanjeevareddy Memorial Lecture.pdf
Academy of Grassroots Studies and Research of India (AGRASRI)
 
RMMM.pdf make it easy to upload and study
sajedul2
 
Cell Structure & Organelles in detailed.
DHANASHREESIVAKUMAR
 

Mixed Effects Models - Effect Size

  • 1. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
  • 2. Follow-Up on Autocorrelation • Autocorrelation: An empirical phenomenon in longitudinal data where the value of a variable at time t correlates with its value at time t+1 • Could test this with a Pearson correlation
  • 4. Follow-Up on Autocorrelation • Autocorrelation: An empirical phenomenon in longitudinal data where the value of a variable at time t correlates with its value at time t+1 • We can incorporate into a mixed-effects model • Unlike a pairwise correlation, accounts for nested structure and other variables • model.auto <- lmer(WarmthToday ~ 1 + Day + WarmthYesterday + (1 + Day + WarmthYesterday|Couple), data=relationship)
  • 5. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
  • 6. Cross-Lagged Models • Our diary study also includes records of emotional support attempts from the partner • Do these cause increased warmth towards the partner?
  • 7. Cross-Lagged Models • Our diary study also includes records of emotional support attempts from the partner • Do these cause increased warmth towards the partner? Wait just a darn minute! Correlation does not imply causation! You didn’t experimentally manipulate these support attempts, so you don’t know which caused which! I’VE FINALLY GOT YOU, FRAUNDORF!!
  • 8. Cross-Lagged Models • Problem: Relation between support attempts & warmth is ambiguous • What could cause this? • Support attempts could increase warmth towards partner • Warmth towards partner could motivate support attempts • A third variable could explain both Perceived warmth Support attempt TIME t ? Relationship commitment
  • 9. Cross-Lagged Models • Problem: Relation between support attempts & warmth is ambiguous • But: Causes precede effects in time • Support attempt on a previous day should influence warmth now Perceived warmth Support attempt TIME t TIME t-1 Support attempt Duckworth, Tsukayama, & May, 2010
  • 10. Cross-Lagged Models • Use lags.fnc() to create a SupportYesterday variable • relationship %>% mutate(SupportYesterday= lags.fnc(relationship, time='Day', group='Couple', depvar='PartnerSupport', lag=1)) -> relationship • Then, use that in a model: • model.lagged <- lmer(WarmthToday ~ 1 + Day + SupportYesterday + (1|Couple), data=relationship) Perceived warmth Support attempt TIME t TIME t-1 Support attempt Duckworth, Tsukayama, & May, 2010
  • 11. Cross-Lagged Models • model.lagged <- lmer(WarmthToday ~ 1 + Day + SupportYesterday + (1|Couple), data=relationship) Perceived warmth Support attempt TIME t TIME t-1 Support attempt Duckworth, Tsukayama, & May, 2010
  • 12. Cross-Lagged Models • Warmth at t can’t be the cause of support at t-1 • Helps clarify which is the cause and which is the effect Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Duckworth, Tsukayama, & May, 2010
  • 13. Cross-Lagged Models • Warmth at t can’t be the cause of support at t-1 • But, warmth at time t-1 could still function as a 3rd variable • Causes support attempts at time t-1 • Leads to greater warmth at time t (autocorrelation) Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth Duckworth, Tsukayama, & May, 2010 X
  • 14. Cross-Lagged Models • To rule this out, we need to include the autocorrelative effect of perceived warmth (our DV) • model.lagged2 <- lmer(WarmthToday ~ 1 + Day + SupportYesterday + WarmthYesterday + (1|Couple), data=relationship) Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth Duckworth, Tsukayama, & May, 2010
  • 15. Cross-Lagged Models • model.lagged2 <- lmer(WarmthToday ~ 1 + Day + SupportYesterday + WarmthYesterday + (1|Couple), data=relationship) Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth Duckworth, Tsukayama, & May, 2010
  • 16. Cross-Lagged Models • Now, we are seeing a time-lagged effect of support attempts over and above what can predicted by previous warmth • No way to explain this in a model where the causation only works in reverse • Strong evidence against the reverse direct of causation Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X Duckworth, Tsukayama, & May, 2010
  • 17. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
  • 18. Establishing Causality • Between-person variation in support attempts predicts within-couple change in warmth Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X Duckworth, Tsukayama, & May, 2010
  • 19. Establishing Causality • But, there’s still the possibility of a third variable that really drives this between-person difference • e.g., relationship commitment could explain variation in previous support attempts and increase in warmth Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X Relationship commitment Duckworth, Tsukayama, & May, 2010
  • 20. Establishing Causality • If relationship is driven by an underlying 3rd variable, then warmth & support don’t have a cause/effect relation • Should see the same relation regardless of their order Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X Relationship commitment Duckworth, Tsukayama, & May, 2010
  • 21. Establishing Causality • To establish causality, show that the direction of the relationship matters • Run the inverse model where support attempts are the DV and previous warmth is the predictor • model.lagged3 <- glmer(PartnerSupport ~ 1 + Day + WarmthYesterday + SupportYesterday + (1|Couple), data=relationship, family=binomial) Perceived warmth Support attempt TIME t TIME t-1 Support attempt Perceived warmth X X
  • 22. Establishing Causality • model.lagged3 <- glmer(PartnerSupport ~ 1 + Day + WarmthYesterday + SupportYesterday + (1|Couple), data=relationship, family=binomial) • No significant effects • Earlier support attempts predict later warmth (model.lagged2) • But earlier warmth doesn’t predict later support attempts (model.lagged3) • Evidence for directionality of effect
  • 23. Establishing Causality • This kind of evidence is called Granger causality • Still one kind of 3rd variable not ruled out: One with immediate effect on support attempts & a delayed effect on warmth • However, much less likely • So, not quite as good as randomized experiment • But, effective when experimental control not possible (e.g., economics, neuroscience) Perceived warmth Support attempt TIME t TIME t-1 Support attempt X Perceived warmth X ???
  • 24. Establishing Causality • This kind of evidence is called Granger causality • Still one kind of 3rd variable not ruled out: One with immediate effect on support attempts & a delayed effect on warmth • However, much less likely • So, not quite as good as randomized experiment • But, effective when experimental control not possible (e.g., economics, neuroscience) Adapted from Kaminski et al., 2011
  • 25. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
  • 26. Effect Size • Revisiting lifexpectancy.csv, let’s run a model predicting Lifespan from fixed effects of YrsEducation and IncomeThousands, and a random intercept of Family • Two variables related to socioeconomic status • Does each matter, when controlling for the other? Which is the most important?
  • 27. Effect Size • Revisiting lifexpectancy.csv, let’s run a model predicting Lifespan from fixed effects of YrsEducation and IncomeThousands, and a random intercept of Family • Two variables related to socioeconomic status • Which significantly predict the number of years that people live? • model.life <- lmer(Lifespan ~ 1 + YrsEducation + IncomeThousands + (1|Family), data=lifeexpectancy) They both do! Which is bigger?
  • 28. Effect Size • Remember that t statistics and p-values tell us about whether there’s an effect in the population • Is the effect statistically reliable? • A separate question is how big the effect is • Effect size
  • 29. Bigfoot: Little evidence he exists, but he’d be large if he did exist Pygmy hippo: We know it exists and it’s small LARGE EFFECT SIZE, LOW RELIABILITY [-.20, 1.80] SMALL EFFECT SIZE, HIGH RELIABILITY [.15, .35]
  • 30. • Is bacon really this bad for you?? October 26, 2015
  • 31. • Is bacon really this bad for you?? • True that we have as much evidence that bacon causes cancer as smoking causes cancer! • Same level of statistical reliability
  • 32. • Is bacon really this bad for you?? • True that we have as much evidence that bacon causes cancer as smoking causes cancer! • Same level of statistical reliability • But, effect size is much smaller for bacon
  • 33. Effect Size • Our model results tell us both Parameter estimate tells us about effect size t statistic and p-value tell us about statistical reliability
  • 34. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
  • 35. Effect Size: Parameter Estimate • Simplest measure: Parameter estimates • Effect of 1-unit change in predictor on outcome variable • “Each additional $1,000 of annual income predicts another 0.25 years of life” • “Each minute of exercise increases life expectancy by about 7 minutes.” (Moore et al., 2012, PLOS ONE) • “People with a college diploma earn around $24,000 more per year.” (Bureau of Labor Statistics, 2018) • Concrete! Good for “real-world” outcomes
  • 36. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
  • 37. Effect Size: Standardization • Which is the bigger effect? • 1 year of education = 0.57 years of life expectancy • $1,000 of annual income = 0.25 years of life expectancy • Problem: These are measured in different, non-comparable units • Years of education vs. (thousands of) US dollars
  • 38. Effect Size: Standardization • Which is the bigger effect? • 1 year of education = 0.57 years of life expectancy • $1,000 of annual income = 0.25 years of life expectancy • Problem: These are measured in different, non-comparable units • Years of education vs. (thousands of) US dollars • Convert to z-scores: # of standard deviations from the mean • This scale applies to anything! • Standardized scores
  • 39. Effect Size: Standardization • scale() puts things in terms of z-scores • New z-scored version of our predictors: • lifeexpectancy %>% mutate( YrsEducation.z = scale(YrsEducation)[,1], IncomeThousands.z = scale(IncomeThousands)[,1]) -> lifeexpectancy • # of standard deviations above/below mean income
  • 40. Effect Size: Standardization • scale() puts things in terms of z-scores • New z-scored version of our predictors: • lifeexpectancy %>% mutate( YrsEducation.z = scale(YrsEducation)[,1], IncomeThousands.z = scale(IncomeThousands)[,1]) -> lifeexpectancy • # of standard deviations above/below mean income • Then use these in a new model • model.life <- lmer(Lifespan ~ 1 + YrsEducation.z + IncomeThousands.z + (1|Subject), data=lifeexpectancy)
  • 41. Effect Size: Standardization • New results: • 1 SD increase in education = +2.1 years of life expectancy • 1 SD increase in income = +2.4 years of life expectancy • Income effect is bigger in this dataset
  • 42. Effect Size: Standardization • Old results: • New results: No change in statistical reliability Effect size is now estimated differently
  • 43. Effect Size: Standardization • Standardized effects make effect sizes more reliant on our data • Effect of 1 SD relative to the mean depends on what the M and SD are! • e.g., Effect of cigarette smoking on life • Smoking rates vary a lot from country to country! • Might get different standardized effects even if unstandardized is the same
  • 44. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
  • 45. Overall Variance Explained • How well can we explain this DV? • In the linear model context: • R2= • But in mixed-effect context: Which variance are we talking about? Model-explained variance Model-explained variance + Error variance Level-1 error variance Variance from level-2 clustering Jaeger, Edwards, Das, & Sen, 2017; Rights & Sterba, 2019
  • 46. • One R2 we can compute in mixed-effects context: • R2 c= • Obtain with the squared correlation between model-predicted and observed values • cor(fitted(model.life), lifeexpectancy$Lifespan)^2 • Here, 46%! Variance explained by fixed & random effects Conditional R2 All variance Jaeger, Edwards, Das, & Sen, 2017; Rights & Sterba, 2019
  • 47. 70 75 80 85 50 60 70 80 90 100 PREDICTED lifespan ACTUAL lifespan • One R2 we can compute in mixed-effects context: • R2 c= Variance explained by fixed & random effects Conditional R2 All variance
  • 48. • One R2 we can compute in mixed-effects context: • R2 c= • But, are random effects really “explaining” the variance? Variance explained by fixed & random effects Conditional R2 All variance Some families have longer average lifespans Some subjects have faster RTs than others
  • 49. • One R2 we can compute in mixed-effects context: • R2 c= • Summary: • Conditional R2 counts both fixed and random effects as explained variance • Preserves R2 as square of the correlation between observed & predicted data • Evaluates model’s ability to make good predictions • But, may overstate scientific/theoretical explanatory power Variance explained by fixed & random effects Conditional R2 All variance
  • 50. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
  • 51. • Another R2 that may be more helpful: • R2 β*= • library(r2glmm) • r2beta(model.life) Variance explained by fixed effects only Marginal R2 All variance Variance explained by all fixed effects combined (14%) Partial R2 for each fixed effect Jaeger, Edwards, Das, & Sen, 2017
  • 52. • Another R2 that may be more helpful: • R2 β*= • library(r2glmm) • r2beta(model.life) • r2dt(model1, model2) to test R2 difference between two models Variance explained by fixed effects only Marginal R2 All variance Jaeger, Edwards, Das, & Sen, 2017
  • 54. • Another R2 that may be more helpful: • R2 β*= • Summary: • Marginal R2 counts only fixed effects as explained variance • Evaluates model’s scientific/theoretical explanatory ability • Probably more useful in most purposes Variance explained by fixed effects only Marginal R2 All variance
  • 55. Week 12.1: Effect Size ! Finish Longitudinal Designs ! Follow-Up on Autocorrelation ! Cross-Lagged Designs ! Cross-Lagged Models ! Establishing Causality ! Effect Size ! Effect Size vs. Statistical Significance ! Unstandardized ! Standardized ! Variance Explained (R2) ! Conditional ! Marginal ! Interpreting Effect Size
  • 56. • Some conventional interpretations of R2 and partial R2: • But, take these with several grains of salt • Cohen (1988) just made them up • Unclear why we care about variance explained (R2) rather than standard deviations (r), in original units • Even small effects can accumulate over time (Funder & Ozer, 2019) Cohen (1988) “Small” .01 “Medium” .06 “Large” .15 Interpreting Effect Size
  • 57. • Some conventional interpretations of R2 and partial R2: Cohen (1988) Funder & Ozer (2019) “Small” .01 “Medium” .06 “Large” .15 Interpreting Effect Size .001 .04 .09 .0025 “Very Small” .16 “Very Large”
  • 58. • Consider in context of other effect sizes in this domain: • vs: • For interventions: Consider cost, difficulty of implementation, etc. • Aspirin’s effect in reducing heart attacks: r = .03, R2 < .01, but cheap! (Rosenthal, 1990) Our effect: .10 Other effect 1: .20 Other effect 2: .30 Our effect: .10 Other effect 1: .01 Other effect 2: .05 Effect Size: Interpretation
  • 59. • For theoretically guided research, compare to predictions of competing theories • The lag effect in memory: • Is this about intervening items or time? Study RACCOON 5 sec. Study WITCH 5 sec. Study VIKING 5 sec. Study RACCOON 5 sec. 1 sec 1 sec 1 sec 1 day Study RACCOON 5 sec. Study WITCH 5 sec. Study VIKING 5 sec. Study RACCOON 5 sec. 1 sec 1 sec 1 sec 1 day POOR recall of RACCOON GOOD recall of RACCOON Effect Size: Interpretation
  • 60. Effect Size: Interpretation • Is lag effect about intervening items or time? • Intervening items hypothesis predicts A > B • Time hypothesis predicts B > A • Goal here is to use direction of the effect to adjudicate between competing hypotheses • Not whether the lag effect is “small” or “large” Study RACCOON 5 sec. Study WITCH 5 sec. Study VIKING 5 sec. Study RACCOON 5 sec. 1 sec 1 sec 1 sec 1 day TEST A: Study RACCOON 5 sec. Study WITCH 5 sec. Study RACCOON 5 sec. 10 sec 10 sec 1 day TEST B: