Effect Size
Download as PPT, PDF7 likes9,843 views
This document discusses the importance of evaluating effect size, not just statistical significance (p-value), when analyzing differences between groups. It provides examples to illustrate effect size and defines some common effect size measures, such as Cohen's d and Hedges' g. The document also discusses how effect size can be used to determine required sample sizes needed to achieve a desired p-value threshold and level of confidence in the results.
![P value vs Effect Size By increasing sample size, you can show there is a statistically significant* difference between two Means. The Effect Size evaluates how material is that difference. P value = probability sample Means are the same . (1 – P) or C.L. = probability sample Means are different . Effect Size = how different sample Means are. *Statistically significant does not imply “significant.” [Webster: … of consequence.]](https://image.slidesharecdn.com/effect-size2-1231175518420205-1/85/Effect-Size-2-320.jpg)
![Cliff Delta efficiently… [(1 x 0) – (1 x 8)]/10 = -0.8](https://image.slidesharecdn.com/effect-size2-1231175518420205-1/85/Effect-Size-11-320.jpg)
Effect Size
- 1. Beyond p value: Effect Size April 4, 2008 Guy Lion
- 2. P value vs Effect Size By increasing sample size, you can show there is a statistically significant* difference between two Means. The Effect Size evaluates how material is that difference. P value = probability sample Means are the same . (1 – P) or C.L. = probability sample Means are different . Effect Size = how different sample Means are. *Statistically significant does not imply “significant.” [Webster: … of consequence.]
- 3. Effect Size in Plain English Large Effect Size is visible without looking at a large sample. Other Effect Size examples: The Kaplan course raises SAT math scores by 60 points; This WOW initiative increases # Solutions by 30 units. When analyzing Effect Size we deal with standardized units… With sea lions gender has a large Effect Size. With pugs gender has a small Effect Size
- 4. Effect Size Measures Cohen’s d = (Mean Pilot – Mean Control )/Pooled Stand. Deviation Cohen’s d is similar to the unpaired t test t value. It relies on Standard Deviations instead of Standard Errors. Hedges’ g is a more accurate version of Cohen’s d Hedges’ Cohen’s d Adjustment for small sample
- 5. Pilots vs Controls In this example, there is a 1.9 Standard deviation difference between Pilots and Controls.
- 7. An Example
- 8. ES Confidence Interval The Effect Size standard deviation formula allows to build Confidence Intervals around Effect Size values.
- 9. 1 st Nonparametric test: Gamma Index Recalculated Gamma Index to make it the same sign as Cohen's d and Hedges' g.
- 10. 2 nd Nonparametric test: Cliff Delta Cliff Delta ranges between +1 when all values of one group are higher than the values of the other group and – 1 when reverse is true. Two overlapping distributions would have a Cliff Delta of 0.
- 11. Cliff Delta efficiently… [(1 x 0) – (1 x 8)]/10 = -0.8
- 12. Cliff Delta vs Cohen’s d If distributions are Normal, Cliff Delta = Percent of Nonoverlap within a Cohen's d framework.
- 13. Effect Size and Sample Size requirement The above formula results from the algebraic transformation of: t stat or Z value = Difference in Means/Group Standard Error.
- 15. Conclusion P value does not tell us how different two samples are. Effect Size and its Confidence Intervals give much information on how different two samples are. Effect Size in units allows us to derive a required sample size to meet a p value threshold.