Yates and cochran
- 1. (C) Stephen Senn 2004 1 The Analysis of Groups of Experiments’ Lessons from Yates and Cochran Stephen Senn
- 2. (C) Stephen Senn 2004 2 Why the Yates & Cochran Paper? • I had heard about this but had not read it • When I got hold of it I was astonished as to how much work it anticipated • Found that in several cases my own work ought to have cited Y&C but had not • This lecture is a way of making amends
- 3. (C) Stephen Senn 2004 3 • I read about it in a rather nice book I like • In fact I wrote it myself • How do I know about this paper?
- 4. (C) Stephen Senn 2004 4 Outline • Yates • Cochran • Cochran and Yates – The paper – How agriculture is different from medicine – How agriculture is similar to medicine – Lessons for meta-analysis today • Examples from my own research • Conclusions
- 5. (C) Stephen Senn 2004 5 Yates • Born 1902, Didsbury Manchester • Studied St John’s College Cambridge – Graduated with a first in mathematics • Surveyor in the Gold Coast (Ghana) 1927-1932 – Deep philosophical, practical & numerical understanding of least squares • Appointed Fisher’s assistant (succeeded Wishart) at Rothamsted 1931 • Appointed head of stats Rothamsted 1933 – when Fisher left for University College London
- 6. (C) Stephen Senn 2004 6 Yates (continued) • Worked on experimental design • Particularly factorial experiments • War work with Zuckerman • Later became interested in survey work • Early interest in computing • Banned matrices at Rothamsted! • Retired 1967 but continued to come in to work • Died 1994
- 7. 7
- 8. (C) Stephen Senn 2004 8 William Gemmell Cochran • Born 1909 , Rutherglen, Scotland • Studied Mathematics and Physics at Glasgow – Graduated 1931 • Cambridge diploma with Wishart • Gave up his PhD to join Yates at Rothamsted – Had already proved “Cochran’s theorem”
- 9. (C) Stephen Senn 2004 9 Yates on Cochran “... it was a measure of good sense that he accepted my argument that a PhD, even from Cambridge, was little evidence of research ability, and that Cambridge had at that time little to teach him in statistics that could not be much better learnt from practical work in a research institute.”
- 10. (C) Stephen Senn 2004 10 Cochran (Cont) • 1939 Joined faculty at Ames Iowa • 1943 joined Wilks at Princeton • 1946 Joined North Carolina Institute of Statistics • 1949-1957 Chair of biostatistics Johns Hopkins • 1957-1976 Harvard • Died 1980 Massachusetts
- 11. 11
- 12. (C) Stephen Senn 2004 12 Yates and Cochran • The analysis of groups of experiments. Journal of Agricultural Science 1938;28(4),556-580. • Covers combination of results from a series of similar experiments • Describes the methodological approaches to what we would now call meta-analysis • Much of the technical details given in Cochran’s earlier paper in JRSSB (1937)
- 13. (C) Stephen Senn 2004 13 How Agricultural Research Differs from Medical Research Agriculture • DF for estimating error usually scarce • Experiments of similar size • Many treatments per trial • Complex treatment structure • Process of measurement relatively easy Medicine • DF for estimating error abundant • Trials of very differing sizes • Few treatments per trial • Simple treatment structure • Process of measurement difficult – Ethics, missing values
- 14. (C) Stephen Senn 2004 14 Another Important Difference The difference between agricultural and medical research is that agricultural research is not done by farmers Michael Healy
- 15. (C) Stephen Senn 2004 15 But Some Similarities • Variability of material – Main effects of fields/centres, experiments/trials • Limited ability to replicate – More serious in medicine • Experiment/trial by treatment interaction – Possibly more serious in agriculture • Modest treatment differences can bring substantial long term gains
- 16. (C) Stephen Senn 2004 16 Y&C On Representativeness “..it is usually impossible to secure a set of sites selected entirely at random…the deliberate inclusion of sites representing extreme conditions may be of value. Lack of randomness is then only harmful in so far as it results in the omission of certain types and in the consequent arbitrary restriction of the range of conditions. In this respect scientific research is easier than technical research.” P558
- 17. (C) Stephen Senn 2004 17 Y&C On Trial by Treatment Interactions “If the mean square for varieties is significant, this indicates the significance of the average differences of the average difference of the varieties over the particular set of places chosen. If varieties place is also significant…it is clear that the choice of place must effect the magnitude of the average difference between varieties..even if varieties place is not significant, this fact cannot be taken as indicating no variation in the varietal differences.” P560
- 18. (C) Stephen Senn 2004 18 Issues Y&C Cover • Fixed effects estimation – Equal weighting – Weighting by observed precision • Dangers of this • ML Estimator when variances unknown – Problems when variances increase with means • Site by treatment interaction – Random effects estimators • Relationship between mean yield and effects
- 19. (C) Stephen Senn 2004 19 Issues I shall Cover • Fixed effects estimation – Equal weighting – Weighting by observed precision • Dangers of this • ML Estimator when variances unknown – Problems when variances increase with means • Site by treatment interaction – Random effects estimators • DerSimonian and Laird estimator • Relationship between mean yield and effects
- 20. (C) Stephen Senn 2004 20 The Variance Dilemma • Salsburg D. in Controlled Clinical Trials* • Claimed homoscedasticity makes it desirable to use meta-analytic techniques rather than the linear model • Claimed ratios of variances between centres of 10 are common in trials with many small centres. *Why analysis of variance is inappropriate for multiclinic trials. Cont Clin Trials 1999; 20:453-468.
- 21. (C) Stephen Senn 2004 21 0 5 1 0 1 5 2 0 2 5 3 0 N u m b e r o f c e n t r e s 0 . 0 0 . 2 0 . 4 0 . 6 0 . 8 1 . 0 P r o b a b i l i t y T w o d e g r e e s o f f r e e d o m F o u r d e g r e e s o f f r e e d o m Probability that ratio of variances will be at least 10 as a function of number of centres
- 22. (C) Stephen Senn 2004 22 The Dangers of Weighting by Observed Information Consider case where true variances equal and all centres balanced of equal size Optimal estimator weights all centres equally Observed variances will vary Weighting by observed variance will produce a variance estimate that is lower than that for the optimal estimator However the estimator is sub-optimal hence its true variance is higher. Hence significance tests anti-conservative
- 23. (C) Stephen Senn 2004 23 Lessons • Considerable variation in observed variances likely • Does not imply heteroscedasticity • Weighting by observed information may inappropriate • Linear model approach may be superior to meta-analysis
- 24. (C) Stephen Senn 2004 24 Yates and Cochran “If the error variances of the various experiments are accurately known the error variance of any form of weighted mean is given by , 2 2 1 2 2 2 2 2 1 2 1 w w w w where w’1,w’2… represent the weights actually adopted. If w’1,w’2… are equal to 1/1 2,1/2 2, this expression reduces to the expression for the error variance of the fully weighted mean, namely 1/(w’1 + w’2 +…)… If, however, the error variances are estimated and the weights depend on these estimates, the above expression will not be correct.” P574
- 25. (C) Stephen Senn 2004 25 Weighting in Meta-Analysis k i i i random i i i k i i i fixed i i i k i i i OLS i i i V V w q V V V w q V q q w n m q i i i 1 * * 2 2 * 1 2 1 1 1 , 1 1 , 1 1 1 1 OLS estimator. Assumes homoscedasticity Fixed effects estimator. In practice i 2 unknown Random effects estimator. In practice i 2 and 2 unknown
- 26. (C) Stephen Senn 2004 26 An Alternative Weighting Scheme 2 2 1 2 2 1 ˆ 1 ˆ ˆ ˆ ˆ 1 ˆ ˆ ˆ k i i i i i i k i i i i i Yates and Cochran, p37 based on Cochran 1937 To be solved iteratively
- 27. (C) Stephen Senn 2004 27 An Example • Trial of recovery after anaesthesia • Whitehead, A, Meta-Analysis of Controlled Clinical Trials , Wiley, (2002) • Outcome is Log-time of recovery • Two treatments A and B
- 28. (C) Stephen Senn 2004 28 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5 Recovery Data from Whitehead (2002) Fixed effects OLS Yates and Cochran
- 29. (C) Stephen Senn 2004 29 Precision Standardised Estimate 0 1 2 3 4 5 -2 0 2 4 Recov ery Data f rom Whitehead Galbraith Plot of Ef f ects f rom the 11 Centres
- 30. (C) Stephen Senn 2004 30 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200 OLS weights 0.00 0.05 0.10 0.15 0.20 Fixed-effect / Yates and Cochran weights 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Centre fixed effects weights Centre Yates and Cochran weights OLS weights Multi-Centre Trial of Recovery Three Systems of Weights
- 31. (C) Stephen Senn 2004 31 Criticisms of Y&C Approach • Effectively assumes that nothing is known about the variances – ML estimator for each variances • It is simply that a component of variance is recovered by “knowing” that the true treatment effects do not vary from trial to trial • In practice if DF are small OLS will be superior to this • Y&C are well aware of this
- 32. (C) Stephen Senn 2004 32 Yates and Cochran “If the experiments are all of equal precision the efficient estimate is clearly the ordinary mean of the apparent responses in each experiment, whether the true responses are the same or vary from experiment to experiment. If on the other hand, some of the experiments are more precise than others, the ordinary mean by giving equal weight to both the less and the more accurate results, may appear at first sight to furnish a considerably less precise estimate than might be obtained by more refined statistical processes….unless the experiments differ widely in accuracy…the ordinary mean is the most satisfactory as well as the most straightforward estimate to adopt.” P572
- 33. (C) Stephen Senn 2004 33 Theoretical Empirical 0.0 0.2 0.4 0.6 0.8 0.2 0.4 0.6 0.8 QQ Plot of P-v alues f or Variances WARNING: The theory of this is v ery approximate
- 34. (C) Stephen Senn 2004 34 Brand and Kragt • Suggestion in Statistics in Medicine (1992) that relationship between treatment effect and baseline risk could be important • Used control group rate to estimate baseline risk • Illustrated this with trial of beta-mimetics in tocolysis (problem of pregnancy)
- 35. (C) Stephen Senn 2004 35 control group rate 0.0 0.2 0.4 0.6 -4 -3 -2 -1 0 Fourteen trials of beta-mimetics in tocoly sis based on Brand and Kragt, 1992
- 36. (C) Stephen Senn 2004 36 Baseline Risk • Does the treatment vary according to baseline risk? • This has been investigated by various authors • But there is a trap – The fact that you are regressing statistics of the form Y – X on X induces a correlation (Senn, 1994)
- 37. (C) Stephen Senn 2004 37 Yates and Cochran On Mean Yield and Effects "The artifice of taking the regression on the mean yield of the difference of one variety from the mean of the others is frequently of use in revealing relations between general fertility and varietal differences....The object of taking the regression on the mean yield rather than on the yield of the remaining varieties is to eliminate a spurious component of regression which will otherwise be introduced by experimental errors."
- 38. (C) Stephen Senn 2004 38 av erage log-odds both groups -2.0 -1.5 -1.0 -0.5 0.0 -4 -3 -2 -1 0 Fourteen trials of beta-mimetics in tocoly sis based on Brand and Kragt, 1992
- 39. (C) Stephen Senn 2004 39 Conclusion • Before Hedges and Olkin, before Glass and certainly long before Senn there was Yates and Cochran (1938) • And before Yates and Cochran there was Cochran (1937) • There is nothing new under the sun (folk- saying) • There is nothing new under the sum (meta- analyst’s saying)
- 40. (C) Stephen Senn 2004 40 • Finally, I would like to leave you with this question • Did you know that there are only 130 shopping days until Christmas?