Jsm2013,598,sweitzer,randomization metrics,v2,aug08
0 likes717 views
This document discusses methods for assessing randomization in clinical trials that use covariate-adaptive randomization designs. It presents metrics for measuring randomness, balance, and efficiency loss in randomization schemes. The document outlines a simulation approach and discusses results from comparing different randomization factors and sample sizes. It proposes future directions such as optimizing randomization parameters and exploring periodicity in system behavior.
![© 2013 Medidata Solutions, Inc. 13© 2013 Medidata Solutions, Inc. 13
Randomness: Entropy/Syntropy
Observed Entropy
≣ Self Information Content
Where: pj ≣ probability of observed
treatment choice for patient j
Syntropy*
• Average & Rescale to [0,1] so that:
0 ⟹ Max Randomness
1 ⟹ Max Determinism
I = − log(pj )∑
Syntropy
* “Syntropy” ― coined by
Buckminster Fuller as the
opposite of entropy
Q: Isn’t DA deterministic?
A: A random element makes
it as random as PB](https://image.slidesharecdn.com/jsm2013598sweitzerrandomizationmetricsv2aug08-130811060442-phpapp02/85/Jsm2013-598-sweitzer-randomization-metrics-v2-aug08-13-320.jpg)
Jsm2013,598,sweitzer,randomization metrics,v2,aug08
- 1. © 2013 Medidata Solutions, Inc. 1© 2013 Medidata Solutions, Inc. 1 Dennis Sweitzer Principal Biostatistician Slides & supplementary material wil be posted at: www.Dennis-Sweitzer.com 7 August 2013, JSM Randomization Metrics: Jointly assessing predictability and efficiency loss in covariate adaptive randomization designs
- 2. © 2013 Medidata Solutions, Inc. 2© 2013 Medidata Solutions, Inc. 2 Outline § Objective What is the right questions, anyway? § Randomness How to measure & from whose perspective? § Balance Why? How to measure to match? § Simulation Validity? § Results NB: Slides & supplementary material will be posted at: www.Dennis-Sweitzer.com blog.mdsol.com
- 3. © 2013 Medidata Solutions, Inc. 3© 2013 Medidata Solutions, Inc. 3 Questions Q: Will unequal subgroups affect randomization performance? Q: What are the impacts of choosing dynamic allocation over permuted block? Q: Dynamic allocation is more deterministic than permuted block, isn’t it? Q: What about randomization performance at interim analysis? What is the best method for randomizing THIS study design in THIS population of patients?
- 4. © 2013 Medidata Solutions, Inc. 4© 2013 Medidata Solutions, Inc. 4 Measuring Randomness Randomness Predictability (by observer) Entropy (no observer) Periodicity (patterns) X ⟶ ⟶ Y (as function of probabilities)
- 5. © 2013 Medidata Solutions, Inc. 5© 2013 Medidata Solutions, Inc. 5 Measuring Balance Balance Efficiency (Variability) Confounding (Bias) Deviation from Target (Convenience) X (as # or % of subjects) ⟶ ⟶ Y (as function of probabilities)
- 6. © 2013 Medidata Solutions, Inc. 6© 2013 Medidata Solutions, Inc. 6 Generic Simulation Convenient covariates… Covariate Levels Ratio 2 sexes {M,F} 50:50 3 age groups {Mid,Yng,Old} 33:33:33 10 sites / variants {a,b,…,j} 10 x 20 each However ➣ Real-life trials are rarely so neat Although ➣ Simulated trials usually this neat pk ∝ 1 k +c( ) a 2 sexes 67:33 3 age groups 55:27:18 10 sites 34 : 17 : 11 : 9 : ... : 3.4 Use… Zipf-Mandelbrot Distribution ➣ Sizes of cities, frequencies of words, species abundance, Website hits… Q: Will unequal subgroups affect performance? ANCOVA Model Outcome = Treatment + Sex + Age + Sex*Age + Variant/Site+error
- 7. © 2013 Medidata Solutions, Inc. 7© 2013 Medidata Solutions, Inc. 7 Balance: Confounding Ad hoc: Score ≣ Total of #subjects in covariate subgroups with 100% of a single treatment ANCOVA Model Outcome = Treatment + Sex + Age + Sex*Age + Variant/Site+ error 0 2 4 6 8 CR Stratified ………... PB(1:1) PB(2:2) PB(4:4) DAS(0%) DAS(15%) Marginal ………… DAM(0%) DAM(15%) Strata + Margins DAE(0%) DAE(15%) Site + DAC(0%) DAC(15%) Site+Strata DAD(0%) DAD(15%) Equally Distributed Covariates Zipf- Mandelbrot Covariates Confounding score⟶ A: Yes, increased confounding!
- 8. © 2013 Medidata Solutions, Inc. 8© 2013 Medidata Solutions, Inc. 8 Balance: Loss of Efficiency ANCOVA Model Outcome = Treatment + Sex + Age + Sex*Age + Variant/site+ error 0 2 4 6 8 10 12 14 16 CR Stratified ………... PB(1:1) PB(4:4) DAS(0%) DAS(15%) Marginal + …. DAM(0%) DAE(0%) Site + Margins…… DAC(0%) DAC(15%) Site+Strata+Margin…. DAD(0%) DAD(15%) Loss of Efficiency (LOE) ⟶ Equally Distributed Covariates Zipf-Mandelbrot Covariates ! ! = !! + !!!! (Atkinson, 2003) Matrix Form of model, where: z ≣treatment allocation α ≣treatment effect β ≣Covariate effects X ≣ Design Matrix Columns ó Covariates Rows ó Subjects Loss of Efficiency: Var( ˆα) = σ 2 zt z−zt X(Xt X)−1 Xt z LOE = zt X(Xt X)−1 Xt z A: But not efficiency
- 9. © 2013 Medidata Solutions, Inc. 9© 2013 Medidata Solutions, Inc. 9 Randomness: Predictability Blackwell-Hodges (1957) guessing rule ☞ Game theory interpretation ☞ Always guesses the next assignment will restore balance Measures Potential Selection Bias F ≣ abs(# Correct – Expected # Correct by chance alone) PotentialSelectionBias(Strata) Q: Impacts of choosing dynamic allocation over permuted block ?
- 10. © 2013 Medidata Solutions, Inc. 10© 2013 Medidata Solutions, Inc. 10 Randomness: PredictabilityPotentialSelectionBias(Strata) A: More efficiency, less predictability Randomization factors Pb ≣ Sex*Age daC ≣ Sex + Age + Variant daD ≣ Sex + Age + Sex*Age +Variant
- 11. © 2013 Medidata Solutions, Inc. 11© 2013 Medidata Solutions, Inc. 11 Randomness: PredictabilityPotentialSelectionBias(Site) Randomization factors Pb ≣ Sex*Age daC ≣ Sex + Age + Site daD= Sex + Age + Sex*Age +Site A: … unless the observer knows too much…
- 12. © 2013 Medidata Solutions, Inc. 12© 2013 Medidata Solutions, Inc. 12 Randomness: PredictabilityPotentialSelectionBias(Site) Randomization factors daD= Sex + Age + Sex*Age + Site daE ≣ Sex + Age + Sex*Age da? = Sex + Age + Sex*Age + ½ Site A: However, can adjust weights
- 13. © 2013 Medidata Solutions, Inc. 13© 2013 Medidata Solutions, Inc. 13 Randomness: Entropy/Syntropy Observed Entropy ≣ Self Information Content Where: pj ≣ probability of observed treatment choice for patient j Syntropy* • Average & Rescale to [0,1] so that: 0 ⟹ Max Randomness 1 ⟹ Max Determinism I = − log(pj )∑ Syntropy * “Syntropy” ― coined by Buckminster Fuller as the opposite of entropy Q: Isn’t DA deterministic? A: A random element makes it as random as PB
- 14. © 2013 Medidata Solutions, Inc. 14© 2013 Medidata Solutions, Inc. 14 Results: Metrics and Changing Sample SizePotentialSelectionBias(Strata) Q: What about randomization performance at interim analysis? A: PB becomes more predictable & a little more efficient A: DAC is and becomes both less predictable and more efficient
- 15. © 2013 Medidata Solutions, Inc. 15© 2013 Medidata Solutions, Inc. 15 Next Directions § Compare more methods: Urn randomization, Optimal-Designs, Novel methods, etc § Randomization Metrics vs statistical properties of analyses § Optimizing parameters & tweaking algorithms § Refining metrics (e.g., Deviation from Target, Periodicity) § Exploring quirks in system behavior. § For more information (slides, bibliography, supplemental material, etc.) see: blog.mdsol.com OR www.Dennis-Sweitzer.com OR www.slideshare.net/denswei
- 16. © 2013 Medidata Solutions, Inc. 16© 2013 Medidata Solutions, Inc. 16 Additional Slides § Bibliography § Randomization factors used § Comparing Methods Example § Periodicity Plot
- 17. © 2013 Medidata Solutions, Inc. 17© 2013 Medidata Solutions, Inc. 17 Bibliography § Atkinson, AC. (2003) The distribution of loss in two-treatment biased-coin designs. Biostatistics, 2003, 4, 2, pp. 179–193 § Blackwell, D. and J.Hodges Jr (1957). Design for the control of selection bias. Ann Math Stat 28, 449-460 § Wikipedia contributors. "Entropy (information theory)." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 23 Apr. 2013. Web. 14 May. 2013. § Lebowitsch, J, et al, (2012). “Generalized multidimensional dynamic allocation method”. Statistics in Medicine,2012;
- 18. © 2013 Medidata Solutions, Inc. 18© 2013 Medidata Solutions, Inc. 18 Covariates vs Randomization Factors Analysis: ANCOVA Model Outcome = Treatment + Site + Sex + Age + Sex*Age Stratification factors in Randomization Strata Imbalances– within combinations of Sex & Age Marginal Imbalances – within each Sex, Age, and Site “S” PB, daS “M” daM “C” daC “D” daD
- 19. © 2013 Medidata Solutions, Inc. 19© 2013 Medidata Solutions, Inc. 19 Comparing Methods & Parameters Predictability vs Loss of Efficiency, (n=50) Not much difference in CI Variations on DA: • daJS, daJM – (Kuznetsova, 2012) • mmS, mmM, baM, baF, baS - experimental PotentialSelectionBias(Strata)
- 20. © 2013 Medidata Solutions, Inc. 20© 2013 Medidata Solutions, Inc. 20 Randomness: Periodicity A la Discrete Fourier Transform • Amplitude of a periodic variation in the max{pi,j} of treatment assignments Periodicity