Sweitzer,Simulating Multi Phase Studies
0 likes430 views
The document discusses the complexities and methodologies in managing multiphase survival trials, particularly focusing on predicting completion times and handling high withdrawal rates. It emphasizes using stochastic modeling to assess patient progression through various trial phases, optimizing enrollment and randomization strategies, and addressing uncertainties related to relapse rates and trial outcomes. Additionally, it outlines the importance of continuous data updating and how simulations can guide trial management decisions effectively.
Sweitzer,Simulating Multi Phase Studies
- 1. Signs of the Timings: Predicting Time of Completion in Multiphase Survival Trials Dennis Sweitzer Ali Falahati Delaware Chapter of the ASA September, 2006
- 3. The Protocols Outcome: • Time to Randomized Relapse Open Label Phase – Up to 36 weeks – Patients must be stable for 12 weeks before randomization – High withdrawal rate (30-70%) – Assumed 50% randomize Randomized Phase – Up to 104 weeks – High withdrawal rate – Assumed 30% Relapse rate – Trial could not end until last Patient randomized >28 weeks
- 4. Sensitivity to Relapse & Discontinuation rates (1) Low Discontinuation relative to Relapse Cumulative Patient statuses as trial progresses 100 Relapse ~Sep Wrong assumptions, wait longer
- 5. Sensitivity to Relapse & Discontinuation rates (2) Higher event rates deplete patient pool Plan to stop enrollment as soon as certain of reaching 100 ~ July Higher Discontinuation Rate, Lower relapse Rate Large delays May never reach goal
- 6. Stopping Enrollment Stopping Criteria • At least 227 Relapses • All patients still in Randomized Phase complete at least 28 weeks of treatment Ideally: • 227th Relapses occurs shortly after: • All patients randomized >28 wks (Per Protocol) • Randomization closed when: • All enrolled patients randomize or discontinue 28 week Requirement later dropped (Protocol Amendment) Presentation: use 200 Relapses
- 7. The Problems Long Lead times • Up to 36 weeks before randomization • Plus 28 weeks Minimum randomization Ideally: Stop enrollment 64 weeks before target #Relapses Must account for • Enrollment D/C (30%-70%) • Randomized D/C (D/C Rate ≈ Relapse Rate) • Relapse Rates vary (Higher Relapse Rate Early) • Competing Relapses (D/C vs Relapse) • Sensitivity to rates (Close Rates High Variability)
- 8. Stopping Enrollment: Issues Too Early: Too Late: Fewer Patients Higher certainty of Fewer randomized reaching Goal Longer wait for Patients possibly in target #Relapse. Open Label at End May never reach Excessive #Relapses at target #Relapses end of study Ethics of Randomizing Excess # pts
- 9. Many Management Questions When do we stop enrollment while being sure of eventually getting target # Relapses? When can we stop randomization & ensure reaching the target? Whats the earliest and latest we can expect to reach the target? When will all pts be randomized >28wks? When can the trial be halted (required # Relapses & all pts randomized >28 wks) Estimated Randomization Rate? Estimated Relapse Rate? How many active patients at the end? How well does the outcome match our assumptions etc etc
- 10. Simulation Solution • Make a stochastic model of the trial • Monthly: – Base model parameters on blinded data observed to date – Incorporate assumptions where data insufficient – Incorporate uncertainty of parameters – Execute 1000’s of simulations of the trial – Compute statistics from the collection of simulated trials – Repeat with new data
- 11. Advantages Transparency • Modeling assumptions can be: Specified -- Graphed -- Debated Data Driven • New Data updates the model • Existing Active Patients are simulated to end • Assumptions become less important as data accumulates
- 12. Vision of Output • Simulation reports varied according to changing team needs (how many open label patients on June 1? When will we reach 150 Relapses? How many randomized patients at time of 200th Relapse? If we stop randomizing on May 15, how many open lable patiestin will there be?……………………………………………….
- 13. Stochastic Modeling Approach 1. Make a cartoon model of a patients progress through the trial 2. What final outcomes are possible? 3. What could happen to the patient? 4. Identify States through which a patient passes 5. Identify Random Processes which take patients between states
- 14. Stochastic Model Discontinued Patients (Open Label Phase) Discontinued Patients Open Label (Randomized Enrolling Phase) Patients Patients Randomized Patients Relapses Continuous Time Markov Chain Markov States: the Bubbles Transitions: the Arrows Transition Probabilities change with time in state
- 15. States Discontinued ) Patients (Open Label Phase) Discontinued Patients Enrolling ) Open Label )(Randomized Phase) Patients Patients Randomized Patients ) ) Relapses 2 Transitory Markov States: Open Label Phase Randomized Phase 3 Terminal Markov States: Discontinued from Open Label Phase Discontinued from Randomized Phase Randomized Relapses
- 16. Transition Processes Discontinued Patients ) ) (Open Label Phase) Discontinued Patients (Randomized Phase) Enrolling Patients Open Label Patients ) Randomized Patients ) Relapses ) 5 Random Transition Processes: 1. Trial Enrollment (Start Open Label) 2. Discontinuation from Open Label Phase 3. Randomization (from Open Label Phase) 4. Discontinuation from Randomized Phase 5. Randomized Relapses
- 17. Trial Enrollment ) Enrolling Open Label Patients Patients For each simulated patient, generate a random length of time since the last patient • Pick an enrollment rate λ (Based on history & judgment) • Assume: #pts/mo ~ Poisson process with mean λ • Time between patients ~ Exponential(1/λ) Can expand enrollment model to evaluate management options: • Incorporate mixture of site performances • Adding/changing sites during the trial
- 18. Markov Process ) Discontinuation Continuing ) Relapse (or Randomization) Probability of transitioning from state i to state j between times s and t
- 19. Aalen-Johansen estimator of Transition Probabilities • Aalen-Johansen estimator of the transition probability matrices For and # obs. Direct transitions from states h to j, visits 1 to t # pts in state h, just prior to visit t
- 20. Aalen-Johansen & Kaplan-Meier • Generalization of Kaplan-Meier Estimation to Non-homogeneous Markov Chains • K-M Estimators easier: – To program (already in SAS) – To understand (Intuitive) – To Explain (Familiar)
- 21. Models • Enrollment: Poisson Process • Open Label Phase: Competing Risk Model 0= Still in OL Phase 1= Randomized 2= Discontinue fr OL phase • Ramdomized Phase: Competing Risk Model 0= Still in Rand Phase 1= Manic event 2= Depressed event 3= Discontinue fr Rand phase
- 22. Competing Risk Model Mutually exclusive events (e.g., Relapse vs Discontinuation, …) 2 Approaches (Pintilie, 2006) • Jointly distributed Random Variables • Latent failure times – Assume both events eventually occur – But we only observe the first – Use only marginal distributions – Assuming independence (between events) – But cannot test for independence, if only observing 1st – Independence: Face validity & Simplest Assumption
- 23. Kaplan-Meier Simulation • Assume event are independent • Model Each process separately using Kaplan- Meier Estimators • Censor on other event, current time in trial • Simulate each event separately • Earliest of the 2 simulated processes is taken as simulated outcome • Caveat: Assumes Independent processes Intuitive, easy to understand, easy to explain
- 24. Open Label Transitions Discontinued Patients ) (Open Label Phase) 2 Competing Open Label Patients Processes: ) Randomized Patients Discontinuation Randomization 1. Generate Random Discontinuation time 2. Generate Random Randomization time 3. Use the earliest event
- 25. Randomized Phase Processes Discontinued Patients (Randomized ) Phase) Randomized Patients Relapses ) 2 Competing Processes: Discontinuation * * Relapse Choose event as previously described. • Current Open Label Patients are simulated to randomization or discontinuation • If simulated randomization, then simulate Randomized Discontinuation or Relapse
- 26. Generic Transition Process Q: When to make the transition? State State A: First: estimate random transition "A" "B" function 1. Generate K-M Survival Functions from data (censoring on all other events) 2. Make assumptions about Survival beyond last event ?
- 27. Simulated Patients State State (p, t) "A" "B" p Q: When to make the transition? t A: Second: Simulate Trials For Each Simulated Trial: • For each simulated patient within a trial – Pick a random p∈(0,1) – Interpolate t from the graph, so that (p, t) is on graph
- 28. Simulating Active Patients State State q (q, s) "A" "B" (q*p, t) q*p For each simulated trial s t • For each observed patient within state “A” for time s – Interpolate q∈(0,1) from the graph, so that (q, s) is on graph – Pick a random p∈(0,1) – Interpolate t from the graph, so that (q*p, t) is on graph
- 29. Incorporating Parameter Uncertainty State State "A" "B" For each simulated trial q • Pick a random quantile r∈ r (0,1) • Simulate all patients using the r%-tile confidence level t of the Kaplan-Meier Curve Simulates: combinations of high & low estimates of Event and D/ C Survival curves
- 30. Limitations Requires representative data from all phases • K-M estimates only through last event • Assumptions must be made about hazard rate after last available event(s) – If assumptions correct, point estimates should be stable while confidence intervals narrow • Up to date data – Special reporting of Relapses (faxes with follow up, monitoring) – IVRS, EDC, monitoring reports • Heterogeneity: – Earliest sites may not be representative of all sites – Procedures may change (hopefully improve) over time – Regional differences (standards of care, patient attitudes, etc)
- 31. Why Not a Parametric Model? Trial Structure: • Events tend to occur on visits granularity ✭ Continuous • Visits vary in spacing ✭ Discrete • Active Tx mixture model Changing Hazard over time • Must make & defend simplifying assumptions
- 32. Diagnostic: Does It Fit? Survival curves of: Observed data vs. Simulated Data (Censored Observed, Active OL Pts, Active Rand. Pts, Entirely simulated Pts)
- 33. Diagnostics Plot K-M curves for each event, time in each phase • Review assumptions (long term behavior) • Identify data anomalies • Identify simulation problems
- 34. Example: Regional Heterogeneity Regional modeling (Trials A & B): Parameters varied by region more than by trial – Estimate parameters within regions – Simulate patients with Trial and Region – Summarize results by Trial In addition to simulations which ignored region Survival curves followed 2 patterns by region & trial
- 35. Reporting the Simulations For each simulated Trial: • Sort Patient Events by occurrence date (enrollment, randomization, relapse, etc) For each scenario • Summarize over Event records which fit scenario Examples: • Summarize over all patients enrolled before a potential enrollment cutoff date. • … over all patients randomized before a cutoff date • Summarize with and without a subset of sites
- 36. Changing Questions • Early in Trial – Are the protocol assumptions accurate? – When to stop enrollment? – Expected # patients (enrolled, randomized, etc) Identify problems & Evaluate fixes • Mid-Trial – When to stop randomizing patients? – Are the revised assumptions accurate? – Were changes effective? • Late Trial – When will the last Relapse occur? – How many patients will be active in various phases? Plan for Closeout & Database lock
- 37. Early Trial For each Enrollment Cutoff Date • Summarize each trial for all patients enrolled before that date • Compute statistics over simulated trials Some trial outcomes: • Dates of: last Relapse; All patients randomized>28 week; PP Completion (>target & >28 weeks) • Event & patient counts at each of above milestones • % of Simulations with ≥200, 190, 180,… Relapses at milestones • # active patients (open label or randomized) at given dates & milestones Pick a cutoff date accordingly (e.g., minimize resource with least risk of running late)
- 38. Trial Completion (1) ≥227 Relapses & All Active Patients Randomized 28wks Earliest Completion: • 75% Certainty: 2 August Cutoff for Nov 2006 Completion • 90% Certainty: 1 Sep Cutoff for Dec 2006 Completion
- 39. Trial Completion (2) ≥227 Relapses Events & All Active Patients Randomized 28wks Earliest Completion: • 75% Certainty: ~1775 Enrolled for Nov 2006 Completion • 90% Certainty: ~1850 Enrolled for Dec 2006 Completion
- 40. 29 Oct prediction from: 528 Enrolled, 272 Rand., & 58 Relapses On 26 June: ~1950 Enrolled, 620 Randomized, 200 Relapses
- 41. Highly uncertain: 3 Relapses, 2 Randomized D/C, 73 Randomized
- 43. 1 Month Later: 16 Relapses, 7 Rand.D/Cs, 182 Randomized
- 44. Slowed Enrollment: 29 Relapses, 22 Rand. D/C, 272 Rand
- 45. Example: Will a trial end? Study E: • Endpoint: 300 type 1, 300 type 2 events • Slower & Fewer than expected • Simulation predicted: – 10% chance of 300 of each – 87% chance of 600 total • Interim Analysis – Supported by simulation – 300 total expected April 25
- 46. Mid-Trial Output For each Randomization Cutoff Date • Summarize each trial for all patients randomized before that date • Compute statistics over simulated trials • Generates same statistics per trial • Summarize for each randomization cutoff date Essentially, replace “enrollment” with “randomization” & execute as before
- 47. • Conclusion: nothing to be gained by randomizing beyond mid June
- 48. Late-Trial Output Refine estimates of last Relapse, etc. For Milestones & Calendar Dates • Estimate #patients in each stage (e.g., how many patients will be active at the end?) Caveats: • Corrected (or just collected) data may change estimates • Old, unreported Relapses may be discovered Useful: • Predict time between milestone to end • Add prediction to best guess milestone
- 49. Bottom line: How accurate? Not bad: • Actual Date of 200th Relapse covered by predicted 80% C.I. • Width of C.I.s narrowed over time
- 53. Value Added Early Refinement of Protocol Assumptions • Protocol: 50% randomized, 30% Relapse rates • Trial A: 33%, 37% Trial B: 55%, 41% Early Identification of problems Quick Response to problems • Changed procedures to improve retention in Trial A • Added sites to Trial B after delays in starting up sites Better allocation of resources
- 54. Trials C, D Mid trial: Regulators requested analysis of late Relapses • Enrollment had already ended for Trial C – Enough patients to reach 88 late Relapses? • Enrollment was still ongoing for Trial D – Extend enrollment how long? Add sites? • How would this affect time lines?
- 57. Dirty Data Problems • Some known Relapses are not usable due to missing data (e.g., unknown randomization date) • Corrected (or lately collected) data may change estimates • Old, unreported Relapses may be discovered • Data may be collected or corrected irregularly – Separate data sources (e.g., Relapse log + IVRS) – Drift & shift over time – End of trial data clean up Solutions: • Estimate time between milestones – Anchor to a known, early milestone Future solutions: • Estimate missing data effects – Use time between Relapse occurrence and Relapse reporting – Estimate number of missing Relapses from times in past
- 58. Some Feedback • “… the simulations are very valuable and the only way we have to plan our timelines. As it has turned out, your simulations seems to be pretty accurate as we have increased the mood event rate significantly … as predicted...” • ... We would have been guessing and spinning our wheels without them.” • Could you simulate trials xyz & uvw?
- 59. Extra Slides
- 60. Diagnostics: Cohort Analysis Cohorts: • By month enrolled • By month Randomized Calculate randomization & Relapse rates Easy to understand Multiple Estimates which must be reconciled Doesn’t Provide Time to Relapses Useful reality check on Simulation Point Estimates insufficient: need C.I.s
- 61. Cohorts by Randomization Month Actual counts of patients by status by month
- 62. Cohorts by Month Randomized (Cumulative Count) Height is # patients randomized in or before month
- 63. Cohorts by Month Randomized (Cumulative Percentage) • >40% randomized eventually have Relapse <N/0.40 randomized to get N
- 64. Open Label Cohorts (Current Status)
- 65. Open Label Cohorts (Cumulative Statuses) • ~ 64% eventually Randomize & >40% Relapse Rate • # open label pt < N/(0.64*0.40)
- 66. Solutions? Crude Relapse Rates of all Patients in Phase Mixture of patients: • Relapse & D/C rates change with exposure • Mixture of Pt. Exposures changes with time Cohorts: Track Relapses over Time Easy to understand Multiple Estimates which must be reconciled Doesn’t Provide Time to Relapses Useful reality check on Simulation Point Estimates insufficient: need C.I.s
- 67. Clinical Trial Management Planning Trials (future) • Is a trial feasible? • Sensitivity to assumptions? • Costs: # Pts, # Pt-mos, #visits, #Sites, #Site-mos Trial Execution (current) • Anticipate delays • No information on outcome • Could be added to simulation Program Planning (future) • Replace “Trial Phases” with “Toll Gates” • Enhance modeling of Trial Enrollment process
- 68. Example: Adding Site Sites Discontinued Patients Randomized Patients New Relapses Sites • Drop OL Phase, expand enrollment process • Simulate time to start up new site, pts/mo at a new site, etc • Report by #Additional Sites instead of cutoff dates