Numbers needed to mislead
Download as PPTX, PDF1 like1,837 views
1) The document discusses how clinical trial responses are often misinterpreted through the use of dichotomies and responder analysis. 2) An example simulation is provided showing how dichotomizing a continuous outcome measure and conducting responder analysis on a single clinical trial can misleadingly suggest some patients respond to treatment while others do not. 3) In reality, the simulation shows that all patients may experience the same proportional benefit from treatment, but dichotomizing the data obscures this and encourages unfounded conclusions about personalized medicine.
Numbers needed to mislead
- 1. (c) Stephen Senn 2016 1 Numbers needed to mislead, meta- analysis and muddled thinking Stephen Senn
- 2. Acknowledgements I am grateful for the opportunity the University of Sheffield is giving me, through the inaugural lecture programme, to address you on what I think is an important topic and also, of course, for having granted me the title of honorary professor. This work is partly supported by the European Union’s 7th Framework Programme for research, technological development and demonstration under grant agreement no. 602552. “IDEAL” 2(c) Stephen Senn 2016
- 3. Outline • An example as to how response in clinical trials is misinterepreted • Why this matters • What we should do about it (c) Stephen Senn 2016 3
- 4. Alas Smith and Jones A question for you (c) Stephen Senn 2016 4 Ms Smith had her headache reduced from 8 hours duration to 6 (reduced by 2 hrs or 25%) Mr Jones had his headache duration reduced from 2hr05’ to 1hr55’ (reduced by10 minutes or 8%) Who had the greater benefit? The International Headache Society recommends the outcome of being pain free two hours after taking a medicine. So does the FDA Mr Jones responded. Ms Smith didn’t.
- 5. (c) Stephen Senn 2016 5 59% had no headache after 2 hours when treated with paracetamol 49% had no headache after 2 hours when treated with placebo 59%-49% = 10% Therefore 10% benefitted The number needed to treat for one extra patient to have a benefit is 10
- 6. (c) Stephen Senn 2016 6 ‘It tells us we can help about 35% of migraine patients’
- 7. Painful comparison Cochrane Collaboration meta-analysis • Meta-analysis of placebo- controlled trials of paracetamol in tension headache • 23 studies • 6000 patients in total • Outcome measure: – Pain free by 2 hours Baayen Significance article • Explanation of Novartis’s MCP-Mod dose-finding approach using a trial run by Merck • 7 doses + placebo • 517 patients in total • Outcome measure – Pain free by 2 hours (c) Stephen Senn 2016 7
- 8. In both cases • The patients were only studied once • A dichotomy of a continuous measure was made • Patients were labelled as responders and non- responders • A causal conclusion was drawn that went beyond simply comparing proportions – Baayen talked about the proportion of patients who would respond – Cochrane talked about the proportion of patients to whom it would make a difference in terms of response (c) Stephen Senn 2016 8
- 9. Headaches or patients? • In fact conclusions about the proportion of patients who will regularly have a response to treatment cannot be drawn from such studies • You cannot separate headaches and patients • Furthermore the dichotomy causes causal confusion (c) Stephen Senn 2016 9
- 10. What I propose to do • Create a simple statistical model to mimic the Cochrane result – In terms of time to pain resolution every patient will have the same proportional benefit • In fact I shall be using a form of proportional hazards model – The dichotomy will classify patients as responders or non-responders – We will be tempted to conclude that some don’t benefit and some do and that this is a permanent feature of each patient (c) Stephen Senn 2016 10
- 11. The Numerical Recipe • I shall generate pain duration times for 6000 headaches treated with placebo – This will be done using an exponential distribution with a mean of just under 3 hours (2.97 hrs to be exact) – Each such duration will then be multiplied by just over ¾ (0.755 to be exact) to create 6000 durations under paracetamol • I shall then take the 6000 pairs and randomly erase one member of the pair to leave 3000 unpaired placebo values and 3000 unpaired paracetamol values • I shall then analyse the data (c) Stephen Senn 2016 11
- 12. Why this recipe? • The exponential distribution with mean 2.970 is chosen so that the probability of response in less than two hours is 0.49 – This is the placebo distribution • Rescaling these figures by 0.755 produces another exponential distribution with a probability of response in under two hours of 0.59 – This is the paracetamol distribution (c) Stephen Senn 2016 12
- 13. (c) Stephen Senn 2016 13
- 14. (c) Stephen Senn 2016 14 Enlarged portion of previous graph
- 15. Dichotomania • We lose information through such dichotomies • We tend to believe our own nonsense labels – Response – Non-response • We then delude ourselves that Nature also believes our nonsense • Next stop: personalised medicine (c) Stephen Senn 2016 15
- 16. However • So far I have only gone half way in my simulation recipe • I have simulated a placebo headache and a corresponding paracetamol headache • However I can’t treat the same headache twice • One of the two is counterfactual • I now need to get rid of one member of each factual/counterfactual pair (c) Stephen Senn 2016 16
- 17. (c) Stephen Senn 2016 17
- 18. (c) Stephen Senn 2016 18 As before but with log scale
- 19. (c) Stephen Senn 2016 19
- 20. To sum up • The results reported are perfectly consistent with paracetamol having the same effect on every single headache • This does not have to be the case but we don’t know that it isn’t • The combination of dichotomies and responder analysis has great potential to mislead • Researchers are assuming that because some patients ‘responded’ in terms of arbitrary dichotomy there is scope for personalised medicine (c) Stephen Senn 2016 20
- 21. A previous Prime Minister of the UK speaks This agreement will see the UK lead the world in genetic research within years. I am determined to do all I can to support the health and scientific sector to unlock the power of DNA, turning an important scientific breakthrough into something that will help deliver better tests, better drugs and above all better care for patients.... David Cameron, August 2014 (my emphasis)
- 22. (c) Stephen Senn 2016 22
- 23. 23 Genes, Means and Screens It will soon be possible for patients in clinical trials to undergo genetic tests to identify those individuals who will respond favourably to the drug candidate, based on their genotype…. This will translate into smaller, more effective clinical trials with corresponding cost savings and ultimately better treatment in general practice. … individual patients will be targeted with specific treatment and personalised dosing regimens to maximise efficacy and minimise pharmacokinetic problems and other side-effects. Sir Richard Sykes, FRS, 1997 My emphasis
- 24. Zombie statistics 1 Percentage of non-responders What the FDA says Where they got it Paving the way for personalized medicine, FDA Oct 2013 Spear, Heath-Chiozzi & Huff, Trends in Molecular Medicine, May 2001 (c) Stephen Senn 24(c) Stephen Senn 24
- 25. Zombie statistics 2 Where they got it Where those who got it got it Spear, Heath-Chiozzi & Huff, Trends in Molecular Medicine, May 2001 (c) Stephen Senn 25(c) Stephen Senn 25
- 26. The Real Truth • These are zombie statistics • They refuse to die • Not only is the FDA’s claim not right, it’s not even wrong • It’s impossible to establish what it might mean even if it were true (c) Stephen Senn 26
- 27. 88.2% of all statistics are made up on the spot Vic Reeves
- 28. The Pharmacogenomic Revolution? • Clinical trials – Cleaner signal – Non-responders eliminated • Treatment strategies – “Theranostics” • Markets – Lower volume – Higher price per patient day
- 29. Implicit Assumptions • Most variability seen in clinical trials is genetic – Furthermore it is not revealed in obvious phenotypes • Example: height and forced expiratory volume (FEV1) in one second • Height predicts FEV1 and height is partly genetically determined but you don’t need pharmacogenetics to measure height • We are going to be able to find it – Small number of genes responsible – Low (or no) interactive effects (genes act singly) – We will know where to look • We are going to be able to do something about it – May require high degree of dose flexibility • In fact we simply don’t know if most variation in clinical trials is due to individual response let alone genetic variability
- 31. Sources of Variation in Clinical Trials (c) Stephen Senn 31 Senn SJ. Individual Therapy: New Dawn or False Dawn. Drug Information Journal 2001;35(4):1479-1494. (c) Stephen Senn 31
- 32. Identifiability and Clinical Trials (c) Stephen Senn 32(c) Stephen Senn 32
- 33. In the Meantime • There is a massive source of unwanted variation • Doctors • Variation in practice is so large that it cannot be justified by variation in patients • This is the basic idea behind the way that Intermountain Health under the leadership of Brent James has been applying Deming’s principles to health care (c) Stephen Senn 33
- 34. (c) Stephen Senn 34
- 35. (c) Stephen Senn 35
- 36. “Guys, it’s more important that you do it the same way than what you think is the right way.” Brent James, Advice to doctors
- 37. (c) Stephen Senn 37(c) Stephen Senn 37(c) Stephen Senn 37
- 38. The supply of truth always greatly exceeds its demand John F Moffitt
- 39. In conclusion (c) Stephen Senn 2016 39 The motto of this great university is To Discover the Causes of Things Dichotomies, responder analysis and numbers needed to treat do not help in this task They make it more difficult They should be abandoned