Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009
Download as PPT, PDF2 likes1,268 views
The document discusses the critical importance of research integrity, highlighting issues like selective publication, reporting biases, and ethical obligations of authors and publishers. It underscores the potential harm of flawed research to patients and public health, urging transparency in reporting outcomes and conflicts of interest. The author calls for improved education and adherence to reporting guidelines to safeguard the reliability of scientific research.
![Published research - flawed, misleading, nefarious John Hoey University of Waterloo, 2009 www.slideshare.net/hoeyj [email_address] om](https://image.slidesharecdn.com/waterloohoey1sept09-090831163602-phpapp02/85/Published-Research-Flawed-Misleading-Nefarious-Use-of-Reporting-Guidelines-University-of-Waterloo-2009-1-320.jpg)
![Research Reporting Guidelines - Clinical Trials [Consort] 22 essential items/elements e.g Randomization Item # 8 How was the sequence generated? Item # 9 Allocation concealment Item # 10 Implementation](https://image.slidesharecdn.com/waterloohoey1sept09-090831163602-phpapp02/85/Published-Research-Flawed-Misleading-Nefarious-Use-of-Reporting-Guidelines-University-of-Waterloo-2009-12-320.jpg)
Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009
- 1. Published research - flawed, misleading, nefarious John Hoey University of Waterloo, 2009 www.slideshare.net/hoeyj [email_address] om
- 2. Why is research important - and what is it? Can we believe what we read in journals? What can be done?
- 3. Why is research important - and what is it? Public health practice Clinical practice Public policy Rational Evidence-based i.e. Research
- 4. Why is research important - and what is it? ++ research designs RCTs Health economics Genomics ... So far ~ 150 guidelines
- 5. 2,000 - 3,000 manuscripts /yr Lancet - 6,000 to 7,000 /yr
- 7. Maintaining the Integrity of the Scientific Record.......why? Distorts the scientific corpus misleads other investigtors Harm to patients and the public Physical harm Financial harm
- 8. It’s also ethically wrong Nuremberg - Helsinki “ Both authors and publishers have ethical obligations. In publication of the results of research, the investigators are obliged to preserve the accuracy of the results. Negative as well as positive results should be published or otherwise publicly available. Sources of funding, institutional affiliations and any possible conflicts of interest should be declared in the publication. Reports of experimentation not in accordance with the principles laid down in this Declaration should not be accepted for publication.”
- 9. Let’s look at an example Qualitative Economic Randomized Clinical Trials Systematic reviews/meta-analyses etc..
- 10. RCT R Rx A Placebo Outcome
- 11. Consort Statement Poor quality of reporting RCTs. Could not be trusted for use in clinical practice Could not be combined in systematic reviews Identified 22 essential items needed to conduct a RCT. Ann Intern Med. 2001;134:663-694. www.annals.org David Moher
- 12. Research Reporting Guidelines - Clinical Trials [Consort] 22 essential items/elements e.g Randomization Item # 8 How was the sequence generated? Item # 9 Allocation concealment Item # 10 Implementation
- 13. Does this really matter?
- 14. Miller AB, et al. Canadian National Breast Screening Study: 1. Breast cancer detection and death rates among women aged 40 to 49 years. CMAJ. 1992; 147(10): 1459–1476 Canadian trial of mammogrphy for breast cancer Begun in about 1980 50,000 asymptomatic women Randomly assigned to mammography usual care Followed for 8 years
- 15. Item # 9 Allocation concealment Item # 10 Implementation
- 16. Randomization in the Canadian National Breast Screening Study: a review for evidence of subversion . Bailar JC, MacMahon B. CMAJ 1997;156:193-9 Harms? Can science be trusted? Ethical responsibility to study subjects Today - still unclear if mammography is beneficial - harms to women, costs to society
- 17. Schulz et al - Quality of RCTS Cochrane Pregancy and childbirth data base 33 meta-analyses - 250 trials Quality of randomization and risk estimates Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methogological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273(5):408-12 (currently cited 1777 times in )
- 18. Randomization Concealment of treatment allocation schedule Generation of allocation sequences Inclusion of all trial subjects in analysis Double blinding (subject and investigator) Of 22 items - one example
- 19. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methogological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273(5):408-12 Quality of 250 trials Concealment % of trials
- 20. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methogological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273(5):408-12 Quality of 250 trials % Exaggeration of Risk Ratio % of trials
- 21. Study design - RCT R Rx A Placebo Outcome Primary Outcome -Specified in Protocol? or fishing expedition?
- 22. Selective publication - Outcome bias publishing the more interesting (usually positive) result Was there an hypothesis? A plan for analysis and reporting of data? In an RCT, this is the primary outcome
- 23. Selective publication - Outcome bias (publishing the more interesting result) 48 trials 1402 outcomes 31% - 59% incompletely reported (40% not reported at all) Chan, A.-W. et al. CMAJ 2004;171:735-740
- 24. Selective publication - Outcome bias (publishing the more interesting result) Interpretation: Intensive multitherapy for patients with poorly controlled type 2 diabetes is successful in helping patients meet most of the goals set by a national diabetes association. However, 6 months after intensive therapy stopped and patients returned to usual care the benefits had vanished, However, 6 months after intensive therapy stopped and patients returned to usual care the benefits had vanished.
- 26. Reporting Bias Kay Dickinson, Reporting and other biases in studies of Neurontin for migraine, psychiatric/bipolar disorders, nociceptive pain, and neuropathic pain . August, 2008 http://dida.library.ucsf.edu/pdf/oxx18r10
- 27. Some definitions Positive = Study shows results favourable to the intersts of the author/sponsor. Usually statistically significant. Negative = Study shows results unfavourable to the interest of the the author/sponsor - may or may not be statisticaly significant.
- 28. Reporting Biases Nefarious or Incompetent? Non-publication of negative or neutral results Selective publication of results - outcome bias Multiple publication bias Language bias - and publishing in the grey literature Time lag bias Undeclared conflicts of interest Ghost writing Dickinson
- 29. Neurontin (gabapentin) images from Wikipedia which also has a nice summary of court proceedings and results
- 30. P Wessely, C Baumgartner, D Klinger, J Kreczi, N … - Cephalalgia, 1987 Neurontin for migraine headaches study Bias Example Publication Final negative primary results not published, only positive preliminary results Selective outcome reporting Outcome reported was not primary or secondary outcome Selective statistical analyses 2 nonrandomized patients assigned to neurotin were include with those randomized Spin Emphasis on “positive” outcomes
- 31. 16 Citations P Wessely , C Baumgartner, D Klinger, J Kreczi, N … - Cephalalgia, 1987 Who cares? General Principles of Migraine Management: The Changing Role of Prevention E Loder, D Biondi - Headache: The Journal of Head and Face Pain, 2005 - Blackwell Synergy Preventive treatment of migraine - SD Silberstein - Trends in Pharmacolog ical Sciences, 2006 - Elsevier M igraine prevention DW Dodick, SD Silberstein - British Medical Journal, 2007 - pn.bmj.com Neuromodulator s for Migraine Prev ention R Kaniecki - Headache: The Journal of Head and Face Pain, 2008 - Blackwell Synergy Practice parameter: Evidence-based guidelines for migraine headache (an evidence-based review) Stephen D. Silberstein , MD, FACP, for the US Headache Consortium Neurology 2000;55:754-762
- 34. www.flickr.com/photos/dariustwin/3362416244/ the story of Dr. Reckless and the pharmaceutical company - Pfizer
- 35. “ Ghost authorship exists when someone has made substantial contributions to writing a manuscript and this role is not mentioned in the manuscript itself. WAME considers ghost authorship dishonest and unacceptable ” www.wame.org/resources/policies
- 36. Neuroptin for pain relief 59 sites in 6 countries. Study ends in 1999. Results negative (no benefit) By 2002, no manuscript produced. Dr. Reckless writes to Pfizer Ultimately not published as a stand alone paper, but incorporated in a paper with other studies. UK doc. (Dr. Reckless) tries to publish his research
- 37. Ghost Writing - the case of Dr. Reckless vs. Pfizer “ I don’t think we should be too hasty with this request,” • “ I agree with your answer. Although I would love to publish SOMETHING about 945- 224, Donna McVey made it very clear that we should take care not to publish anything that damages neurontin’s marketing success.” “ ..We would need to have ‘editorial’ control, but would certainly involve Dr. Reckless in the process, asking for his expert comment.” “ I think that we can limit the potential downsides of the 224 study by delaying the publication for as long as possible and also from where it is published. More importantly it will be more important to how WE write up the study. We are using a medical agency to put the paper together which we will show to Dr. Reckless. We are not allowing him to write it up himself.”
- 38. So what? http://dida.library.ucsf.edu/pdf/oxx18r10 In total, I find that there were 43 million off-label prescriptions of Neurontin as a result of the ... promotional activities related to the off-label uses ... that would not have occurred absent the challenged conduct. ” Meredith Rosenthal
- 39. what can be done? Education in research Skepticism ++ *Conflict of interest *Use Reporting guidelines
- 40. Conflict of Interest
- 41. 1109 Canadian clinicaltrials.gov 732 investigators 11 ideal practices to mitigate fCOI - control over design - data collection - analysis - interpretation -write up -authorship -decision to publish -etc. 6% of investigators met all ideal practices Rochon P, et al. - in peer review
- 43. Reporting Guidelines Consort - RCTs Prism- Systematic reviews Moose - Meta-analyses Stard - Diagnositic tests Grade - Clinical guidelines Strobe - Observational studies etc. (~160) www.equator-network.org
- 44. Reporting Guidelines Protocol Ethics Funding Write up Journal
- 45. to sum up Patient Care, Public Health & Policy Rationale and Evidence Based Much research is Unreliable, confused, distorted, conflicted So read cautiously and skeptically Be familiar with good research standards Use guidelines Teach others about them
- 46. Thank-you
Editor's Notes
- #13: sequence generation = i.e. ‘assign every other subject to placebo’ table of random numbers etc. Allocation concealment = did those assigning know the assignment destination (placebo, active group)? Implementation = specifically how was it done?
- #19: Quality
- #20: 33 meta-analyses in cochrane pregnancy and childbirth database - 250 trials. assessed for quality of reporting randomization and assignemnt - effect of RR Objective.-To determine if inadequate approaches to randomized controlled trial design and execution are associated with evidence of bias in estimating treatment effects. Design.-An observational study in which we assessed the methodological quality of 250 controlled trials from 33 meta-analyses and then analyzed, using multiple logistic regression models, the associations between those assessments and estimated treatment effects. Data Sources.-Meta-analyses from the Cochrane Pregnancy and Childbirth Database. Main Outcome Measures.-The associations between estimates of treatment effects and inadequate allocation concealment, exclusions after randomization, and lack of double-blinding. Results.-Compared with trials in which authors reported adequately concealed treatment allocation, trials in which concealment was either inadequate or unclear (did not report or incompletely reported a concealment approach) yielded larger estimates of treatment effects (P<.001). Odds ratios were exaggerated by 41% for inadequately concealed trials and by 30% for unclearly concealed trials (adjusted for other aspects of quality). Trials in which participants had been excluded after randomization did not yield larger estimates of effects, but that lack of association may be due to incomplete reporting. Trials that were not double-blind also yielded larger estimates of effects (P=.01), with odds ratios being exaggerated by 17%. Conclusions.-This study provides empirical evidence that inadequate methodological approaches in controlled trials, particularly those representing poor allocation concealment, are associated with bias. Readers of trial reports should be wary of these pitfalls, and investigators must improve their design, execution, and reporting of trials
- #21: 33 meta-analyses in cochrane pregnancy and childbirth database - 250 trials. assessed for quality of reporting randomization and assignemnt - effect of RR Objective.-To determine if inadequate approaches to randomized controlled trial design and execution are associated with evidence of bias in estimating treatment effects. Design.-An observational study in which we assessed the methodological quality of 250 controlled trials from 33 meta-analyses and then analyzed, using multiple logistic regression models, the associations between those assessments and estimated treatment effects. Data Sources.-Meta-analyses from the Cochrane Pregnancy and Childbirth Database. Main Outcome Measures.-The associations between estimates of treatment effects and inadequate allocation concealment, exclusions after randomization, and lack of double-blinding. Results.-Compared with trials in which authors reported adequately concealed treatment allocation, trials in which concealment was either inadequate or unclear (did not report or incompletely reported a concealment approach) yielded larger estimates of treatment effects (P<.001). Odds ratios were exaggerated by 41% for inadequately concealed trials and by 30% for unclearly concealed trials (adjusted for other aspects of quality). Trials in which participants had been excluded after randomization did not yield larger estimates of effects, but that lack of association may be due to incomplete reporting. Trials that were not double-blind also yielded larger estimates of effects (P=.01), with odds ratios being exaggerated by 17%. Conclusions.-This study provides empirical evidence that inadequate methodological approaches in controlled trials, particularly those representing poor allocation concealment, are associated with bias. Readers of trial reports should be wary of these pitfalls, and investigators must improve their design, execution, and reporting of trials
- #24: Chan et al looked at studies funded by the Canadian Institutes of Health Research during the late 1990’s and then tried to find published versions of these studies. They compared the funded protocol with the published papers. Despite careful literature searches and contacts with the funded researchers they were unable to find publications for 40% of the funded trials. (unreported). Of the reported trials between 31% (for efficacy outcomes) and 59% (for harm outcomes) were incompletely reported.
- #27: Reporting bias ought to concern editors. It means that published reports - whether published in peer reviewed journals, abstracts to meetings or in other grey literature (web sites, non-indexed literature, and so on) is biased, meaning that the estimates of results are not only wrong, they are biased from the truth. As editors we ought to be concerned. Our job is to spot this type of bias and ask authors to correct it before publication, or reject the paper. We are doing a bad job of this. Even journals with large editorial staffs have accepted papers for publication that contained only part of the evidence or evidence that was biased.
- #28: These meanings reflect current popular usage and I’ll use them here. Dickison makes these distinctions.
- #29: Most of what I used to see when editing a medical journal was probably incompetence or carelessness or both. I suspect that most journal editors have the same experience. The high-traffic journals get more manuscripts, so they can more easily just discard a lot. Smaller journals can’t. At least that was my experience. When I arrived at the Journal in 1996, CMAJ was publishing about 200 research articles a year. When I left we were publishing but 50. This allowed us to increase the rejection rate to about 90%, decrease our workload as editors, devote more time to articles and hopefully correct some of the incomeptence (and be aware of the very rare malfeasance), preform our function as editors more in line with the Helsinki guidelines, (and increase our impact factor). Fortunately, at the same time as we were doing this, online journals exploded in number, so papers we rejected had a pretty good chance of geting published elsewhere. Or so we hoped. Non-publication of negative results is self-evident. Selective publication of results usually involves non-publication results describing outcomes of the primary objectives in a RCT or substitution of secondary objectives and portraying them as primary. (Need to verify the protocol for RCTs.) Same situation for observational studies. Was there a primary hypothesis? What was it? Is it reported as the main result? Reported at all? Multiple publication of the same results increases citations, confuses and devalues meta-analyses and systematic reviews. Language bias referes to publishig negative results in literature that is more difficult to access and may be missed. (non-English, the grey literature - web pages, letters to editor, abstracts to meetings). Time lag bias - delay publication (a sister of non-publication). Undeclared conflicts of interest. Result in less careful scrutiny by editors, peer reviewers and readers. Should serve as a warnig sign to be attentive. Ghost writing - slide following Dickinsen lists several reporting biases and carefully documents them. We don’t have time to cover them all here. Most are familar to you all. The bias is almost always in favour of the positive outcome. There is a growing literature documeting the effect size of these biases. The effect sizes are substantial. Dickinson has a nice bibliography for those interested in a particular bias.
- #30: Approved by US FDA 1994 RX partial seizures By 2003, one of Pfizer’s best selling drugs for minor seizures. Off-label uses account for 90% of sales migraine, bipolar disorders, OCD, depression, insommnia, etc.. Adverse effects - dizziness, mood swings etc.. hepatotoxcity, depression, suicide, Court cases - for illegally marketing a drug based on no evidence of efficacy
- #31: Dickinson’s analysis of another paper that was used by Pfizer to promote off-label uses of Neurontin. Clearly this is a seriously flawed RCT, as published by an editor, I hope not one in this room.
- #32: Does it matter. Well yes. This misleadingly reported study is still being cited, the drug is still being recommended for prevention of migraine and worse, has found it’s way into guidelines that are Evidence Based, whatever that means in this context.
- #34: Group 2 are highly recommended based on RCT evidence.
- #36: “ Ghost authorship exists when someone has made substantial contributions to writing a manuscript and this role is not mentioned in the manuscript itself. WAME considers ghost authorship dishonest and unacceptable (emphasis mine) . Ghost authors generally work on behalf of companies, or agents acting for those companies, with a commercial interest in the topic, and this compounds the problem.” ( http://www.wame.org/resources/policies accessed August 1, 2008)
- #37: Dr. Reckless study Neuroptin for neruopathic pain UK Investigator multicentre study at 59 sites in 6 countries 1998-99 Results showed no benefit.April 2002 - Results not written up - Dr. Reckless complains to Pfizer:
- #38: Study 945-224 was a multicenter, placebo controlled trial, conducted at 59 sites in the UK, France, Germany, Italy, Spain, and 2 in South Africa, comparing three doses of Neurontin® for treatment of neuropathic pain. Statements in the informed consent document that those enrolling would benefit others by their participation were carefully made: “ Information gained in this study may eventually benefit other persons with painful diabetic neuropathy.” (RR 720-04130 p.259) The studied period was May 1998 to September 1999. No statistically significant differences were observed between any of the three Neurontin dosage groups and the placebo group for the primary endpoint (mean pain score). Several secondary outcomes were statistically significant, depending on the dosage group. On 18 April 2000, Dr. Reckless, a UK investigator in the 945-224 study, complained to the Clinical Trials Monitor for Parke-Davis about failure to publish the study.
- #39: http://dida.library.ucsf.edu/pdf/oxx18r10
- #41: http://dida.library.ucsf.edu/pdf/oxx18r10