Challenges in evaluating eHealth applications
9 likes1,237 views
There are several key challenges in evaluating eHealth interventions and applications. One major challenge is attrition, as adherence tends to be low for online interventions. High attrition needs to be accounted for using appropriate statistical methods and reporting "attrition curves" over time. Another challenge is controlling the control group, as similar online interventions may be accessible to both groups. Maintaining participant identity and preventing re-registration can also be issues. Other difficulties include intervention complexity, data quality, messages being identified as spam, and cybersquatting. Addressing these methodological challenges is important for rigorously evaluating eHealth technologies.
![Power Distance Index (P.D.I.) [Geert Hofstede] = measurement of how much a particular culture values and respects authority. Countries with a high P.D.I. generally value being more deferential towards authority, and thus not contradicting a superior.](https://image.slidesharecdn.com/twente-symposium20090528-090608103146-phpapp02/85/Challenges-in-evaluating-eHealth-applications-6-320.jpg)
![“ It saved me spiritually and psychologically,” she says. “ I could compare notes with patients around the world, and I have made friends through [this mailing list] that rival lifelong relationships.” Anecdotes suggest that (p2p) online services have an impact on patients…](https://image.slidesharecdn.com/twente-symposium20090528-090608103146-phpapp02/85/Challenges-in-evaluating-eHealth-applications-11-320.jpg)
Challenges in evaluating eHealth applications
- 1. Associate Professor Department of Health Policy, Management and Evaluation, University of Toronto; Canada Senior Scientist , Centre for Global eHealth Innovation, Division of Medical Decision Making and Health Care Research; Toronto General Research Institute of the UHN, Toronto General Hospital, Canada Challenges in evaluating eHealth applications Visiting Professor, Faculty of Behavioural Sciences University of Twente, The Netherlands Editor & Publisher, Journal of Medical Internet Research (www.JMIR.org) Keynote at “Supporting Health by Technology II”, Enschede, NL, May 28th, 2009 Gunther Eysenbach MD MPH Gunther Eysenbach MD MPH
- 4. The ethnic theory of plane crashes (Chapter in “Outliers” by Malcolm Gladwell)
- 5. “ The single most important variable in determining whether a plane crashes is not the plane, it's not the maintenance, it's not the weather, it's the culture the pilot comes from. Planes are flown safely when the pilot and co-pilot are in open and honest communication. And in cultures where it is difficult for a junior person to speak openly to a superior, you have lots of plane crashes.” (Source: Gladwell)
- 6. Power Distance Index (P.D.I.) [Geert Hofstede] = measurement of how much a particular culture values and respects authority. Countries with a high P.D.I. generally value being more deferential towards authority, and thus not contradicting a superior.
- 7. Analogies to Healthcare Pilot = health care professional Co-pilot = patient Power Distance = Inverse Patient Empowerment
- 8. Eysenbach. Random Research Rants (Blog) http://www.webcitation.org/5h5jkooUX
- 9. Why patient empowerment / patient involvement? Safety/Quality Trust Health Outcomes?
- 10. Community (listserv) Communication (email) Content (WWW) stress anxiety depression Public with peers (P2P) with health professionals (P2D) Private with family & friends Self- efficacy Doctor- patient- communication Virtual social support (weak ties) Realistic expectations satisfaction confusion Benefit pathway Harm pathway information Asking right questions Local social support (strong ties) Θ Increases Decreases Θ Θ Positive health outcomes knowledge Θ (?) Θ Shared decision making/ compliance Too much / low quality Tailored / high quality Θ Θ loneliness Θ Eysenbach G. The impact of the Internet on cancer outcomes. CA Cancer J Clin 2003; 53(6): 356-371 “ Internet” О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О +
- 11. “ It saved me spiritually and psychologically,” she says. “ I could compare notes with patients around the world, and I have made friends through [this mailing list] that rival lifelong relationships.” Anecdotes suggest that (p2p) online services have an impact on patients…
- 12. “ Until I discovered Internet support, my experience of parent support had been both positive and beneficial. The Internet changed all that. During a downturn in my daughter's health it was suggested to me to join the Internet community associated with her disorder and (…) I decided to give it a try. What followed has been an experience that added to my trauma beyond what words could express.” ..though not all of them are only positive…
- 13. Challenges in evaluating eHealth interventions
- 14. Meyer B, Berger T, Caspar F, Beevers CG, Andersson G, Weiss M Effectiveness of a Novel Integrative Online Treatment for Depression (Deprexis): Randomized Controlled Trial J Med Internet Res 2009;11(2):e15 URL: http://www.jmir.org/2009/2/e15/ The Gold Standard: Randomized Controlled Trial
- 15. Problem: Adherence => Attrition RCT open Eysenbach G The Law of Attrition J Med Internet Res 2005;7(1):e11 <URL: http://www.jmir.org/2005/1/e11/>
- 16. www.JMIR.org
- 17. “ Predictors of adherence included disease severity, treatment length, and chronicity. “ Christensen H, Griffiths KM, Farrer L Adherence in Internet Interventions for Anxiety and Depression J Med Internet Res 2009;11(2):e13 URL: http://www.jmir.org/2009/2/e13/
- 18. What drives / motivates consumers + patients? Healthy Acute Condition Chronic/Severe Condition Motivation
- 19. Ease of use Usability User-centered Design Other predictors for attrition / lack of adherence
- 20. Essential: Needs assessment, formative evaluation, usability testing (iterative & ongoing) Focus Groups Usability lab In-depth interviews with stakeholders
- 22. Gaps between patient and provider needs / expectations “ Patients are particularly likely to anticipate that shared records will be empowering (...).” “ Physicians, by contrast, are especially likely to anticipate that laboratory results will confuse patients and that shared records will make patients worry more. “ Expectations of Patients and Physicians Regarding Patient-Accessible Medical Records Stephen E Ross, MD, Jamie Todd, MS-IV, Laurie A Moore, MPH, Brenda L Beaty, MSPH, Loretta Wittevrongel, Chen-Tan Lin, MD J Med Internet Res 2005 (May 24); 7(2):e13
- 23. Gaps between patient and provider needs / expectations Credits: Selina Brudnicki & Claudette DeLenardo
- 24. Gaps between patient and provider needs / expectations Credits: Selina Brudnicki & Claudette DeLenardo
- 25. Ease of use People will not enter health information to a significant degree…
- 26. … (perhaps there are some exceptions)…
- 27. … rather, eHealth sites / PHR (or PHA platform) must be populated seamlessly and effortlessly… Web 2.0 (collaborative, data entered by others) Mobile technologies, SMS Domotics, Ambient, pervasive computing, Intelligent car Applications with geospatial awareness Electronic Medical Record (Provider) PHR / PHA Platform Natural speech interfaces Personal Monitoring Tools
- 28. Sorbi MJ, Mak SB, Houtveen JH, Kleiboer AM, van Doornen LJP Mobile Web-Based Monitoring and Coaching: Feasibility in Chronic Migraine J Med Internet Res 2007;9(5):e38 <URL: http://www.jmir.org/2007/5/e38/> Health issues do not only occur at home - mobile interfaces are essential
- 32. But even ubiquitous computing / sensors is not always appropriate and accepted
- 33. “ I wouldn't want to track (a variable or in general) because tracking would… “ Not apply to me: (eg, smoking, alcohol drinking, pets) Not provide new information: (ie, “I already know this”) Not provide valuable information Provide too much information (information overload) Threaten self-image (“would feel criticized”) Not provide actionable information Lead to social conflict Promote obsessive or unhealthy reactions: (“becoming obsessed”) Force too much structure (“Approaching life too analytically”) Not be suitable for particular activity or behavior Be too complicated, error-prone, or disruptive Beaudin JS, Intille SS, Morris ME To Track or Not to Track: User Reactions to Concepts in Longitudinal Health Monitoring J Med Internet Res 2006;8(4):e29 <URL: http://www.jmir.org/2006/4/e29/>
- 34. Communities / peer-pressure as a tool to enhance adherence
- 35. Social Uses of Personal Health Information Within PatientsLikeMe, an Online Patient Community: What Can Happen When Patients Have Access to One Another’s Data Jeana H Frost, Michael P. Massagli J Med Internet Res 2008 (May 27); 10(3):e15
- 37. Medicine 2.0 (“next generation medicine”) From: Gunther Eysenbach. Medicine 2.0: Social Networking, Collaboration, Participation, Apomediation, and Openness J Med Internet Res 2008; 10(3):e22 http://dx.doi.org/ 10.2196/jmir.1030 DOI: 10.2196/jmir.1030 Gunther Eysenbach MD, MPH , www.medicine20congress.com Consumer / Patient Health Professionals Biomedical Researchers Science 2.0 Peer-review 2.0 Personal Health Record 2.0 Virtual Communities (peer-to-peer) Professional Communities (peer-to-peer) Health 2.0 HealthVault Google Health HealthBook Sermo WebCite CiteULike Medting WiserWiki eDoctr BioWizard Dissect Medicine E-learning PLoS One BMC JMIR Wikis Blogs RSS RDF, Semantic Web Virtual Worlds Web 2.0 Technologies & Approaches Apomediation Participation Social Networking Collaboration XML AJAX Openess Revolution Health PatientsLikeMe PeerClip Connotea ALIVE HealthMap caBIG Doctorshangout.com Asklepios
- 38. Gunther Eysenbach MD, MPH , www.medicine20congress.com September 17-18, 2009
- 39. Venkatesh,Viswanath;Morris,Michael G.;Davis,Gordon B.;Davis,Fred D., “User acceptance of information technology: Toward a unified view”, MIS Quarterly, 2003, 27, 3, 425-478.
- 40. Teresa Chiu, PhD Thesis, University of Toronto
- 41. Reporting Attrition/Adherence Attrition is the norm, not the exception - don’t try to cover it up “ Attrition curves” now a standard for reporting website use over time (at least at the J Med Internet Res ) Image source: Meyer B, Berger T, Caspar F, Beevers CG, Andersson G, Weiss M Effectiveness of a Novel Integrative Online Treatment for Depression (Deprexis): Randomized Controlled Trial J Med Internet Res 2009;11(2):e15 URL: http://www.jmir.org/2009/2/e15/
- 42. Dealing with Attrition/Adherence Use appropriate statistical methods for dealing with missing data Refrain from LOCF (last observation carried forward) Use models instead (ANOVA etc)
- 43. Challenge 2: The problem of “controlling” the control group On the Internet, similar interventions may be accessible for the control group (e.g. smoking cessation) difficult to “control” what the control group does privacy / ethical concerns limit the amount of data which can be gathered (e.g. logging all accessed URLs) Eysenbach G Issues in evaluating health websites in an Internet-based randomized controlled trial J Med Internet Res 2002;4(3):e17 <URL: http://www.jmir.org/2002/3/e17/>
- 44. Challenge 3: Re-registration, participant identity re-registration of (anonymous) participants eager to get the intervention “ neither requesting optional provision of offline contact details, nor monitoring IP addresses satisfactorily addressed the issue” Murray E, Khadjesari Z, White IR, Kalaitzaki E, Godfrey C, McCambridge J, Thompson SG, Wallace P Methodological Challenges in Online Trials J Med Internet Res 2009;11(1):e9 URL: http://www.jmir.org/2009/2/e9/
- 45. Other challenges Complexity of interventions Data quality Spam (reminder emails mistaken for spam) Cybersquatting Murray E, Khadjesari Z, White IR, Kalaitzaki E, Godfrey C, McCambridge J, Thompson SG, Wallace P Methodological Challenges in Online Trials J Med Internet Res 2009;11(1):e9 URL: http://www.jmir.org/2009/2/e9/
- 46. Thank you!