Challenges of Summative Usability Testing in a Community Hospital Environment HFES15
1 like405 views
The document discusses the challenges of electronic health record (EHR) usability testing in a community hospital setting, emphasizing how current EHR systems often do not align with clinical workflows, leading to increased cognitive loads and potential threats to patient safety. It identifies common usability myths and presents findings on the difficulties physicians face while using EHRs, such as disorganized pick lists and excessive clicks needed to perform tasks. The document concludes by outlining barriers to effective usability testing and advocating for policies that emphasize user-centered design to improve EHR functionality and support healthcare providers.
![Complaint Fatigue
“None of the pain points developed in that meeting have been fixed. I
came into work on my day off to attend this meeting (unpaid time) just
for the hope of improving my work environment and improving my
ability to rapidly and effectively service mine and [Hospital’s] patients. I
will not be attending any more meetings on improving [EHR Product]”
“End -users (physicians) have been IGNORED when tweaks requested.
After implementation of system; the company providing the EHR
assumed that if it worked, no matter how clunky, it was good enough”
“The current system has many clear problems, but it seems that the
priority is preparing for meaningful use rather than fixing the day to day
problems that plague physician work flow.”](https://image.slidesharecdn.com/challengesofsummativeusabilitytestinghfes15shared-150510221632-lva1-app6891/85/Challenges-of-Summative-Usability-Testing-in-a-Community-Hospital-Environment-HFES15-25-320.jpg)
Challenges of Summative Usability Testing in a Community Hospital Environment HFES15
- 1. Can Physicians Be Empowered Consumers: Challenges of EHR Usability Testing in a Community Hospital Setting David M. Schlossman, M.D., Ph.D., FACP, M.S. (Informatics), CPHIMS Boone Hospital Center
- 2. Creative Commons License This presentation is shared under a Creative Commons Attributional-Non Commercial- No Derivatives License 4.0 http://creativecommons.org/li censes/by-nc-nd/4.0/legalcode
- 3. Effect of ARRA on Physician Adoption of EHRs Hsiao C-J, Hing E. (2014). Use and characteristics of electronic health record systems among office-based physician practices: United States, 2001–2013. NCHS data brief, no 143. Hyattsville, MD: National Center for Health Statistics.
- 4. Medical Economics/MPI Provider survey Verdon, D.R. (2014). Physician outcry on EHR functionality, cost will shake the health information technology sector. Medical Economics 91(3): 18-27
- 5. Healthcare Providers’ User Experience • EHRs not developed with clinical workflow in mind • Information not formatted to fit physician cognitive models or support clinical decision making • Entering structured data distracts physician’s attention from the patient • Structured data unable to adequately represent the complex nuanced details of patient history or clinician reasoning • Increased cognitive load and decreased situational awareness.
- 6. EHR Usability Affects Patient Safety • Designed and applied inappropriately, health IT can add an additional layer of complexity to the already complex delivery of health care, which can lead to unintended adverse consequences… • The committee believes poor user-interface design, poor workflow, and complex data interfaces are threats to patient safety.
- 7. Usability Myth #1 • Clinicians are uncomfortable with technology • Clinicians won’t do the training and hard work necessary to become proficient with EHRs • Clinicians are unwilling to change their practices in order to improve care quality and cost efficiency. Staggers N, Xiao Y, Chapman L. (2013). Debunking health IT usability myths. Appl. Clin. Inf. 4: 241-250. http://dx.doi.org/10.4338/ACI-03-IE-0016
- 8. Reality: Clinicians Want the Right Technology • Clinicians love technology but hate IT that slows down their work • Current generation EHRs often don’t fit the way physicians think and work – Can’t get a “big picture” of the patient – No value returned for time as a data entry clerk – Too much information, too poorly organized – Can’t see trends over time or across categories – Have to integrate information across screens – Too many clicks, system delays, ambiguous terminology – The most important information is hard to find – Lack of integrated systems even in one facility – Attention distracted from the patient
- 9. Usability Myth #2 Usability is all just subjective anyway Staggers N, Xiao Y, Chapman L. (2013). Debunking health IT usability myths. Appl. Clin. Inf. 4: 241-250. http://dx.doi.org/10.4338/ACI-03-IE-0016
- 10. Reality: Usability Can Be Defined International Organization for Standardization: Usability is the effectiveness, efficiency, and satisfaction with which specific users can achieve a specific set of tasks in a particular environment. ISO 9241-11 Schoeffel, R. (2003). ISO Bull 34: 6-7
- 11. Usability Involves Lightening “Loads” 10 8 6 4 2 0 • Navigation – Clicks, scrolls, keystrokes, mouse movements • Reading – Legibility, signal to noise ratio, layout, emphasis, eye tracking • Thinking – Icon meaning, recall vs. recognition, cognitive load • Text Entry – Typing, pick lists, dictation • Emotional Factors – Task stress, situational awareness, dissonance
- 12. User Interface Design Heuristics 1. System status visibility – What’s going on… 2. System matches real world – Natural, logical, clear 3. User control & freedom – Cancel, undo, redo 4. Consistency & standards – Words, actions, locations 5. Error prevention – Avoid errors, confirm before committing 6. Recognition, not recall – Objects, actions visible 7. Flexibility and efficiency – Accelerators, personalization 8. Aesthetic, minimalist design – Only relevant, commonly needed information 9. Help users recognize, diagnose, recover from errors – Clear error messages with constructive solutions 10. System help and documentation – Easy to search, concise, context sensitive, step-by-step Nielsen, J. (1995). http://www.nngroup.com/articles/ten-usability-heuristics/
- 13. Attributes of EHR Usability: HIMSS • Simplicity: Lack of visual clutter • Naturalness: Map tasks to expectations • Cognitive Load: Data fits the task at hand • Efficient Interactions: Minimal steps per task • Forgiveness: Reversibility and data protection • Feedback: Information about actions taken • Language: Clear, unambiguous, targeted • Presentation: Density, color, readability, consistency • Context: Visual focus, WYSIWYG Belden, J., Grayson, R., Barnes, J. et al. (2009). Defining and Testing EHR Usability: Principles and Proposed methods of EHR evaluation and Rating. Report of the HIMSS EHR Usability Task Force.
- 14. Usability Can Be Measured Qualitative Inquiry • Field studies • Surveys • User focus groups Inspection • Expert evaluations • Reviews based on validated principles • Heuristic checklists Testing • Think aloud • Scenario based simulations • Testing lab vs. real world • Usability testing software –MORAE –TURF • Performance –Clicks, keystrokes, timings –Task success and failure –Appearance and verbalizations
- 15. Can Physicians Learn to Measure Usability? • Demonstrate EHR usability problems in a more objective, quantitative, actionable manner • Identify and correct EHR problems that can lead to user errors • Develop semi-automated testing systems to make usability evaluation accessible to busy clinicians who are not experts in this area • Assess and overcome barriers to summative scenario-based EHR usability testing in a private practice community hospital setting
- 16. Can Usability Testing Advance Other Goals? • Open dialogues between physicians and software developers to reach consensus on measuring usability and on best practices for the application of user-centered design principles • Contribute to the development of EHRs that provide better workflow and cognitive task support • Inform the purchasing decisions of physicians and healthcare organizations and help them measure progress in improving usability
- 17. Usability Testing in a Community Hospital? Boone Hospital Center Physician IT Resource Center NISTIR 7741: Guide to the Process Approach for Improving the Usability of EHRs
- 18. Acknowledgements Challenge Number 1: Lack of Resources
- 19. Usability Testing Methods •Create scenarios •Define the tasks –Define success criteria –Define optimal paths •Define markers •Define surveys •Construct measurement configuration
- 20. Next Challenges Challenge Number 3: Data Privacy and Security Challenge Number 2: Time and effort
- 21. Testing Scenario Paul Usabilitymd is a 71 year old man you follow for type 2 diabetes, hypertension, hyperlipidemia, and history of coronary artery disease status post stent placement. He presents to your office complaining of cough, shortness of breath and production of green sputum. He has a temperature of 102.3° F, pulse of 125 bpm, oxygen saturation of 84% on room air, and dullness to percussion and absent breath sounds in a right lower lobe distribution, and a swollen left leg. You decide to admit the patient to the hospital for further treatment, and he needs admission orders.
- 22. Task List • Open the EHR, find and open patient chart, and open the CPOE System • Enter admission status, code status, primary and secondary diagnoses • Enter condition, allergies, diet and vital signs orders, and MD notification orders • Enter admission laboratory orders including at least CBC, CMP, ABG, blood cultures, urinalysis and culture, and PT/PTT • Enter orders for imaging studies including at least a Chest X-ray and a Chest CT Scan with PE Protocol • Enter orders for pulmonary medicine and other indicated consultations • Enter home medications to be continued (list provided) • Enter antibiotic regimen and other necessary treatment orders • Save the orders and exit the CPOE System
- 23. System Recording with MORAE • Time for each task and subtask • Success rate for each task • Deviations from optimal path • Errors during each task • Use of the CDS in the EHR • Usefulness of the CDS in the EHR • Perceived effort of each task set • Overall satisfaction with the EHR
- 24. If You Build it, Will They Come? Challenge Number 4: Recruiting
- 25. Complaint Fatigue “None of the pain points developed in that meeting have been fixed. I came into work on my day off to attend this meeting (unpaid time) just for the hope of improving my work environment and improving my ability to rapidly and effectively service mine and [Hospital’s] patients. I will not be attending any more meetings on improving [EHR Product]” “End -users (physicians) have been IGNORED when tweaks requested. After implementation of system; the company providing the EHR assumed that if it worked, no matter how clunky, it was good enough” “The current system has many clear problems, but it seems that the priority is preparing for meaningful use rather than fixing the day to day problems that plague physician work flow.”
- 26. Participants 10 physicians (9M, 1F) from the BHC Medical Staff • Average age 55.7 years • Time in current position 21.7 years • Number of EHRs used 3.4 • Experience with test EHR 6.8 years • Self rated facility with computers: intermediate to advanced • Specialties: medical oncology, general surgery, nephrology, emergency medicine, family medicine, hospital medicine
- 27. Findings: Order Sets Do Not Hold Place 1. After linking out of a main order set and entering details of an order on a sub-form, the user is returned to the very top of the order set
- 28. Findings : Order Sets Do Not Hold Place
- 29. Findings : Order Sets Do Not Hold Place
- 30. Findings : Order Sets Do Not Hold Place
- 31. Prebuilt Order Sets Were Poorly Utilized • 0/10 used the standard Pneumonia (non-ICU) admission order set • 4/10 utilized any prebuilt admission order set (2 General Medical, 1 General Surgery, and 1 ED Transition orders) • 1/10 utilized embedded selection logic to order guideline based antibiotics for community acquired pneumonia (CAP) • 5/10 ordered correct guideline based antibiotics without help • 1/10 accessed the hyperlinked reference material about community acquired pneumonia and immediately exited that system finding it too complex and verbose
- 32. Findings: Disorganized Pick Lists • Over 150 choices displayed on 3 screens in alphabetical but not logical order • Most appropriate choice, community acquired pneumonia, is obscured • Many users chose a less appropriate option just to get on with the workflow • Subsequent aggregation of such structured data could lead to errors • Similar problems occur with entering medication and radiology orders.
- 33. Findings: Six Clicks to Enter an Allergy
- 34. Findings: Six Clicks to Enter an Allergy
- 35. Findings: Six Clicks to Enter an Allergy
- 36. Findings: Six Clicks to Enter an Allergy Hives
- 37. Findings: Six Clicks to Enter an Allergy
- 38. Findings: Six Clicks to Enter an Allergy
- 39. Findings: Six Clicks to Enter an Allergy
- 40. Confusing Order Review Screen One physician commented “This is the part I hate because it’s so busy. It’s very hard to look and see if you’ve got everything.” Another said “Totally useless. There’s no way you’re going to pick up a mistake in that. It’s not laid out in any logical order.”
- 41. System Usability Scale The average System Usability Scale (SUS) for working with this CPOE system was 25.5 ± 14.16, indicating very low user satisfaction (the average SUS for many hundreds of systems tested in the literature is approximately 68 with 0 being the worst and 100 being the best).
- 42. Barriers to Educating and Empowering Physician EHR Users 1.Financial Challenges a)Few resources for purchasing computer hardware and software b)Inability to compensate participants hindered recruiting c)Lost professional time creating protocol components and test patient data, conducting the testing, and analyzing the data
- 43. Barriers to Educating and Empowering Physician EHR Users 2. Technical Challenges a) HIPAA compliance: data privacy and security required use of the Training rather than the Production EHR environment b) Inserting test patient data into the EHR electronically was not possible, requiring cumbersome manual workarounds c) The responsiveness of the Training environment did not perfectly reflect that of the Production environment, d) Occasional system “freezes” with no feedback to the user about system status impacted user performance
- 44. Barriers to Educating and Empowering Physician EHR Users 3. Sociocultural Challenges a) The high level of complaints about EHR and CPOE suggested Medical Staff would welcome such research b) Despite broadcast emails, presentation at Medical Staff meeting, and personal appeals to multiple physicians with special interest in IT, recruiting physicians to participate in the study was extremely difficult c) The major recruiting challenge was a pervasive belief, based on 8 years experience, that rational arguments about IT best practices would not influence IT policy or functioning at BHC d) Multiple other professional, financial, and administrative factors competed for physician time i. Decreased efficiency following the implementation of health IT (fewer patients per hour, longer workdays) ii. Declining reimbursement rates and adapting to new physician compensation models (ACOs, value based payments) iii. Increased regulatory burden (PQRS, Meaningful Use) iv. Keeping current and providing best care in an era of explosive growth in the biomedical knowledge base
- 45. Study Limitations • Clinician bias of the principal investigator • Low participation, single institution, and focus on one functionality of a particularly outdated, poorly functioning EHR • Despite limitations, a consistent set of usability deficiencies, affecting all users, was rapidly identified • How could a user centered design process with clinician input have missed so many commonly identified problems?
- 46. Policy Implications • There is a factual basis for the my colleagues’ belief that policymakers and regulators are unreceptive to physician attempts at constructive engagement. • Stage 1 MU was highly successful in overcoming the adoption barrier. • Experience from Stage 2 MU suggests that emphasizing burdensome data collection processes, which detract from improving software design and productivity without clear benefits to care quality and safety is not the best strategy • The NPRM for Stage 3 continues to emphasize the same rigid, prescriptive, process oriented measures.
- 47. Usability Arises From User Centered Design (UCD) © 2013, Matthew B. Weinger MD, Russ Beebe, and Vanderbuilt University, All Rights Reserved
- 48. Improving Usability Is a Shared Responsibility “The clinical systems of today are great advances from what were available a decade ago but are still imperfect. Progress depends on further research, a vibrant vendor community that collaborates well with academia to enhance features such as interoperability and usability, and highly trained applied informaticians, many of whom are also practicing clinicians.” Detmer, D.E. and Shortliffe, E.H. (2014). Clinical Informatics: Prospects for a New Medical Subspecialty. Journal of the American Medical Association 311 (20): 2067-2068 doi: 10.1001/jama.2014.3514.
- 49. Solutions? • Resources, such as clinically plausible test scenarios and task lists, test patient data, validated best practices in user centered design, and compensation for lost professional time are needed to attract clinicians to participate in summative, and even more importantly formative, usability testing • Policy innovations that refocus Meaningful Use incentives on value and outcomes rather than processes, and provide the support and time needed for software developers and clinicians to engage in robust UCD, based on common real world use cases, and leading to intuitive health IT which improves clinical workflow efficiency and decreases cognitive load. • Open platform architectures with publicly accessible APIs for healthcare (FHIR, ReSTful APIs)