CAHPS Survey Completion Rate and Patient Loyalty Metrics are Unaffected by Issues Screening Technology Used to Check Patient Wellbeing and Experience

ABSTRACT

Description

Issues Screening Technology (IST) enables the collection of prompt feedback to enhance patient experience and address concerns. This study assesses the influence of IST on three critical metrics: ED CAHPS survey completion rate, Net Promoter Score (NPS), and Likeliness to Refer (LTR).

 Objective

This study assessed whether emergency departments can implement IST without negatively affecting the ED CAHPS survey completion rate, NPS, and LTR.

 Processes

Two patient groups were compared. One group utilized IST, while the other group did not. The study evaluated outcome measures, including ED CAHPS completion rates, NPS, and LTR.

 Outcomes

Group 1 included 3,897 ED patients seen in August 2022. Of these, 5.1% completed the ED CAHPS survey, with a confidence interval of 4.6% - 5.6% (p < 0.05). During the same month, 94.2% of patients discharged home engaged with IST, and 2,414 responded - 29.5% via text message and 4.2% via email. Group 2 included 3,745 ED patients seen in September 2022. Of these, 4.8% completed the CAHPS surveys, with a confidence interval of 4.2% - 5.4% (p < 0.05). During this month, IST was paused.

 Conclusion

Hospitals can implement IST without negatively impacting the ED CAHPS survey completion rate or the LTR and NPS metrics.

 ARTICLE

Introduction

The Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey is valuable for collecting patient feedback, especially for external patient experience (PX) benchmarking.[1] Developed by the Agency for Healthcare Research and Quality, the survey encompasses 35 or more questions covering various aspects of patient experience, including nine demographic, seven visit-related, and nineteen PX performance-related inquiries. It also offers the option to add supplemental questions if needed.[2]

Patients can complete the survey via various channels, including US mail, email, and telephone, with a requirement of 450 annual responses for participation. The survey incorporates two patient loyalty metrics: the Net Promoter Score (NPS) and Likeliness to Refer (LTR). These metrics provide insights into the patient experience and the likelihood of recommending healthcare services to others.

The NPS is a standard instrument used across all business sectors to measure customer loyalty. It classifies customers as promoters (rating of 9 or 10), neutrals (rating of 7 or 8), or detractors (rating of 0 to 6).[3] The NPS is the percent promoters minus the percent detractors and ranges from +100 (all promoters) to -100 (all detractors). Neutrals are omitted. For example, 60% promoters, 30% neutrals, and 10% detractors result in an NPS of 50. The LTR represents the percentage of patients who responded, 'Definitely yes,' to 'Would you recommend this emergency room to your friends and family?'.

Nurse post-discharge callbacks uncover patient concerns and improve Press Ganey survey scores. EDs can analogously use issues screening technology (IST) to gather prompt patient feedback and address wellbeing and service concerns after an ambulatory encounter.[4] Thus, using IST has been described as the evolution of ED callbacks.[5] Also, the high rate of patient feedback generated by IST allows internal PX benchmarking of individuals.

Discussion

In its patient-centered approach, the Beryl Institute prioritizes patient wellbeing by prioritizing 'How are you doing?' before 'How did we do?'.[6] It stands to reason that clinical outcomes and satisfaction scores improve by resolving patient well-being and service issues.

CAHPS scores, reflecting patients' perception of quality, closely align with actual quality and are converted to publicly reported star ratings, influencing the patient selection of healthcare services and providers.[7] High CAHPS scores also impact Medicare reimbursement models positively. However, the low ED CAHPS sampling rate leads to excessive variability, causing EDs to change percentile quartiles up or down frequently.[8] This variability issue can be addressed by IST, which yields a much higher completion rate (30.3%) compared to ED CAHPS surveys (1.9%), providing accurate monthly PX performance data for physicians.[9]

The low ED CAHPS survey completion rate is primarily due to its length, which causes survey fatigue.[10] To combat low completion rates, surveys should be concise and relevant, contain a maximum of ten questions, and take no more than ten minutes to complete.[11] Redundant questions and sensitive topics like race, ethnicity, education level, and mental health may discourage patients from participating in ED CAHPS surveys.

Method

This study evaluated the influence of ED CAHPS surveys on IST directed at discharged patients. The main objective was to assess whether utilizing IST impacts the completion rate of ED CAHPS surveys and the resulting patient loyalty metrics, NPS, and LTR.

In 2022, we conducted surveys at our comprehensive suburban ED, which receives 78,849 annual visits, and our freestanding suburban ED, which receives 37,838 visits. We utilized Press Ganey to send ED CAHPS surveys to some discharged adult patients alongside IST to evaluate patient wellbeing and the experience of those discharged home. Table 1 compares the ED CAHPS survey and IST.

The IST involved SMS outreach in the morning, with non-respondents receiving an email one hour later. However, specific patient groups, such as those who left without being seen, had an active case management plan or had particular diagnoses (e.g., intoxication or psychosis), were excluded from the survey. The sequence of screen images below illustrates how the IST was designed to gather feedback on well-being (Figure 1a) and service issues (Figure 1b)  from patients in this study.

Two study groups of ED patients discharged home and receiving ED CAHPS surveys were identified. Group 1 consisted of patients seen in August 2022, during which IST was utilized, while Group 2 comprised patients seen in September 2022, when the IST was temporarily paused. To assess the adequacy of the sample size and determine if the two groups exhibited any significant differences in the completion rate of ED CAHPS surveys or the patient loyalty metrics, a reliable open-source calculator was employed, ensuring a 95% confidence level for the analysis. [12]

Results

In Group 1, which included 3,897 ED patients seen in August 2022, 5.1% completed the ED CAHPS surveys, with a confidence interval of 4.6% - 5.6% (p < 0.05). During the same month, 94.2% of patients discharged home engaged with IST, and over one-third responded - 29.5% via text message and 4.2% via email.

In Group 2, consisting of 3,745 ED patients seen in September 2022, 4.8% completed the CAHPS surveys, with a confidence interval of 4.2% - 5.4% (p < 0.05). During this month, the IST was intentionally paused. A summary of these results is presented in Table 2.

The ED CAHPS PX metrics, NPS, and LTR showed nearly identical results in both study groups. In September 2022, when the IST was paused, the number of patients per day decreased by 3.4%. However, due to staffing adjustments, the number of patients seen per physician/nurse per hour worked remained the same (2.2/0.9) as in August 2022.

The IST in this study attributes PX performance to individuals with the modified Net Promoter Score (mNPS). Patients rate their experience with each care team member on a 5-point Likert scale, with 'Very good' being the highest rating. The mNPS is calculated by subtracting the percentage of patients who rated their experience as 'Average,' 'Bad,' or 'Very bad' from the percentage of patients who rated their experience as 'Very Good.' The middle rating, 'Good,' is excluded. The mNPS ranges from +100 (when all patients rated their experience as 'Very good') to -100 (when no patients rated their experience as 'Very good' or 'Good'). 

A summary of these metrics is presented in Table 3.

In a 2017 study, Press Ganey stated that statistical validity requires at least 30 completed surveys.[13] However, in this study, the average monthly ED CAHPS completion rate was only six per physician and three per nurse. As a result, it takes physicians around five months and nurses around ten months to be individually benchmarked on patient experience (PX) performance. This reporting frequency needs to be shorter to manage low performers effectively.

In contrast, IST produces nearly seven times the patient engagement in ED CAHPS surveys, enabling us to benchmark physicians and nurses accurately every month. This reporting frequency is optimal as it allows for the timely identification of performance trends and opportunities for improvement.

Conclusion

The study's overlapping confidence intervals indicate that using IST does not impact the completion rate of the ED CAHPS survey. The global ED CAHPS patient experience metrics, NPS, and LTR, were nearly identical in both study groups.

The study findings indicate that hospitals can effectively utilize IST without negatively impacting the ED CAHPS survey completion rate or the NPS and LTR metrics.

Our study focused on two suburban EDs. Further research is necessary to assess the generalizability of these results to other types of EDs.

Additional research is required to establish a direct correlation between resolving patient wellbeing and service issues using IST and improving clinical outcomes and satisfaction scores.

References

[1] Tessler R, Sorra J. CAHPS Survey Administration: What We Know and Potential Research Questions. October 2017. Accessed August 30, 2023. https://www.ahrq.gov/sites/default/files/wysiwyg/cahps/about-cahps/research/survey-administration-literature-review.pdf.

[2] U.S. Centers for Medicare & Medicaid Services. Emergency Room Patient Survey. July 2020. Accessed August 30, 2023. https://www.cms.gov/files/document/ed-cahps-10-2-column-survey-english-july-2020.pdf.

[3] Wikipedia. Net promoter score. August 28, 2023. Accessed August 30, 2023. https://en.wikipedia.org/wiki/Net_promoter_score.

 [4] Scaletta, T., Rising, K., & Ranney, M. (2017). Case 14 - Automated Patient Follow-Up Program. In Value and quality innovations in acute and emergency care (pp. 170–177). essay, Cambridge University Press.

[5] Papa, A., & Lefton, C. (2015). Discharge texting: The evolution of ED callbacks. Journal of Emergency Nursing, 41(4), 345–346. https://doi.org/10.1016/j.jen.2015.04.002

[6] Wolf, J. (2019, December). How are YOU doing? A New Perspective for Experience Measurement. https://www.linkedin.com/pulse/how-you-doing-new-perspective-experience-measurement-jason-a-

[7] Doyle, C., Lennox, L., & Bell, D. (2013). A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ Open, 3(1). https://doi.org/10.1136/bmjopen-2012-001570

[8] Pines, J. M., Penninti, P., Alfaraj, S., Carlson, J. N., Colfer, O., Corbit, C. K., & Venkat, A. (2018). Measurement under the microscope: High variability and limited construct validity in emergency department patient-experience scores. Annals of Emergency Medicine, 71(5). https://doi.org/10.1016/j.annemergmed.2017.11.011

[9] Scaletta, T., Hare, E., & Sung Lee, C. (2019). A next-day, brief e-survey overcomes the excessive variability seen in CAHPS-style emergency department surveys so that individual physician performance can be assessed on a regular basis. https://pxjournal.org/cgi/viewcontent.cgi?article=1376&context=journal

[10] Porter, S. R., Whitcomb, M. E., & Weitzer, W. H. (2004). Multiple surveys of students and survey fatigue. New Directions for Institutional Research, 2004(121), 63–73. https://doi.org/10.1002/ir.101

[11] Jotform. (2020, May 21). How many questions should be include in an online survey? The Jotform Blog. https://www.jotform.com/blog/how-many-questions-should-a-survey-have/

[12] Calculator.net. (2023, May). Home. Calculator.net. https://www.calculator.net/sample-size-calculator.html

[13] Lang, S. C., Weygandt, P. L., Darling, T., Gravenor, S., Evans, J. J., Schmidt, M. J., & Gisondi, M. A. (2017). Measuring the correlation between emergency medicine resident and attending physician patient satisfaction scores using Press Ganey. AEM Education and Training, 1(3), 179–184. https://doi.org/10.1002/aet2.10039