Evaluating the performance of artificial intelligence-based speech recognition for clinical documentation: a systematic review

Abstract

Background

Clinical documentation is vital for effective communication, legal accountability and the continuity of care in healthcare. Traditional documentation methods, such as manual transcription, are time-consuming, prone to errors and contribute to clinician burnout. AI-driven transcription systems utilizing automatic speech recognition (ASR) and natural language processing (NLP) aim to automate and enhance the accuracy and efficiency of clinical documentation. However, the performance of these systems varies significantly across clinical settings, necessitating a systematic review of the published studies.

Methods

A comprehensive search of MEDLINE, Embase, and the Cochrane Library identified studies evaluating AI transcription tools in clinical settings, covering all records up to February 16, 2025. Inclusion criteria encompassed studies involving clinicians using AI-based transcription software, reporting outcomes such as accuracy (e.g., Word Error Rate), time efficiency and user satisfaction. Data were extracted systematically, and study quality was assessed using the QUADAS-2 tool. Due to heterogeneity in study designs and outcomes, a narrative synthesis was performed, with key findings and commonalities reported.

Results

Twenty-nine studies met the inclusion criteria. Reported word error rates ranged widely, from 0.087 in controlled dictation settings to over 50% in conversational or multi-speaker scenarios. F1 scores spanned 0.416 to 0.856, reflecting variability in accuracy. Although some studies highlighted reductions in documentation time and improvements in note completeness, others noted increased editing burdens, inconsistent cost-effectiveness and persistent errors with specialized terminology or accented speech. Recent LLM-based approaches offered automated summarization features, yet often required human review to ensure clinical safety.

Conclusions

AI-based transcription systems show potential to improve clinical documentation but face challenges in accuracy, adaptability and workflow integration. Refinements in domain-specific training, real-time error correction and interoperability with electronic health records are critical for their effective adoption in clinical practice. Future research should also focus on next-generation “digital scribes” incorporating LLM-driven summarization and repurposing of text.

Clinical trial number

Not applicable.

Background

Clinical documentation, defined as the systematic recording of a patient’s medical history, diagnoses, treatment plans and care provided, remains a cornerstone of effective healthcare. It is critical in ensuring accurate communication among healthcare providers, legal accountability, and continuity of care [1]. However, traditional documenting methods, such as manual note-taking or transcription, are often labour-intensive, prone to errors and can detract from the quality of patient-clinician interactions [2, 3]. These inefficiencies not only contribute to clinician burnout but also risk compromising the accuracy of medical records and patient safety.

In recent years, Artificial Intelligence (AI) has begun to transform clinical documentation through the use of advanced technologies like automatic speech recognition (ASR), large language models (LLM) and natural language processing (NLP) [4, 5]. These AI-driven transcription tools automate the process of converting spoken language into structured electronic medical records (EMRs), thereby alleviating the burden of manual data entry [5]. By streamlining this process, AI transcription systems offer the potential to improve the accuracy and completeness of clinical documentation while allowing clinicians to focus more on patient care and communication.

Despite the promise of AI in this domain, the effectiveness of AI transcription tools remains inconsistent across different clinical settings [4]. Studies report varying levels of accuracy, time savings and user satisfaction [4, 5]. While some tools demonstrate significant improvements in documentation speed and precision, others face challenges with speech recognition (SR) errors, the need for manual post-editing and inconsistencies in real-world clinical use [4, 5]. These mixed outcomes highlight the complexity of integrating AI tools into diverse healthcare environments and underscore the need for a thorough evaluation of their performance.

This review aims to synthesize the current evidence on AI transcription tools, focusing on their accuracy, efficiency and usability in clinical practice. By examining the successes and challenges of implementing these technologies, the review seeks to provide insights that can guide the development and integration of AI-driven documentation systems, ultimately shaping the future of clinical workflows and improving the quality of patient care.

Methods

Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [6] for identifying, selecting and synthesizing evidence, the protocol for this review was developed and registered in PROSPERO (registration number CRD42024597200).

Search strategy

A comprehensive literature search was performed on February 16, 2025, and it was conducted across multiple electronic databases, including MEDLINE (via OVID), Embase and the Cochrane Library, covering all records published up to February 16, 2025. The search strategy was developed in consultation with the help of medical information experts to identify studies that evaluated the performance of AI-based medical transcription software in clinical settings. Key words and the following Medical Subject Headings (MeSH) were applied: “Artificial Intelligence”, “Digital Scribe”, “Medical Transcription”, “Speech Recognition”, “Natural Language Processing”, “Electronic Health Records” and “Clinical Documentation”. Details of the search strategy can be found in the Supplementary Material (Table S1).

Additionally, grey literature was searched via Google search engine to capture relevant non-peer-reviewed studies. To further enhance the comprehensiveness of the search, forward and backward searching were performed on the reference lists of relevant studies to identify additional literature that may not have been captured in the initial search.

Inclusion and exclusion criteria

The inclusion criteria for this systematic review were as follows: the population of interest included studies that involved clinicians, such as physicians and nurses, who used AI-based transcription software for clinical documentation. The intervention of focus was the use of AI-driven transcription tools, which may include technologies like ASR, LLM, and NLP systems. Eligible studies must report on one or more key outcomes, such as transcription accuracy (measured through Word Error Rate or WER), time savings, clinician satisfaction or the impact on patient care. The review included empirical studies of various designs, including randomized controlled trials (RCTs), cohort studies, cross-sectional studies, comparative evaluations and proof-of-concept studies. Only studies published in English or with an English translation, and indexed up to February 16, 2025, were considered.

Studies that did not involve AI-based transcription tools were excluded. This included studies conducted in clinical settings which did not involve a physician facing a patient, such as laboratory-based evaluations, reports generated by radiologists and/or pathologists, as well as those focusing on non-English language transcription. Additionally, conference abstracts, editorials, commentaries and opinion pieces that did not provide empirical data were excluded as well.

Study selection

All identified studies were imported into Covidence (Veritas Health Innovation, Melbourne, Australia) to facilitate the screening process. Four independent reviewers (J.J.W.N., E.W., C.X.L.G. and G.Z.N.S.) screened titles and abstracts to exclude studies that did not meet the inclusion criteria. Studies passing this initial screening underwent a full-text review by two independent reviewers (J.J.W.N. and E.W.). Discrepancies in study inclusion were resolved through discussion, with arbitration by a third, senior reviewer (Q.X.N., H.K.T. or S.S.N.G.) if necessary.

Data extraction

Data were extracted from each included study using a standardized data extraction form developed for this review. The extracted data encompassed study characteristics, including the software or model used, the type of AI model, study design, clinical setting and country in which the study was conducted. Additionally, details about the study population, sample size, whether the study was vendor-initiated, the reference standard and the comparator type were also recorded. Key performance metrics, such as F1 score, precision, recall, and WER, along with paper-specific outcomes related to AI transcription proficiency and key findings, were also included. Novel features of the AI transcription tools were documented to provide a comprehensive overview of the studies.

Assessment of risk of bias and study quality

The Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool [7] was used to systematically assess the risk of bias and applicability of the studies included in our review. QUADAS-2 is a widely used tool that evaluates risk of bias across four key domains: patient selection, index test, reference standard and flow and timing [7]. Each domain was independently assessed for potential bias by reviewing key study characteristics and determining if the conduct or interpretation of results could have introduced bias. We also evaluated whether the applicability of each study matched our review question.

For each domain, the risk of bias was rated as either “low,” “high,” or “unclear,” depending on the completeness and clarity of the study’s reported methods. Specifically, we looked at factors such as the selection of patients or datasets, how the index test was performed and interpreted, whether the reference standard was appropriate, and if there were any exclusions that could have influenced the results. Any discrepancies were resolved through discussion among the reviewers (E.W., J.J.W.N. and X.Z.). This careful assessment ensured that the studies included in the analysis were reliable and applicable to our research objectives.

Data synthesis

Given that a meta-analysis was not feasible due to anticipated heterogeneity in the study design, interventions and outcomes, a narrative synthesis was conducted, as guided by Popay et al. [8]. Findings were narratively synthesized by summarizing key outcomes such as transcription accuracy, clinician satisfaction, impact on patient care, and usability, and by identifying common patterns across the studies.

Results

Literature retrieval

A total of 5,244 records were initially identified through database searches. After removing 1,011 duplicates using Covidence, 4,233 studies were screened based on titles and abstracts. During this screening phase, 4,210 studies were excluded for not meeting the inclusion criteria, leaving 60 studies for full-text review. All of these studies were retrieved for detailed assessment. After applying the inclusion and exclusion criteria, 25 studies were included. As illustrated in Fig. 1, an additional four studies were identified through the forward and backward citation searching, bringing the final total to 29 studies. The key study characteristics and findings of the 29 studies [4, 9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36] are summarised in Tables 1 and 2, respectively.

Fig. 1

PRISMA flowchart showing the study selection process

Study results

Table 1 provides an overview of each study’s design, setting, participant information, AI transcription tools and indication of vendor involvement (or not). Study designs ranged from RCTs [11, 16] to comparative or observational studies [9, 10, 12, 20, 21], with a growing number of recent publications employing qualitative or pre-post approaches to capture both performance metrics and user perspectives [23, 28, 33,34,35,36]. These studies spanned diverse environments including emergency departments (EDs), inpatient wards, specialty outpatient clinics (e.g., gastroenterology, urology, dermatology) and simulated clinical scenarios.

Key findings

Accuracy and error rates

While some systems demonstrate impressive precision and recall Happe et al. achieved a precision of 0.73 and recall of 0.90 in a specialized vocabulary environment [10] and Suominen et al. reached F1 scores of up to 0.856 for nursing tasks [15], other studies highlight notable shortcomings. For instance, Lybarger et al. reported a much lower maximum F1 score of 0.49 [18], and Zhou et al. found an F1 score of 0.416 in nursing contexts despite real-world training data [19]. Similarly, WER ranged from as low as 0.087 in controlled scenarios (Issenman et al. [12]) to more than 2.9 in real-time, multi-specialty outpatient encounters (Biro et al. [32]). van Buchem et al. demonstrated modest ROUGE F1 scores (0.32 unedited vs. 0.41 human-edited) for automated summaries [31]; the fact that human editing still improved these outputs underscores the potential but also the current limitations of LLM-driven summarization.

Workflow efficiency and time savings

Results on workflow efficiency were mixed. While Zick et al. [9] and Issenman et al. [12] observed decreased turnaround time (from days to hours or minutes), Hodgson et al. [16] and Blackley et al. [21] found that post-editing often negated potential time gains. More recent LLM-based systems (e.g., Bundy et al. [23], Ma et al. [34]) claim to shorten overall documentation time for certain specialties, but these claims often relied on small sample sizes or single-site studies, limiting generalizability.

Cost implications

Cost-effectiveness was inconclusive across the included studies. Early work in EDs (Zick et al. [9]) suggested significant cost savings with ASR, whereas Issenman et al. [12] found that voice recognition could be twice as expensive in pediatric gastroenterology. These differences highlight how cost can vary based on clinical setting, complexity of cases and existing staffing models.

Clinical documentation quality and patient care

Studies such as Almario et al. [13, 14] showed that AI-assisted documentation captured more clinically relevant red flags than physician-typed notes. Other research (e.g., Kodish-Wachs et al. [17], Sezgin et al. [30]) observed that high error rates or poor summarization fidelity could pose risks to patient safety, especially if omissions go unnoticed by overburdened clinicians. Notably, more recent digital scribes and LLM-based solutions are designed to create structured summaries (e.g., SOAP notes), although these still generally require human review for accuracy.

Clinician satisfaction, burnout and adoption

Goss et al. [20] identified higher satisfaction when clinicians encountered fewer transcription errors and minimal editing demands. Misurac et al. [28] and Shah et al. [36] further reported decreased burnout levels following the adoption of AI tools, indicating a potential benefit for clinician well-being. Despite improved satisfaction in some quarters, many clinicians expressed reluctance to rely fully on AI scribing, citing concerns about real-time error correction and the need for manual review (e.g., Bundy et al. [23], Moryousef et al. [35]).

Table 1 Key study characteristics for studies included in this review

Table 2 Key study findings for studies reviewed

Risk of bias and study quality

Most of the included studies were assessed to have a low risk of bias and low risk of applicability concerns, as shown in Table 3 and illustrated in Figs. 2 and 3. For studies identified to have a moderate or high risk of bias and applicability concerns, the greatest contributing factor was patient selection, followed by index test. This was possibly due to some studies having unclear patient selection criteria, and some studies having controlled test environments, which may bias the index tests. The QUADAS-2 assessment of the included studies showed predominantly low risk of bias and applicability concerns, supporting the reliability of findings on LLMs as transcription tools in medical domains (Figs. 2 and 3). However, some studies, such as Zick et al. [9], Blackley et al. [21], Hodgson et al. [16], and Kodish-Wachs et al. [17], had high risk of bias in patient selection and applicability concerns, which may limit generalizability. Unclear reference standards in studies like Bundy et al. [23] and Goss et al. [20] suggest potential gaps in validation. While the overall low risk in flow and timing strengthens confidence in the results, variability in methodological rigor underscores the need for standardised evaluation for future studies to ensure consistent and reliable conclusions.

Table 3 Detailed risk of bias assessment for the various studies using QUADAS-2 tool

Fig. 2

Stacked bar chart displaying risk of bias of the studies reviewed using QUADAS-2 tool

Fig. 3

Stacked bar chart for applicability concerns based on QUADAS-2 tool

Discussion

Our review identified 29 studies that investigated the applications of ASR and NLP in medical transcriptions across a variety of clinical settings. The included studies spanned environments such as EDs, inpatient wards, specialized clinics (e.g., gastroenterology, psychiatry, and endocrinology) and even simulated scenarios replicating ambulatory primary care workflows. Owing to this diversity and the significant heterogeneity in study designs, sample sizes and performance metrics, making direct comparisons was challenging. Nonetheless, the findings underscore the wide-ranging potential of AI-based transcription technology in healthcare and highlight certain common challenges to overcome in order to advance the field.

A broad array of AI models and software systems emerged from the review, from older ASR-based tools, such as Dragon NaturallySpeaking Medical Suite and Dragon Medical One [9, 12, 15, 16, 20], to more advanced products that incorporate LLMs, including DAX Copilot and GPT-4–driven systems [31, 33, 34]. The newer studies tend to describe ambient AI scribes, which not only transcribe but also summarize and repurpose clinical notes. Such systems may overcome some limitations of standalone SR tools. While these technologies differ in their underlying architectures, their shared aim is to automate or accelerate clinical documentation by converting speech into text, extracting medically relevant content, and in some cases summarizing or repurposing notes for different uses. Such variety in technological approaches was mirrored by the diversity of clinical settings in which these tools were tested. Some studies relied on synthetic data sets, such as nursing handover records, while others evaluated real-world interactions in high-pressure environments like the ED. This variety demonstrates the adaptability of AI transcription solutions but also reveals that performance is heavily context dependent. Tools that excel in structured, repetitive ED workflows may struggle with varied discussions in multi-specialty clinics or with more complex, freeform patient-doctor dialogues. Likewise, whether a study was conducted using real or simulated encounters also influenced performance, as differences in setting and complexity affect metrics such as WER or F1 scores.

In general, the accuracy of AI-driven transcription remains mixed. WER varied from as low as 0.087 in highly controlled settings (Issenman et al. [12]) to over 50% in conversational or multi-speaker encounters (Kodish-Wachs et al. [17]). Some tools achieved more favorable precision/recall in domain-specific contexts, particularly when leveraging specialty vocabularies (e.g., Happe et al. [10], Suominen et al. [15]). However, others (e.g., Lybarger et al. [18]) highlighted persistent transcription errors that require substantial manual correction. It is, however, also important to interpret performance estimates from older systems cautiously, as rapid technological advances—particularly in neural network and transformer-based models—have likely rendered those results outdated or less generalizable to current AI transcription capabilities.

Besides accuracy, studies conveyed mixed evidence concerning time efficiency. Zick et al. and Issenman et al. both reported substantial reductions in documentation turnaround times [9, 12], whereas newer research from Blackley et al. and Hodgson et al. found negligible or even negative impacts once clinicians’ editing tasks were factored in [16, 20]. Similarly, cost analyses yield no consensus. Zick et al. posited that voice recognition could be up to 100 times less expensive than manual transcription [9], but Issenman et al. found it to be more costly in a pediatric gastroenterology context [12]. As these examples illustrate, site-specific factors—such as the prevalence of templated text, local staff costs, and volume of standard phrases—likely determine the effectiveness of AI transcription.

Although AI transcription systems do not directly deliver patient care, they can indirectly influence clinical outcomes by improving documentation completeness and quality. Almario et al. reported a higher identification of red-flag symptoms in AI-drafted notes [13, 14], while others showed that accurate automated transcription may reduce cognitive load on clinicians. Nevertheless, any potential gains are offset by persistent concerns about error rates. High WER or omissions, as highlighted by Kodish-Wachs et al. [17], remain a threat to real-time decision-making, and this problem has not disappeared with the advent of LLM-based scribes, as seen in Bundy et al., van Buchem et al. and Biro et al. [23, 31, 32]. The subsequent post-editing burden also continues to challenge clinicians’ time management, particularly in busy and dynamic outpatient settings with a variety of patient presentations. Moreover, the efficiency gains from AI transcription are not guaranteed and initial investment costs can be prohibitive [37]. Clinicians’ opinion, acceptance and burnout also surfaced as important considerations for AI adoption. Surveys by Goss et al. [20] and interventions assessed by Misurac et al. [28] revealed that while some clinicians appreciate the potential reduction in documentation burdens, many remain cautious, dissatisfied with high error rates, or concerned about the reliability of AI-generated transcripts.

Several recurring challenges in this space require further attention. First, transcription accuracy often degrades with longer or more complex audio, which suggests the need for incremental or real-time correction features. Second, accented or non-native speech frequently leads to transcription mistakes, highlighting a need for accent adaptation or multi-accent training modules [11, 20, 22, 38]. While systems like AEGIS and NOMINDEX demonstrated high accuracy in specific clinical environments [39], their performance may not generalize well across diverse settings, particularly where speech patterns differ. This is especially true in multinational healthcare systems or regions with a high percentage of non-native English speakers. Third, the training of specialty-specific AI models can be hampered by privacy concerns, as clinicians and institutions may be reluctant to share sensitive patient transcripts for model fine-tuning. Fourth, only a minority of tools currently offer robust real-time error correction, meaning that any short-term gains in typing speed may be negated by lengthy revision processes. Beyond technical refinement, the limited pace of AI transcription adoption in healthcare might reflect deep structural barriers, including regulatory scrutiny over patient safety, a fragmented EHR environment that impedes easy integration, and unclear financial incentives. Moreover, as some studies have indicated (e.g., Issenman et al. [12]), frustrated or unreceptive physicians may be unwilling to incorporate new documentation technologies, especially if these tools require significant training or produce large volumes of errors. Further progress will depend on resolving issues of accuracy, accent variability, system interoperability and cost. Future research should also incorporate advanced evaluation metrics, like ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and BLEU (Bilingual Evaluation Understudy) [40], to systematically assess the quality of AI-generated summaries beyond simple WER.

It might be worth mentioning that workflows, already in limited clinical use (e.g., Tortus, Heidi) in the UK since early 2025 [41], may represent the logical next step for AI documentation: integrating LLMs to summarize and repurpose transcripts without requiring pristine input accuracy. Transcription, in this sense, is merely a transitional stage toward more comprehensive AI scribe solutions that ultimately address patient-clinician interactions holistically.

Limitations of review

This review is not without limitations. Firstly, this review searched three major databases (Medline, Embase and Cochrane Library) as well as grey literature but did not consult IEEE Xplore, which is a key database for engineering and technology-related research, including AI and machine learning. This exclusion may have limited the review’s ability to capture relevant studies on AI transcription systems, particularly those focused on technical innovations in SR and NLP (although they may not be applied in health care). Secondly, most of the studies included in this review were short-term evaluations or proof-of-concept studies conducted in controlled environments or with small sample sizes. There is a lack of long-term, real-world data on the sustained use of AI transcription tools in clinical practice. As a result, this review cannot fully assess the long-term impact of AI transcription on clinician efficiency, patient care outcomes or system-wide healthcare improvements. Thirdly, given the narrative synthesis approach, this review lacked the ability to draw strong, statistically powered conclusions about the overall effectiveness of AI transcription tools. Lastly, the review focused primarily on outcomes such as accuracy, time savings and clinician satisfaction, without addressing other potentially important dimensions, such as cost-effectiveness, user training requirements or implementation barriers. These additional factors could significantly affect the adoption and success of AI transcription tools in clinical practice, but they were not consistently reported in the studies reviewed.

Conclusions

In conclusion, this systematic review revealed that AI SR and transcription software has certain potential to improve clinical documentation, enhance workflow efficiency and reduce the documentation burden on clinicians. The tools designed for specific medical domains can achieve high levels of accuracy, as evidenced by systems like AEGIS and NOMINDEX, which outperformed manual documentation. However, there was significant variability in the performance of AI SR and transcription tools across different software platforms and clinical environments, with general-purpose SR systems often producing high error rates and requiring time-consuming manual corrections. This variability highlights that AI transcription software is still in a developmental phase, with much room for refinement, particularly in adapting systems to accents and complex medical language and improving real-time error correction before widespread adoption can be achieved. Future work should also expand the scope beyond transcription alone—exploring end-to-end AI scribe capabilities and evaluating their real-world effectiveness.