🤖 AI

Examine Clinicians' Modification of Hedging Language in Ambient AI Documentation: A Comparative Study of AI Drafts and Final Notes

This study analyzes 62,811 paired ambient AI drafts and clinician-edited notes to reveal that clinicians systematically introduce more hedging language and shift toward greater uncertainty during revisions, with significant variations observed across different AI vendors and clinical specialties.

Original authors: Yiliang Zhou, Yawen Guo, Di Hu, Sairam Sutari, Emilie Chow, Steven Tam, Danielle Perret, Deepti Pandita, Kai Zheng

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiliang Zhou, Yawen Guo, Di Hu, Sairam Sutari, Emilie Chow, Steven Tam, Danielle Perret, Deepti Pandita, Kai Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a doctor's visit as a conversation that gets transcribed by a very fast, very smart robot scribe. This robot, an "Ambient AI," listens to the chat between the doctor and patient and instantly writes a draft of the medical note. However, before this note becomes an official part of the patient's permanent record, the doctor has to read it and edit it.

This study is like a "spot the difference" game. The researchers looked at thousands of these AI drafts and the final versions doctors signed off on. Their main question: When doctors edit the robot's writing, do they make the notes sound more sure of themselves, or more unsure?

Here is the breakdown of what they found, using simple analogies:

1. The "Safety Net" Effect

In medicine, doctors often use "hedging" language. Think of this as a linguistic safety net. Instead of saying, "You definitely have the flu," a doctor might say, "It looks like the flu," or "We cannot rule out the flu." This isn't because they are confused; it's because medicine is rarely 100% certain, and these words protect the doctor from being wrong if new information comes up later.

The study found that when doctors edited the AI's draft, they added more of these safety nets.

The Analogy: Imagine the AI wrote a sentence like, "The patient has a broken leg." The doctor changed it to, "The patient likely has a broken leg."
The Result: Doctors were much more likely to add words like "likely," "possible," or "may" to the AI's text than they were to remove them. They were turning the AI's confident statements into more cautious, medically accurate ones.

2. The "Tentative" Tendency

When the researchers looked closely at the specific words doctors swapped out, they saw a clear pattern.

The Analogy: If the AI said, "This is definitely a problem," the doctor often changed it to, "This is possibly a problem."
The Result: The edits consistently pushed the notes toward greater uncertainty. Doctors weren't trying to make the AI sound more definitive; they were trying to make it sound more humble and careful.

3. Different Robots, Different Drafts

The study looked at two different AI companies (Vendor A and Vendor B).

The Analogy: Think of them as two different chefs. Chef A might write a recipe that is 90% right, while Chef B writes one that is 85% right.
The Result: Both chefs needed the doctor to add more "maybe" and "likely" words to their recipes. However, the amount of editing needed varied slightly between the two. One AI system's drafts required a bit more "cautioning" than the other, but the overall trend was the same: doctors made both systems sound more uncertain.

4. Different Specialties, Different Nuances

The researchers also checked if doctors in different fields (like heart specialists vs. skin doctors) edited differently.

The Analogy: A carpenter and a painter might both fix a robot's blueprint, but they might fix different parts of it.
The Result: While general doctors (Primary Care) and specialists generally did the same thing (adding uncertainty), the specifics varied wildly depending on the specialty. Some medical fields naturally deal with more guesswork than others, so their notes ended up with different amounts of "hedging" language.

The Bottom Line

The AI is great at listening and typing, but it tends to be a bit too confident in its guesses. Doctors act as the quality control managers, stepping in to add the necessary "wiggles" and "maybe's" to ensure the note reflects the messy, uncertain reality of real-life medicine. They aren't fixing typos; they are fixing the confidence level of the notes, ensuring the final record is safe, accurate, and honest about what is known and what is still a guess.

Technical Summary: Clinicians' Modification of Hedging Language in Ambient AI Documentation

Problem Statement
Ambient Artificial Intelligence (AI) documentation systems utilize automatic speech recognition (ASR) and generative AI (GenAI) to convert patient-clinician conversations into clinical note drafts. While these tools aim to reduce documentation burden, clinicians routinely revise these drafts before finalizing them in Electronic Health Records (EHRs). A critical gap exists in understanding how these revisions specifically alter "hedging language"—lexical and syntactic cues (e.g., "may," "likely," "cannot rule out") that signal uncertainty, ambiguity, or provisional clinical reasoning. Previous work suggests clinicians view hedging as a necessary risk-management and communication strategy, yet it remains unclear whether AI drafts under- or over-represent uncertainty and how clinicians recalibrate this language during the editing process. Furthermore, it is unknown if these editing patterns vary significantly across different AI vendors or clinical specialties.

Methodology
The study conducted a paired analysis of 62,811 note sections derived from 98,311 ambient AI drafts and corresponding clinician-finalized notes from the University of California Health system (late 2023). The data involved two commercial vendors (Vendor A and Vendor B) across primary care and 32 specialty care settings.

Data Processing: The researchers aligned draft and final notes at the sentence level using pairwise similarity scores. Sentences were categorized as "equal," "drops" (deleted), "inserts" (added), or "replacements" (modified). The primary analysis focused on replacement spans, where the draft text was directly edited by the clinician, providing a symmetric pre-edit/post-edit comparison unit.
Hedging Lexicon: A curated lexicon of 536 unique hedging terms was compiled from prior literature and CoNLL-2010 resources. Three independent reviewers annotated each term for its directional effect on certainty: -1 (greater certainty), 1 (greater uncertainty), or 0 (neutral). Inter-rater agreement was substantial (Fleiss' $\kappa$ = 0.72).
NLP Pipeline: A rule-based clinical NLP pipeline using SciSpaCy and an EntityRuler was developed to detect hedging term mentions and assign assertion labels.
Statistical Analysis:
- Prevalence: McNemar's test and one-sided Wilcoxon signed-rank tests were used to compare hedging presence and frequency in pre-edit versus post-edit replacement spans.
- Directionality: A "replace net score" was calculated (addition score minus deletion score) to determine if edits shifted text toward greater certainty or uncertainty. One-sample Wilcoxon signed-rank tests assessed if net scores deviated significantly from zero.
- Stratification: Analyses were stratified by vendor and by provider grouping (Primary Care vs. Specialty) and individual specialties, utilizing Chi-square, Mann–Whitney U, and Kruskal–Wallis tests.

Key Contributions

Quantification of Editing Patterns: The study provides the first granular, paired analysis of how clinicians modify hedging language in AI-generated drafts, distinguishing between the introduction of new hedging and the removal of existing hedging.
Directionality Metrics: It introduces a quantitative framework for measuring the "directionality" of edits, demonstrating that clinician revisions systematically shift text toward greater uncertainty rather than greater certainty.
Contextual Heterogeneity: The research characterizes significant variations in hedging modification patterns across different AI vendors and, more notably, across diverse clinical specialties.

Results

Prevalence of Hedging: Clinicians were significantly more likely to introduce hedging into previously non-hedged text than to remove hedging from previously hedged text. In replacement spans, 4,803 sections transitioned from "no hedging" to "hedging present," while only 3,135 transitioned from "hedging present" to "no hedging" ( $P < 0.001$ ). Post-edit text contained significantly more hedging mentions than pre-edit text.
Directionality: Among sections with directional hedging edits, there was a significant overall tendency toward greater uncertainty. The mean replace net score was positive (0.23, $P < 0.001$ ) for sections with directional edits. Specifically, 5.48% of sections shifted toward greater uncertainty, compared to 4.01% shifting toward greater certainty.
Vendor Differences: While both vendors showed a trend toward increased uncertainty, Vendor B exhibited a stronger shift (mean net score 0.34) compared to Vendor A (0.20). Post-edit hedging prevalence also differed significantly between vendors (10.9% vs. 10.2%).
Specialty Differences: Primary Care (PCP) and Specialty care showed similar directional tendencies. However, when analyzing individual specialties, significant heterogeneity emerged. For instance, Dermatology and Rheumatology showed the highest mean replace net scores (0.49 and 0.38, respectively), indicating a strong tendency to increase uncertainty, whereas Endocrinology showed a negative net score (-0.26), indicating a shift toward greater certainty.

Significance and Claims
The paper claims that clinician editing of ambient AI drafts is not merely a correction of factual errors but a systematic recalibration of uncertainty. The findings suggest that ambient AI drafts often lack sufficient hedging, prompting clinicians to actively add qualifiers to align the documentation with the provisional nature of clinical reasoning and risk-management practices.

The authors assert that these patterns indicate hedging recalibration is a cross-vendor phenomenon, suggesting that current AI models may not adequately ground uncertainty language in clinical context. The observed heterogeneity across specialties implies that a "one-size-fits-all" approach to AI documentation may be insufficient; instead, systems may require specialty-specific tuning to better represent diagnostic ambiguity and evidence strength. The study concludes that understanding these editing behaviors is essential for improving the reliability of AI-generated notes and reducing unnecessary clinician correction burdens.

Limitations noted by the authors include the focus on replacement spans (excluding deleted/inserted text analysis), the lack of sociodemographic controls, and the absence of multivariable modeling to account for case-mix differences.