Linguistic Blind Spots in Clinical Decision Extraction

Technical Summary: Linguistic Blind Spots in Clinical Decision Extraction

Problem Statement
Automatically extracting medical decisions from clinical notes is essential for clinical decision support and patient-facing summaries. However, clinical documentation exhibits significant stylistic heterogeneity due to time pressures and domain-specific conventions. This variation creates predictable failure modes for extraction models. While prior work has addressed decision extraction and linguistic complexity separately, there is a lack of systematic quantification regarding how specific linguistic properties of target spans relate to extraction performance at the span level. This study investigates whether linguistic characteristics vary across decision categories and whether these differences explain extraction failures.

Methodology
The authors utilize the MedDec dataset, comprising 451 discharge summaries from MIMIC-III annotated with decision spans according to the Decision Identification and Classification Taxonomy for Use in Medicine (DICTUM). The analysis focuses on nine decision categories (excluding legal/insurance due to limited annotation).

The methodology involves two primary research questions (RQs):

1. RQ1 (Linguistic Variation): How do linguistic characteristics differ across medical decision categories?
2. RQ2 (Extraction Reliability): Are these linguistic characteristics associated with extraction errors?

To address these, the authors compute seven linguistic indices for each gold-standard span using the LFTK library and lexicon-based augmentation:

• Readability/Complexity: Flesch-Kincaid Grade Level (FKGL) and average syllables per word.
• Lexical Composition: Proportion of proper nouns (entity density) and stopwords (telegraphic vs. narrative style).
• Discourse Markers: Proportion of pronouns, and binary presence of hedging and negation cues.

A standard RoBERTa-based span extraction model (trained on the MedDec training split) serves as the baseline. Evaluation is conducted on the validation split using two criteria:

1. Exact Match: The predicted span must have the correct category and an exact text match to the gold span.
2. Relaxed Overlap: A gold span is matched if a predicted span of the same category has a token-level Intersection over Union (IoU) $\ge$ 0.5.

Performance is measured via recall (TP/(TP+FN)), stratified by linguistic index bins and decision categories. Statistical significance is assessed using cluster-bootstrapped confidence intervals and logistic regression controlling for category fixed effects.

Key Results

• Linguistic Signatures: Decision categories exhibit distinct linguistic profiles. Drug-related and problem-defining decisions are "entity-dense" and "telegraphic," characterized by high proper noun density and low stopword proportions. Conversely, advice and precaution decisions are "narrative," featuring higher proportions of stopwords, pronouns, hedging, and negation.
• Extraction Performance: The baseline model achieves an overall exact-match recall of 48%. However, performance varies drastically based on linguistic features:
  • Stopwords: There is a 34-point recall gap between the lowest and highest stopword-proportion bins (58% vs. 24%).
  • Entity Density: Spans with high proper noun proportions achieve 61% recall, compared to 43% for low-density spans.
  • Discourse Markers: Spans containing hedging or negation cues show lower recall (37% and 40%, respectively) compared to those without.
• Boundary Disagreements: Under the relaxed overlap criterion (IoU $\ge$ 0.5), overall recall increases to 71%. This suggests that many extraction errors are not complete misses but rather disagreements on span boundaries, particularly in narrative-style texts.
• Regression Analysis: Logistic regression confirms that the proportion of stopwords is strongly negatively associated with extraction success ($\beta = -0.56$), even after controlling for decision category. The effects of hedging and negation become less robust when category is controlled, suggesting they act largely through their association with specific decision types (e.g., advice).

Contributions
The paper makes three specific contributions:

1. Characterization: It provides a detailed characterization of linguistic variation across nine medical decision categories in discharge summaries, identifying distinct "signatures" for different decision types.
2. Error Analysis: It links linguistic features to span-level extraction failures, quantifying a substantial recall degradation (34 points) associated with high stopword proportions.
3. Downstream Implications: It connects these error patterns to downstream utility, noting that decision categories critical for patient understanding (e.g., advice and precaution) disproportionately exhibit the linguistic features most associated with extraction failures.

Significance and Claims
The authors conclude that narrative-style spans, which are common in advice and precaution decisions, represent a consistent "blind spot" for standard extraction models under exact-matching evaluation. The study argues that the telegraphic, entity-dense style of drug-related decisions aligns well with current model capabilities, whereas the conditional, hedged, and pronoun-heavy nature of patient instructions poses significant challenges.

The paper claims that downstream systems should incorporate boundary-tolerant evaluation and extraction strategies to better handle these narrative spans. It does not propose new model architectures but rather highlights the need for evaluation metrics and deployment strategies that account for the specific linguistic heterogeneity of clinical decision making. The work underscores that documentation disparities (such as the style of advice vs. drug orders) can propagate into downstream clinical NLP models, affecting reliability.