🤖 machine learning

Complete Evidence Extraction with Model Ensembles: A Case Study on Medical Coding

This paper introduces a complete evidence extraction task for high-stakes medical coding and demonstrates that aggregating token-level attributions from a small ensemble of models (a Rashomon ensemble) significantly improves evidence recall with minimal overhead compared to individual models.

Original authors: Katharina Beckh, Sven Heuser, Stefan Rüping

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Katharina Beckh, Sven Heuser, Stefan Rüping

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Finding the "Whole" Story

Imagine you are a detective trying to solve a crime. Usually, you just need one solid clue to make an arrest. That's like most AI systems today: they look for the shortest, most obvious piece of evidence to make a decision.

But in high-stakes fields like medical billing and safety, a single clue isn't enough. You need the entire story. If a doctor is deciding whether a patient needs to stay in the hospital longer, they can't just look at one symptom; they need to see every piece of evidence in the patient's chart that supports that decision. Missing even one small detail could mean a patient gets sent home too early, which is dangerous.

The paper calls this "Complete Evidence Extraction." It's not just about finding a reason; it's about finding all the reasons hidden in a massive wall of text (like a 6,000-word hospital discharge summary).

The Solution: The "Rashomon" Team

The researchers asked a simple question: What if we don't rely on just one detective, but instead hire a whole team?

They used a concept called the Rashomon effect (named after a famous movie where different witnesses tell the same story in different ways).

The Setup: They took several AI models that were all equally good at the job (like hiring 10 detectives with the same skill level).
The Twist: Even though they were equally skilled, each model "looked" at the text differently. One might notice the word "tumor," while another notices "pain," and a third spots "swelling."
The Strategy: Instead of picking the "best" single model, they created a team (an ensemble). They asked all 10 models to read the same document, highlight the clues they found, and then combined all those highlights into one giant list.

The Experiment: The Medical Coding Test

The team tested this on a real-world task: Medical Coding.

The Task: A computer reads a patient's medical notes and assigns a specific code (like a barcode) to describe their condition.
The Ground Truth: They had a special dataset where human experts had already highlighted every single word in the text that justified a specific code. This was the "answer key."
The Comparison: They compared how many clues the single best model found versus how many clues the team of 10 models found when they combined their answers.

The Results: Better Together

The results were clear and surprising:

The Team Wins: The group of models found significantly more of the "correct" clues than any single model could on its own.
- Analogy: If a single detective finds 6 out of 10 clues, the team found 8 or 9 out of 10.
Small Price to Pay: The only downside was that the team highlighted a few extra words that weren't strictly necessary (like highlighting the word "the" or "and").
- Analogy: The team was slightly more chatty, but in a 6,000-word document, adding just 10 extra words is like adding a single grain of sand to a beach. It's a tiny cost for a huge gain in safety.
You Don't Need a Huge Team: You don't need 10 models to see the benefit. A team of just three models was already better than the best single detective.

What They Learned About Training

The researchers also tested two different ways to train these AI models:

Method A (IGR): Teaching the model to ignore boring words.
Method B (EGT): Showing the model the human answers during training so it learns to spot the right words.

The Finding: Method B (EGT) made the models more precise (they highlighted fewer wrong words), but it didn't help them find more clues when they worked as a team. The team approach worked best regardless of how they were trained, simply because different models naturally "saw" different things.

The Bottom Line

If you need to make a critical decision where missing a detail is dangerous, don't rely on a single AI.

By combining the "opinions" of several different AI models, you create a safety net that catches almost all the relevant evidence. It's like having a team of detectives where, even if one misses a clue, the others are likely to catch it. The paper proves that for medical coding, this teamwork approach finds more of the truth with very little extra effort.

Technical Summary: Complete Evidence Extraction with Model Ensembles

Problem Statement
In high-stakes domains such as regulatory compliance and clinical medicine, decision support systems must provide complete evidence—defined as all relevant input tokens supporting a decision—rather than just minimal sufficient justifications. While prior work focuses on short, sufficient evidence (often a single word or phrase), critical applications like psychiatric care or medical billing require the identification of all possible indicators across a document to avoid risks such as premature patient discharge or billing errors. The challenge lies in the fact that true predictive features in text are latent, and single models often fail to capture the full set of human-annotated evidence, particularly in long clinical texts.

Methodology
The authors formulate complete evidence extraction as a task and investigate it using a Rashomon ensemble approach. The study is grounded in the Rashomon effect, where different models achieve similar classification performance by relying on distinct solutions (attending to different valid cues).

Dataset: The study utilizes the MDACE dataset, derived from MIMIC-III, containing English clinical discharge summaries (averaging 6,000 tokens) with ICD-9 codes. Crucially, the dataset includes human-annotated evidence spans in both "sufficient" and "complete" styles. The evaluation focuses on the "complete" subset, which contains roughly three times more evidence than the sufficient subset.
Models: The authors use existing domain-specific transformer encoder models fine-tuned on MDACE. Two training regimes are compared:
1. Input Gradient Regularization (IGR): Uses a gradient penalty to reduce irrelevant token importance without using evidence labels.
2. Evidence-Guided Training (EGT): Uses human evidence spans as supervision to guide attention.
  Each regime includes 10 models trained with different random seeds, totaling 20 models.
Ensemble Strategy: A Rashomon ensemble is formed by aggregating the token-level evidence from multiple models. Specifically, the ensemble's evidence is the union of extracted tokens from all models in the group.
Evidence Extraction & Metrics: Evidence is retrieved using AttInGrad feature attribution scores. Performance is measured by token-level recall (the proportion of ground-truth evidence tokens recovered) and precision. Recall is prioritized as missing evidence is considered costlier than checking extra tokens. Statistical significance is assessed via 10,000-fold bootstrap resampling.

Key Results

Recall Improvement: Rashomon ensembles significantly increase evidence recall compared to individual models.
- For IGR, the ensemble recall increased from 0.60 (single model) to 0.87 (ensemble of 10).
- For EGT, recall increased from 0.63 to 0.86.
- These gains are statistically significant ( $p < 0.0002$ for EGT, $p = 0.0036$ for IGR).
Ensemble Size: The marginal utility of adding models declines as the ensemble size increases. However, ensembles of only three models already outperform the best single model. From three (or four for IGR) models onward, even the lowest-performing combination achieves higher recall than the best single model.
Precision Trade-off: Aggregating evidence via union increases recall but reduces precision due to the inclusion of additional tokens. EGT models generally show higher precision than IGR models, but evidence-guided training did not yield a notable recall improvement for the ensemble compared to IGR.
Diversity: Pairwise Jaccard similarity between models ranges from 0.52 to 0.57, indicating substantial overlap but also systematic differences in the evidence patterns retrieved by different models.
Token Overhead: An ensemble of 10 models retrieves approximately 10–11 tokens per document. Given the average document length of 6,000 tokens, this overhead is considered small and acceptable for environments where missing evidence is costly.

Significance and Claims
The paper claims to present the first systematic case study of complete evidence extraction in a medical coding setting with human-annotated evidence. Its primary contribution is demonstrating that exploiting model diversity through Rashomon ensembles is a viable strategy for recovering more complete evidence than any single model can achieve alone.

The authors argue that while single models are often sufficient for classification, they are insufficient for exhaustive evidence retrieval. By aggregating evidence from multiple equally-performing models, the system can recover information that individual models miss. The study suggests that for tasks requiring exhaustive coverage, such as clinical decision support, Rashomon ensembles offer a practical solution with a manageable token overhead. The paper modestly notes limitations, including the reliance on a single feature attribution method and the small size of the complete-evidence test set, but observes that the patterns of evidence similarity hold across the larger, sufficient-evidence test set.