💻 computer science

Iterative Feedback--Refinement for Faithful Structured Clinical-Note Representation

This paper proposes an iterative feedback-refinement framework using large language models to dynamically induce adaptive schemas for converting unstructured clinical notes into faithful, structured representations, which outperforms existing baselines by significantly reducing omission and hallucination without requiring labeled data or fine-tuning.

Original authors: Eslam Ahmed Mohamed, Irini Logothetis, Adrian Bingham, Kon Mouzakis

Published 2026-07-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Eslam Ahmed Mohamed, Irini Logothetis, Adrian Bingham, Kon Mouzakis

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to read a doctor's messy handwriting. This isn't just about typing words into a computer; it's about Clinical Natural Language Processing, a field where scientists try to turn the chaotic, free-flowing stories doctors write in patient files into neat, organized data that computers can actually use. Think of it like trying to turn a wild, uncharted jungle into a perfectly mapped city grid. The problem is that doctors write in all sorts of different styles, with different sections and different ways of describing the same illness. If you just ask a robot to "summarize" this jungle, it might make things up (a problem called hallucination) or forget important details (called omission). The goal is to get the robot to build a reliable, structured map of the patient's health without inventing new trees or losing the existing ones.

This paper introduces a clever new way to teach the robot how to build that map. Instead of just asking the robot to write a summary and hoping for the best, the researchers created a "feedback loop" where the robot actually learns to design its own map template. They found that by letting the robot critique its own map design and fix it over and over again, it could capture the patient's story much more accurately than standard summarizing methods. It's like the difference between a student who just writes an essay and hopes the teacher likes it, versus a student who keeps redrawing their outline based on the teacher's notes until the structure is perfect. The result? A system that is shorter, more accurate, and much less likely to make things up, even when using smaller, less powerful computer brains.

The Problem: The Jungle of Medical Notes

Doctors write notes that are goldmines of information, but they are messy. One doctor might write a long paragraph about a patient's cough, while another uses bullet points, and a third might bury the diagnosis in the middle of a sentence about the weather. These are unstructured clinical notes. Computers hate this kind of chaos. To help doctors make better decisions or to research diseases, we need to turn these messy notes into structured data—think of it like turning a pile of loose LEGO bricks into a specific, organized model.

The usual way to do this is summarization. You ask a computer, "Read this long note and give me a short version." But here's the catch: computers are great at sounding smart, but they are terrible at being accurate. They might invent a symptom the patient never had (hallucination) or forget to mention a critical allergy (omission). Another approach is to force the computer to fill out a pre-made form (a schema), like a checklist. But if the form was designed by a human who didn't think of every possible way a doctor might write, the computer will either miss information or force the data into the wrong boxes.

The Solution: The "Architect and Critic" Game

The researchers at Deakin University proposed a new idea: instead of just asking the computer to write the notes, let's ask it to design the form it will use to write the notes. They call this Iterative Feedback–Refinement.

Imagine you are building a treehouse.

The First Draft: You build a basic frame (the initial schema).
The Critic: A second AI model (the "Evaluator") looks at the treehouse and the actual tree it's supposed to be built on. The AI says, "Hey, you forgot a ladder! And that window is too small for the cat."
The Architect: A third AI model (the "Refiner") listens to the Critic and fixes the blueprints. It adds a ladder and makes the window bigger.
Repeat: You do this over and over, testing the new blueprints against different parts of the tree. With every round, the blueprint gets better and better at capturing the tree's unique shape.

In the paper, they used Large Language Models (LLMs) to play both roles. One AI acts as the Evaluator, looking at a patient note and the current "form" to see what's missing or wrong. A separate AI acts as the Refiner, taking that feedback and actually changing the form (the JSON schema). They didn't just do this once; they did it iteratively, moving through different groups of patient notes (like different types of trees) to make sure the form worked for everyone.

What They Found: Better Maps, Less Guessing

The team tested this new method against the old ways (just summarizing or using a fixed form) using 515 real clinical notes from a public database called MTSamples. They tried it with eight different AI models, ranging from small ones to massive ones.

Here is what happened:

Accuracy Wins: The iterative method was the clear winner. It achieved the highest scores for faithfulness, meaning it captured the truth of the notes better than anything else.
No More "Making Stuff Up": The system was incredibly good at avoiding hallucinations. It didn't invent symptoms.
No More "Forgetting": It was also the best at avoiding omissions. It remembered to include the important details that other methods dropped.
Shorter is Better: The structured notes it produced were much shorter than the original text—reducing the length to just 34.5% of the original—while still keeping all the important facts.
Size Doesn't Matter (As Much as You Think): Surprisingly, the smaller AI models (like the 4-billion-parameter ones) did just as well as the giant 109-billion-parameter models when using this iterative method. It turns out that having a good "feedback loop" is more important than just having a huge brain.

Why This Matters

The paper suggests that the secret to getting computers to understand medical notes isn't just about making the computers smarter or asking them to write better summaries. Instead, the key is to optimize the structure itself. By letting the computer design and refine its own "form" based on real feedback, we get a system that is reliable, compact, and ready to be used in real-world medical tools.

The researchers are careful to note that while this is a huge step forward, it's not a magic wand that solves everything instantly. They tested it on English notes, and real-world medical records can be even messier. But the core idea—that we should treat the template as something to be improved, not just the text—offers a promising new path for making medical data useful for everyone.

Technical Summary: Iterative Feedback–Refinement for Faithful Structured Clinical-Note Representation

Problem Statement

Unstructured clinical notes in Electronic Health Records (EHRs) contain rich patient information but remain difficult to automate due to variability in layout, narrative style, and section coverage across specialties. Existing approaches to converting these notes into structured representations face two primary limitations:

Free-text Summarization: While improving readability, these methods often suffer from hallucinations (generating unsupported content) and omissions (excluding critical details), and lack the structural constraints necessary for interoperability and information reuse.
Fixed Schema Population: Approaches that populate pre-defined, manually designed schemas (e.g., JSON templates) generalize poorly across diverse documentation styles. Furthermore, deficiencies in the initial schema design propagate systematically to every extraction, as the schema itself is not optimized.

Current methods optimize the generated output rather than the underlying representation. Additionally, standard evaluation metrics (e.g., ROUGE) often fail to capture factual alignment with the source, and post-hoc correction strategies do not resolve representation-level deficiencies.

Methodology

The authors propose a novel Iterative Feedback–Refinement Framework that treats structured clinical documentation as a problem of adaptive schema induction rather than instance generation from a fixed template. The framework utilizes Large Language Models (LLMs) to progressively refine a JSON-based schema without requiring labeled data or task-specific fine-tuning.

Data and Preprocessing

Dataset: The study uses the MTSamples corpus, specifically the 'Consult - History and Physicals' category (515 reports).
Preprocessing:
- Cleaning: Removal of boilerplate and formatting artifacts.
- Outlier Removal: Length-based filtering using a z-score threshold ( $\tau=0.7$ ) to manage GPU memory constraints (KV-cache limits) while retaining sufficient information, reducing the corpus from 515 to 337 documents.
- Clustering: Documents are embedded (using Qwen3:0.6b) and clustered via UMAP and HDBSCAN into 42 thematic clusters. This ensures the refinement process encounters diverse note structures in a controlled sequence rather than random sampling.
- Partitioning: An 85/15 split creates a Refinement Subset (286 notes) for iterative schema updates and a Validation Subset (51 notes) for final evaluation.

The Iterative Framework

The core process involves a dual-LLM loop executed across the 42 clusters:

Schema Initialization: A zero-shot prompt generates an initial, minimal JSON schema ( $S^{(0)}$ ).
Per-Note Evaluation (Evaluator):
- For each note in a cluster, the current schema is used to generate a structured document.
- The evaluator scores the schema on five dimensions: Coverage, Accuracy, Clarity, Conciseness, and Utility.
- The evaluator generates structured feedback in four categories: Missing Information, Representation Issues, Redundancy, and Improvements.
Batch Aggregation: Feedback from all notes in a cluster is aggregated. Mean quality scores are calculated, and feedback items are deduplicated (while preserving frequency signals) to identify systematic schema deficiencies.
Schema Refinement (Refiner): A separate stateless LLM updates the schema ( $S^{(k)}$ ) based on the aggregated feedback. It follows a conservative strategy, prioritizing extensions (adding fields) over deletions and ensuring changes are likely to generalize. The process repeats for each cluster, yielding a final optimized schema ( $S^*$ ).
Final Application: The finalized schema is applied to the validation subset using a stateless filler client to produce the final structured documents.

Baselines

The proposed approach is compared against:

Extractive Summarization: BERT-Centroid (selects salient sentences verbatim).
Abstractive Summarization: Standard single-turn and Chain-of-Thought (CoT) approaches.
In-Context Learning (ICL) Schema: A non-iterative approach where a schema is generated from the entire refinement subset in a single pass.

Evaluation Metrics

Faithfulness is defined as the agreement between the generated representation and the source note, decomposed into Precision (absence of hallucination) and Recall (absence of omission).

LLM-as-Judge: Three metrics using gpt-5-mini: SummarizationMetric (F1 of precision/recall), G-Eval (rubric-based precision/recall), and DAGMetric (decision-tree based).
Reference-Based: SelfCheckGPT (BERTScore consistency and MQAG question-answering consistency).
Compression: Word-level compression ratio relative to the source.

Key Results

The framework was evaluated on 8 open-weight LLMs (ranging from 1B to 109B parameters).

Superior Faithfulness: The Iterative Schema approach achieved the highest mean F1 scores across all three judge-based metrics (SummarizationMetric: 0.939, G-Eval: 0.926, DAGMetric: 0.910), outperforming the strongest abstractive baseline by +0.063 to +0.066 F1 points.
Precision-Recall Balance: Unlike the Extractive baseline (high precision, low recall) or abstractive baselines (lower recall), the Iterative Schema maintained high precision comparable to baselines while significantly improving recall. This indicates the method reduces omissions without increasing hallucinations.
Model Agnosticism: The gains were consistent across model sizes. Mid-sized models (e.g., Qwen3-4B, Llama-3.1-8B) matched or exceeded larger models (e.g., Llama-3.3-70B) when using the iterative refinement strategy.
Compression: The structured representation achieved a mean word compression ratio of 0.345 (34.5% of source length), significantly more compact than abstractive summaries (~0.63–0.66) while maintaining higher faithfulness.
Consistency: The results held across different evaluation mechanisms (question-answering, rubric, decision tree, and distributional metrics), with strong correlations between metrics confirming the robustness of the findings.

Significance and Claims

The paper claims that optimizing the schema itself provides an effective, model-agnostic route to faithful structured clinical representation.

Representation-Centric Optimization: By shifting the optimization target from individual outputs to the shared schema, improvements accumulate and propagate across all subsequent extractions.
Reduced Omission: The framework addresses the persistent limitation of clinical summarization (omission) by explicitly defining required information fields in the schema, encouraging systematic coverage.
No Labeled Data: The approach requires no human-labeled data or task-specific fine-tuning, relying instead on unsupervised clustering and LLM-driven feedback.
Clinical Utility: The resulting machine-readable outputs are naturally compatible with interoperability standards (e.g., FHIR) and knowledge graphs, potentially reducing errors in downstream decision-support systems.

The authors conclude that the key leverage in structured clinical extraction lies less in iterative prompting or output refinement, and more in treating the schema itself as the object of optimization.