💬 NLP

Toward Automatic Filling of Case Report Forms: A Case Study on Data from an Italian Emergency Department

This paper addresses the scarcity of annotated data for automatic Case Report Form (CRF) filling by introducing a new Italian Emergency Department dataset, defining the task and evaluation metrics, and demonstrating that while state-of-the-art Large Language Models can perform this task in a zero-shot setting, their outputs require correction for inherent biases such as excessive caution.

Original authors: Gabriela Anna Kaczmarek, Pietro Ferrazzi, Lorenzo Porta, Vicky Rubini, Bernardo Magnini

Published 2026-02-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gabriela Anna Kaczmarek, Pietro Ferrazzi, Lorenzo Porta, Vicky Rubini, Bernardo Magnini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor working in a busy emergency room. Every time a patient walks in, you have to write down a story of what happened: their symptoms, their history, what tests you ran, and what you think is wrong with them. You write this in a free-flowing, messy way, like a diary entry. This is the Clinical Note.

Now, imagine that later, a researcher wants to study these patients to find patterns (like "Do people with high fevers usually have a specific type of headache?"). To do this, they can't read thousands of messy diary entries. They need the data to be neat, organized, and in a specific box format. This box format is called a Case Report Form (CRF).

The Problem: The "Copy-Paste" Nightmare

Traditionally, a human has to read the messy diary entry and manually fill out the neat box. It's slow, boring, and prone to typos.

In recent years, we've built super-smart AI computers (called Large Language Models or LLMs) that are great at reading and understanding language. The idea is simple: Let the AI read the messy diary and automatically fill out the neat box for us.

But there's a catch: To teach an AI how to do this, you need thousands of examples where someone has already done the work perfectly (a messy note next to a perfectly filled box). For a long time, these examples didn't exist, especially in Italian. It was like trying to teach a robot to drive without ever showing it a picture of a road.

The Solution: A New "Training Gym"

This paper introduces a new "training gym" for AI. The researchers went to the San Giovanni Bosco Hospital in Turin, Italy, and collected 290 real emergency room notes. They then had real doctors carefully fill out the corresponding "neat boxes" (CRFs) for each note.

The CRF they used is huge—it has 134 different questions (like "Did the patient have a fever?", "What was the blood pressure?", "What is the final diagnosis?").

The Catch with the Data:
Because of strict privacy laws, they had to scrub all names and IDs. This means the AI can't see a patient's full story over time (like a note from the morning and a note from the evening for the same person). It only sees one note at a time. This makes the job harder, but it's safer for patient privacy.

The Experiment: Teaching the AI

The researchers took a powerful, open-source AI (called LLaMA-3) and gave it a simple test:

Input: A messy Italian note from the ER.
Task: "Here is a list of 134 questions. Read the note and answer them. If the note doesn't say the answer, write 'Unknown'."
Output: A filled-out form.

They did this in a "Zero-Shot" setting, meaning they didn't spend months training the AI on this specific data first. They just asked it to do the job immediately, like asking a smart student to take a test without studying the specific textbook first.

The Results: Good, but Cautious

Here is what happened:

The "Lazy" Baseline: If you just told the AI to answer "Unknown" for every single question, it would be right about 96% of the time. Why? Because most of the time, the note doesn't contain the answer to every single one of the 134 questions. (Imagine a note about a broken leg; it won't mention the patient's eye color, so "Unknown" is the correct answer).
The AI's Performance: The AI was slightly less accurate overall (92%) because it tried to be helpful and guess answers when it wasn't sure. However, it was much better at finding the rare, important answers that the "Lazy" baseline missed.
The "Cautious" Bias: The AI was a bit too shy. It preferred to say "Unknown" rather than risk guessing wrong. In medicine, being too cautious is actually okay (it's better to say "I don't know" than to give a wrong diagnosis), but for a computer to be truly useful, we need to teach it to be brave enough to find the hidden clues without making mistakes.

The Big Picture

Think of this paper as laying the foundation for a new building.

The Foundation: They built the first public dataset of Italian emergency notes paired with structured forms.
The Blueprint: They defined exactly how to measure if the AI is doing a good job.
The Stress Test: They showed that current AI can do the job, but it needs more training and tuning to be perfect.

In simple terms: The researchers proved that AI can read Italian emergency room notes and fill out complex medical forms automatically. It's not perfect yet (it's a bit too cautious), but now that we have the data and the method, we can build better, smarter tools to help doctors and researchers save time and save lives.

1. Problem Statement

The core challenge addressed is the automatic population of Case Report Forms (CRFs) from unstructured clinical notes.

Context: CRFs are essential for clinical research but traditionally rely on manual data entry, which is time-consuming and prone to error. While Electronic Health Records (EHRs) exist, extracting structured data from them remains difficult.
Limitations of Current Approaches:
- Existing solutions often rely on simple keyword matching or vocabulary lookups, failing to capture the contextual richness of clinical language.
- There is a scarcity of annotated datasets specifically designed for training and testing Large Language Models (LLMs) on CRF-filling tasks, particularly using real-world data from clinical settings.
- Previous attempts to generate data often rely on semi-automatic methods from scientific documents rather than realistic clinical narratives.
Specific Challenge: The task involves cross-lingual information extraction (Italian clinical notes $\to$ English CRF schema) and handling extreme class imbalance, where the majority of fields are legitimately "unknown" because the information is absent from the note.

2. Methodology

A. Dataset Construction

The authors created a new, publicly available dataset derived from the Emergency Department of San Giovanni Bosco Hospital (Turin, Italy).

Source: 290 anonymized emergency-care notes collected between Jan 2021 and Dec 2023.
Categories: Notes span seven types: anamnesis, triage, nursing care, specialist consultation, medical visits, diagnostic reports, and discharge summaries.
Anonymization: Strict protocols removed all identifiers (names, IDs, etc.). Notably, notes from the same patient across different categories cannot be linked, ensuring privacy but limiting longitudinal reconstruction.
Annotation:
- Performed by clinicians using Label Studio.
- Annotators extracted text spans and mapped them to a 134-item CRF schema.
- The schema covers seven groups: History taking, Clinical examination, Diagnostic test results, Lab results, Imaging results, Treatment, and Final diagnosis.
- Value Sets: Items use binary (yes/no), ordinal (normal/abnormal), or measured/unknown values. Crucially, "unknown" is a valid and frequent label for missing information.

B. Task Definition

Objective: Automatically fill the 134-item CRF for a given clinical note.
Input: Italian clinical note.
Output: A value for each of the 134 items from a predefined closed set (including "unknown").
Evaluation Metrics:
- Micro-F1 (Accuracy): Measures overall correctness. Heavily skewed by the "unknown" class.
- Macro-F1: Averages performance across all possible values for an item, providing a better view of how well the model handles rare, clinically informative values (non-"unknown").

C. Experimental Setup

Model: LLaMA-3-8B Instruct (open-source).
Setting: Zero-shot (no fine-tuning).
Prompting Strategy:
- Each CRF item is queried independently to avoid cascading errors.
- Prompts include: System role, item group context, specific instruction with allowed values, and the full Italian note.
- Constraint: The model must select only from the predefined allowed values.
Baseline: A "Most-Frequent-Value" baseline that always predicts "unknown" for every item.

3. Key Contributions

New Annotated Resource: Introduction of the first public dataset linking real-world Italian emergency department notes to a structured 134-item CRF schema.
Task Definition & Benchmark: Formal definition of the CRF-filling task with specific metrics (Micro/Macro-F1) to address the class imbalance problem inherent in clinical data.
Feasibility Study: A pilot study demonstrating that open-source LLMs can perform this task in a zero-shot setting, even across languages (Italian to English).
Analysis of Clinical Reality: Detailed characterization of the data, highlighting the sparsity of annotations (average 5.7 items per note out of 134) and the dominance of specific groups (History and Clinical Exam) over others (Lab/Imaging).

4. Results

Metric	Most-Frequent Baseline ("Unknown")	LLaMA-3 8B (Zero-Shot)
Micro-F1	0.963	0.920
Macro-F1	0.353	0.548

Micro-F1 Analysis: The baseline achieves a deceptively high score (0.963) because ~96% of the data is "unknown." The LLM scores slightly lower (0.920) because it attempts to predict actual values, occasionally making errors on the majority class.
Macro-F1 Analysis: The LLM significantly outperforms the baseline (0.548 vs. 0.353), proving it can successfully retrieve non-trivial, informative values that the baseline ignores.
Prediction Breakdown:
- When the ground truth is "unknown," the model is correct 90.42% of the time.
- When the ground truth is a specific value (non-unknown), the model is correct only 1.57% of the time (though it attempts to predict them).
Precision/Recall: The model shows balanced precision (0.562) and recall (0.569) for non-trivial classes, indicating it does not strictly favor one over the other but struggles with low-frequency classes.
Constraint Adherence: The model consistently adhered to the schema, selecting only from the predefined valid values.

5. Significance and Future Work

Validation of LLMs: The study confirms that LLMs can approach CRF-filling in a zero-shot setting without fine-tuning, even with cross-lingual requirements.
Bias Identification: The results highlight a cautious bias in LLMs; they tend to favor "unknown" answers to avoid hallucination. While this ensures safety, it reduces the extraction of valuable clinical data.
Path Forward:
- Current "out-of-the-box" tools are insufficient for clinical deployment.
- Future work requires systematic, large-scale experimentation and specialized engineering (e.g., fine-tuning, retrieval-augmented generation, or bias correction) to improve the recovery of non-trivial values while maintaining high accuracy on "unknown" fields.
- The dataset serves as a critical benchmark to drive progress in automating clinical narrative structuring.

In summary, this paper bridges the gap between NLP research and clinical application by providing a realistic, annotated dataset and demonstrating that while current LLMs show promise, significant work remains to optimize them for the rigorous demands of clinical research data extraction.