Technical Summary: EDEN – A Large-Scale Corpus of Clinical Notes for Italian

Problem Statement

Clinical notes contain a wealth of patient information but remain largely inaccessible for research due to regulatory barriers (e.g., GDPR), technical challenges (unstructured, heterogeneous natural language), and data fragmentation across isolated systems. This scarcity is particularly acute for languages other than English, such as Italian. While scientific literature (e.g., PubMed) is available, it cannot replace real-world clinical data for applications like automatic digital health record compilation. Furthermore, emergency medicine (ED) research has historically lagged behind other specialties due to the fragmented, time-pressured nature of ED practices, which makes traditional manual data collection infeasible. There is a critical need for large-scale, anonymized, real-world clinical datasets in Italian to support the development and evaluation of Large Language Models (LLMs) for medical applications.

Methodology

Data Collection and Anonymization

The EDEN (Emergency Department Electronic Notes) corpus was collected under the NLP-DeVal study, part of the Horizon Europe-funded eCREAM project. Data was sourced from the Emergency Departments of two Italian hospitals: San Giovanni Bosco (Turin) and Sant'Andrea (Vercelli), covering the period 2021–2023.

• Protocol: The study received approval from hospital ethics committees and adhered to strict data protection regulations.
• Anonymization: A two-stage process was employed:
  1. Removal of all direct references to patients and clinical context (names, dates, diagnoses, disposition).
  2. Application of certified anonymization software (AnonymAI) to eliminate residual third-party references (e.g., relatives' contact details).
• Scope: The resulting corpus comprises approximately 4.2 million clinical notes (221 million words) across 10 common categories (e.g., Anamnesis, Triage, Discharge, LIS, RIS) and hospital-specific categories.

Annotation Scheme (CRF Filling)

To create a structured benchmark, a subset of 5,746 notes from the San Giovanni Bosco (SGB) dataset was manually annotated by trained clinicians.

• Task: The annotation follows a "CRF Filling" paradigm, where clinicians extract information from free-text notes to populate a virtual Case Report Form (vCRF).
• Structure: The vCRF contains 132 items organized into clinical domains (e.g., Clinical Examination, History Taking, Lab Results).
• Value Types: Items support four types of values:
  1. Binary (Y/N): e.g., Administration of diuretics.
  2. Categorical: e.g., Cardiovascular failure (acute/chronic).
  3. Numerical: e.g., pH levels.
  4. Mixed (Number-or-Category): e.g., Heart Rate (numerical value or categories like bradycardic/tachycardic).
• Handling Missing Data: If evidence is insufficient, items are marked as "unknown."
• Process: Annotation was performed using Label Studio, preceded by a calibration phase to resolve ambiguities. The dataset is highly imbalanced; while some notes are densely annotated, ~30% contain no filled items, and the majority of the 132 fields per note are typically "unknown."

Experimental Setup

The authors evaluated the feasibility of using LLMs for this structured extraction task in a zero-shot setting.

• Models: Two models were tested: Gemma-27B (general purpose) and MedGemma-27B (biomedically adapted variant).
• Prompting Strategies: Three configurations were evaluated to balance efficiency and control:
  1. Item: One prompt per item (132 calls per note).
  2. Group: One prompt per clinical group (7 calls per note).
  3. Full: All 132 items in a single prompt (1 call per note).
• Metrics: Macro-F1 was selected as the primary metric to address extreme class imbalance (where "unknown" dominates), alongside Micro-F1, Weighted-F1, and Accuracy.
• Baselines: Random prediction and "Most Common" (always predicting "unknown").

Key Contributions

1. EDEN Corpus: The largest freely available corpus of Italian clinical notes, containing ~4 million anonymized notes from real ED settings.
2. Annotated Subset: A manually curated subset of 5,746 notes with 15,528 annotations across 132 structured CRF items, covering diverse value types and clinical domains.
3. CRF Filling Benchmark: The introduction of a structured information extraction task specifically designed for clinical notes, moving beyond standard Named Entity Recognition (NER) to complex value extraction.
4. Zero-Shot Baselines: The first evaluation of 27B-parameter LLMs on Italian clinical CRF filling, demonstrating that biomedical domain adaptation (MedGemma) yields significant improvements over general models.

Results

• Baseline Performance: The "Most Common" baseline (predicting "unknown" for all items) achieved high Micro-F1 (0.978) and Accuracy (0.978) due to class imbalance, but a low Macro-F1 (0.404), highlighting the difficulty of extracting informative, non-default values.
• Model Performance:
  • MedGemma-27B consistently outperformed Gemma-27B in the Item and Group configurations, achieving a Macro-F1 of 0.702 in the Group configuration. This confirms the value of biomedical domain pre-training for Italian clinical tasks.
  • The Full prompting configuration showed a slight performance drop or reversal for MedGemma, suggesting that generating 132 predictions in a single response overwhelms the model's instruction-following capabilities.
• Efficiency: The Group configuration emerged as the optimal trade-off, reducing inference time by roughly 10x compared to the Item configuration while maintaining the highest Macro-F1 scores.

Significance and Claims

The paper positions EDEN as a critical resource to fill the gap in Italian clinical data, enabling research into LLMs for high-stakes medical applications. The authors claim that:

• EDEN is the largest freely available corpus of Italian clinical notes.
• The dataset is unique in its structured output diversity (binary, categorical, numerical, mixed) across multiple clinical domains.
• Large Language Models can extract meaningful clinical information from Italian notes without task-specific supervision (zero-shot), particularly when using biomedical-adapted models and grouped prompting strategies.
• The resource aims to catalyze research in Italian clinical NLP, structured information extraction, and the reliable evaluation of LLMs in emergency care settings.

The authors remain modest regarding the current state of the technology, noting that the zero-shot results, while promising, leave room for improvement via few-shot and fine-tuning approaches, which are planned for future work. They also acknowledge the structural bottlenecks in ethics approval and data transfer that hindered the collection process, highlighting a need for standardized regulatory frameworks for retrospective NLP studies.