🤖 AI

EPPC-OASIS: Ontology-Aware Adaptation and Structured Inference Refinement for Electronic Patient-Provider Communication Mining in Secure Messages

The paper introduces EPPC-OASIS, an ontology-aware framework that combines Wasserstein alignment during fine-tuning with structured inference refinement to achieve consistent improvements in automatically extracting fine-grained communication behaviors from secure patient-provider messages.

Original authors: Samah Fodeh, Sreeraj Ramachandran, Elyas Irankhah, Muhammad Arif, Afshan Khan, Ganesh Puthiaraju, Linhai Ma, Srivani Talakokkul, Jordan Alpert, Sarah Schellhorn

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Samah Fodeh, Sreeraj Ramachandran, Elyas Irankhah, Muhammad Arif, Afshan Khan, Ganesh Puthiaraju, Linhai Ma, Srivani Talakokkul, Jordan Alpert, Sarah Schellhorn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive library of letters written between patients and their doctors. These aren't just simple "hello" notes; they contain complex conversations about medication, insurance worries, emotional support, and shared decisions about treatment.

The problem is that these letters are messy. A single sentence might contain three different types of information, and the way people write them is informal. Doctors and researchers want to read thousands of these letters to understand how care is delivered, but reading them one by one is impossible. They need a way to automatically sort these letters into neat, organized categories.

This paper introduces a new system called EPPC-OASIS to do exactly that. Here is how it works, broken down into simple concepts:

1. The Challenge: The "Messy Closet" Problem

Think of the doctors' communication rules (called an Ontology) as a very specific, organized closet with 9 main sections and 34 smaller drawers.

The Goal: When a computer reads a letter, it needs to figure out which drawer the information belongs in and point to the exact sentence that proves it.
The Problem: Standard AI models are like a child trying to clean that closet. They might put a "sock" in the "shoe" drawer because they look similar, or they might find the right drawer but forget to point to the sock itself. They often get the general idea right but mess up the specific details.

2. The Solution: Two-Step Training

The authors built a two-step system to teach the AI to be a professional organizer.

Step A: The "Map-Making" Phase (Ontology-Aware Adaptation)

Instead of just showing the AI examples of letters and answers (like a standard teacher), the researchers gave the AI a map of the closet.

How it works: They taught the AI that certain drawers are neighbors. For example, "Medication" and "Insurance" are related, so the AI should treat them as close friends in its brain.
The Analogy: Imagine teaching a student not just by giving them flashcards, but by drawing a map of the city so they understand how neighborhoods connect. If the student gets lost, they know which street to turn onto to get back on track.
The Result: The AI learned to organize its thoughts so that similar medical concepts stay close together, reducing confusion between similar-sounding categories.

Step B: The "Double-Check" Phase (Structured Inference Refinement)

Even with a good map, the AI might still make a mistake when writing the final answer. So, the system adds a "quality control" step before the answer is finalized.

How it works: The AI generates a few different versions of the answer. It then acts like a strict editor:
- Self-Verification: "Did I actually find this sentence in the letter, or did I just guess?"
- Self-Consistency: "If I asked the same question five times, would I get the same answer?"
- Hybrid Correction: "This version has the right category, but that version has the better sentence. Let's combine them."
The Analogy: It's like a team of editors reviewing a draft. One checks the facts, another checks the grammar, and a third makes sure the final story makes sense before it gets published.

3. The Results: A Better Organizer

The researchers tested this system on thousands of real patient letters using different sizes of AI models (from small, fast ones to huge, powerful ones).

The Score: The best version of their system got a score of 77.13% on correctly identifying the categories and 63.83% on correctly identifying the category plus the exact sentence evidence.
The Gain: This was a small but consistent improvement (about 1 to 2 points) over the best standard methods.
The Trade-off: The paper notes that while other methods were slightly better at finding the exact sentence (the "evidence"), their method was much better at keeping the categories organized correctly. It's like saying their system was better at sorting the laundry into the right piles, even if the other systems were slightly better at folding the individual shirts.

4. What This Means (And What It Doesn't)

What it does: It proves that teaching an AI to understand the "structure" of medical rules (the map) and then having it double-check its work (the editors) makes it much better at sorting complex medical letters.
What it doesn't do: The paper is very clear that this is a research tool. It is designed to help researchers look back at old letters to study how patients and doctors talk. It is not currently a tool that doctors should use to make real-time decisions about patient care, nor does it claim to improve patient health outcomes directly. It is a method for "mining" information, not a medical device.

In short: The paper presents a smarter way to teach computers to read and sort messy medical letters by giving them a mental map of the rules and a strict editing process, resulting in more reliable data for researchers to study.

Technical Summary: EPPC-OASIS

Problem Statement
Secure patient-provider messages contain rich, clinically significant communication behaviors (e.g., care coordination, emotional concerns, shared decision-making) that are difficult to characterize manually at scale. While the Electronic Patient-Provider Communication (EPPC) framework provides a structured ontology for coding these behaviors, automated extraction remains challenging. The task requires more than single-label classification; models must identify communication behaviors, map them to a hierarchical system of high-level codes and specific sub-codes, and ground these predictions in specific text spans from the original message.

Current approaches face two primary limitations:

Representation Gap: Standard supervised fine-tuning (SFT) treats annotations as isolated output sequences, failing to leverage the hierarchical, imbalanced, and semantically adjacent nature of the EPPC ontology. Consequently, models often make structured errors, such as assigning a plausible sub-code under the wrong parent code or confusing semantically similar behaviors.
Inference Rigidity: Inference is typically treated as a single generation step. However, errors in EPPC extraction often manifest differently for label assignment versus evidence grounding (span selection). A greedy output might have a correct span but a wrong sub-code, while a sampled output might have the correct label but a drifted span. Standard methods do not address these distinct failure modes simultaneously.

Methodology
The authors propose EPPC-OASIS, a two-stage framework designed to improve structured EPPC extraction: Ontology-Aware Adaptation (training) and Structured Inference Refinement (inference).

1. Ontology-Aware Adaptation (Training)
The core innovation is the integration of the EPPC ontology structure directly into the fine-tuning objective using a Wasserstein alignment signal.

Ontology Vector Construction: Gold annotations are converted into multi-hot vectors representing the union of active codes and sub-codes.
Ontology Similarity Prior: A prior matrix ( $P$ ) is constructed based on semantic similarity between code descriptions. This defines a target similarity between examples, transforming flat label overlap into a structured similarity measure that respects the EPPC hierarchy.
Wasserstein Memory-Bank Alignment: To address the sparsity of related examples in standard mini-batches (especially for large models), the method employs a pre-filled, frozen memory bank containing representations of all training examples.
- For each input, the model computes a representation-neighborhood distribution ( $p_{rep}$ ) based on hidden state similarities.
- It compares this against an ontology-neighborhood distribution ( $p_{ont}$ ) derived from the target similarity scores.
- An entropically regularized optimal transport cost (Sinkhorn loss) aligns these two distributions.
Objective: The total loss combines standard supervised extraction loss ( $\mathcal{L}_{SFT}$ ) with the ontology-alignment loss ( $\mathcal{L}_{ont}$ ), weighted by $\lambda_{ont}$ . This encourages the model to organize its representation space to mirror the EPPC ontology structure.

2. Structured Inference Refinement
To address residual errors, the framework applies a deployable refinement layer that does not alter the training objective but improves the final annotation set through verification and aggregation:

Chain-of-Thought Self-Refinement (CoT-SR): Generates a greedy output, verifies span grounding (checking for verbatim presence or high Jaccard overlap), and uses a self-refinement prompt to correct unsupported triplets or fix minor span errors.
Self-Consistency: Samples multiple outputs to stabilize label assignment (Code/Sub-code pairs) via voting, selecting the span with the highest consensus similarity.
Hybrid Correction: Combines the strengths of greedy and self-consistency outputs. It retains the evidence span from a greedy prediction if it overlaps sufficiently with a self-consistency span, replacing the labels with the more stable self-consistency labels.
Selector/Ensemble: Uses internal criteria (e.g., span validity, distinct pair counts) to select the best prediction from CoT-SR and self-consistency candidates, or merges predictions across multiple random seeds for high-capacity models.

Key Contributions

EPPC-OASIS Method: An ontology-aware adaptation method that uses the EPPC code/sub-code inventory to define a Wasserstein alignment objective during fine-tuning, ensuring model representations align with the ontology's hierarchical structure.
Structured Inference Framework: A deployable pipeline combining self-verification, self-consistency, hybrid label correction, and selection/ensembling to address specific failure modes in label assignment and evidence grounding.
Comprehensive Evaluation: Evaluation across multiple open-weight model families (Llama, Mistral, Qwen, DeepSeek) and scales (1B to 70B), comparing against prompting, standard SFT, preference-based optimization (DPO, IPO, etc.), and robustness-oriented baselines (STaR-DRO).
Diagnostic Analysis: Ablation studies separating the effects of ontology-aware training from inference-time refinement, and distinguishing deployable pipeline performance from theoretical upper bounds (component-wise best scores).

Results
The framework was evaluated on a de-identified corpus of 867 annotated examples (5,516 annotations) using micro-averaged F1 scores for Code, Sub-code, Span, Code+Sub-code, and Triplet (Code+Sub-code+Span).

Performance Gains: Across model families, the best deployable pipeline achieved 77.13% Code+Sub-code F1 and 63.83% Triplet F1.
Comparison to Baselines: These results represent modest but consistent absolute gains of +1.39 and +2.12 F1 points, respectively, over the strongest supervised fine-tuning baseline.
Ontology Impact: EPPC-OASIS consistently outperformed standard SFT and preference-based baselines on structural metrics (Code+Sub-code F1), demonstrating superior preservation of the label hierarchy. While preference-based methods (e.g., DPO) showed competitive performance on exact span localization, EPPC-OASIS was superior in maintaining label coherence.
Model Scale: The benefits of ontology-aware adaptation were most pronounced in smaller models (e.g., Llama-3.2-1B) and were further stabilized across different architectures (including Qwen and DeepSeek) through weight averaging of checkpoints.
Ablation Findings: Removing the ontology-alignment term, the memory bank, or the semantic similarity prior resulted in performance degradation, confirming that gains were not merely due to task-specific fine-tuning but specifically due to the structured alignment signal.

Significance and Claims
The paper claims that EPPC-OASIS supports scalable, retrospective mining of electronic patient-provider communication by improving the reliability of structured extraction.

Informatics Enablement: The primary value is enabling retrospective research, quality measurement, and cohort-level analysis of communication patterns, rather than immediate clinical decision support or real-time automation.
Auditability: By extracting grounded evidence spans alongside labels, the system allows researchers to audit the text underlying predicted behaviors, which is critical for sensitive categories like Social Determinants of Health (SDOH).
Methodological Advance: The work demonstrates that incorporating domain ontologies directly into the adaptation objective (via Wasserstein alignment) is more effective for hierarchical, multi-label tasks than treating the ontology solely as a post-hoc evaluation schema or output vocabulary.
Caveats: The authors explicitly state that external validation is required before operational use. The current results do not establish improvements in clinical outcomes, triage, or care coordination. Furthermore, the inference refinement pipelines involve increased computational cost (multiple passes or sampling), which is acceptable for retrospective analysis but may require optimization (e.g., distillation) for real-time deployment.