Combining Token Classification With Large Language Model Revision for Age-Friendly 4M Entity Recognition From Nursing Home Text Messages: Development and Evaluation Study

This study presents and evaluates a multi-stage pipeline that combines a fine-tuned Bio-ClinicalBERT token classifier with locally deployed open-source large language models for revision, demonstrating that this hybrid approach significantly improves the accuracy and efficiency of extracting structured Age-Friendly 4M (What Matters, Medication, Mentation, and Mobility) information from informal nursing home text messages compared to single-stage models.

The Gist

The Big Picture: Finding Gold in a Mountain of Notes

Imagine a nursing home as a busy hospital ward where nurses, doctors, and therapists are constantly chatting. Instead of writing long, formal reports, they send quick, short text messages to each other about patients.

These messages are full of important clues:

• "Mrs. Jones is confused today." (Mentation)
• "He's refusing to walk." (Mobility)
• "She wants to go home for Christmas." (What Matters)
• "Give her the new heart pill." (Medication)

This is the 4M Framework (What Matters, Medication, Mentation, Mobility). It's a gold standard for good care.

The Problem: Right now, these messages are like gold dust scattered on the floor. Once a nurse reads a text, the information is "used" and then disappears. No computer system is reading these texts to organize them, track trends, or help the hospital meet quality standards. It's all unstructured chaos.

The Goal: The authors wanted to build a robot that could read these messy, short, informal texts and automatically pull out the 4M clues, turning them into neat, organized data.

---

The Challenge: Why is this so hard?

You can't just ask a standard AI (like a generic chatbot) to do this. Why?

1. The texts are messy: They use abbreviations, typos, and slang.
2. They are short: "Restless. HR 110." is hard to interpret without context.
3. They are ambiguous: "No energy" could mean the patient is tired (Mobility) or depressed (Mentation).

If you just ask a smart AI to read these, it often guesses wrong or misses things entirely.

---

The Solution: The "Detective and the Editor" Team

The authors built a two-step pipeline called 4M-ER. Think of it as a team of two workers with very different jobs:

Step 1: The "Super-Scanner" (Bio-ClinicalBERT)

First, they use a specialized AI trained on medical texts. Let's call him The Scanner.

• His Job: He reads every single text message and highlights everything that might be important.
• His Style: He is very cautious. He would rather highlight 10 things and be wrong about 2 of them, than miss the one thing that matters. He has High Recall.
• The Result: He produces a long list of "candidate" clues. Some are right, but some are false alarms (like highlighting "DNS" thinking it meant "Do Not Resuscitate" when it actually meant a specific office).

Step 2: The "Smart Editor" (The Large Language Model)

Next, they bring in a second AI, a Large Language Model (LLM). Let's call her The Editor.

• Her Job: She doesn't read the raw messages from scratch. Instead, she looks at the list of highlights The Scanner made. She uses her deep understanding of language and context to edit the list.
• Her Superpower: She can tell the difference between a real clue and a false alarm.
  • Example: The Scanner highlighted "DNS office." The Editor looks at the whole sentence, realizes it's an office location, and says, "Delete that, it's not a medical clue."
  • Example: The Scanner highlighted "pain." The Editor looks at the context and says, "Actually, this is about the patient's preference to avoid pain, so let's label it 'What Matters' instead of just 'Medication'."
• The Result: She fixes the boundaries (making sure the highlighted text is the exact right phrase) and removes the mistakes.

The Magic Trick: The authors found that if they let the Editor do both jobs (find the clues AND fix them), it was slow and sometimes made mistakes. But if they let the Scanner do the finding and the Editor do only the fixing, the system became incredibly accurate and fast.

---

The "Silver" Training: Learning from Mistakes

The team realized they didn't have enough "perfect" examples to teach the Scanner everything. So, they created a Silver Labeling process.

• Imagine they had a mountain of unmarked texts.
• They used a powerful AI to guess the labels on these texts (creating "Silver" data).
• They taught the Scanner to learn from these guesses, but they were very strict, only keeping the guesses the AI was 100% sure about.
• The Result: This extra training helped the Scanner get much better at spotting tricky things like "Mobility" issues, which are often hard to describe in short texts.

---

Why This Matters (The "So What?")

This isn't just a tech demo; it changes how nursing homes work:

1. Real-Time Surveillance: Instead of waiting for a monthly report, the system can tell a doctor right now if a patient's mobility is dropping or if they are becoming confused, by reading the texts the staff is already sending.
2. Better Shift Handoffs: When a nurse leaves for the day, the system can automatically generate a summary: "Today, Mr. Smith had mobility issues and refused his meds." The next nurse gets a clean, organized report instead of digging through old texts.
3. Compliance: Hospitals are being judged on how well they care for the elderly (the 4M framework). This system turns messy texts into the official data needed to prove they are doing a good job.
4. Cost-Effective: They achieved this using open-source models that run on standard computers, not expensive supercomputers. It's a "frugal innovation" that any hospital can afford.

The Bottom Line

The authors built a system that acts like a smart filter. It takes the chaotic, informal chatter of nursing home staff and turns it into a clean, structured database. It does this by pairing a "catch-all" scanner with a "context-aware" editor, proving that you don't need a massive, expensive AI to solve complex medical problems—you just need the right team.

Technical Summary

1. Problem Statement

Nursing home (NH) interdisciplinary care teams generate vast amounts of clinically relevant data via secure text messages (TMs). This data often contains information aligned with the Age-Friendly Health Systems 4M framework (What Matters, Medication, Mentation, and Mobility). However, this information remains unstructured, fragmented, and ephemeral, making it impossible to monitor systematically, aggregate for quality reporting, or use for compliance with emerging CMS measures.

Extracting these entities is technically challenging due to the nature of TMs:

• Brevity and Informality: Messages are short, conversational, and lack formal structure.
• Ambiguity: Clinical context is often implied rather than explicit.
• Syntax: Messages contain abbreviations, typos, and fragmented sentences.

Existing solutions face trade-offs: traditional Named Entity Recognition (NER) models (like BERT) may lack contextual reasoning, while Large Language Models (LLMs) are often too computationally expensive to fine-tune or require massive resources for inference, and zero-shot prompting often fails on domain-specific clinical nuances.

2. Methodology: The 4M-ER Pipeline

The authors propose a multi-stage pipeline (4M-ER) that combines a high-recall token classifier with an inference-only LLM revision step. The system is designed to run entirely on locally deployed open-source models to ensure data privacy and reduce costs.

A. Dataset

• Source: 1,169 secure text messages from 16 Midwest nursing homes.
• Annotation: Expert-annotated by clinical researchers into four domains: What Matters, Medication, Mentation, Mobility.
• Split: ~75% Training/Validation, ~25% Testing.
• Silver Data: An additional 2,502 messages were weakly labeled using a "silver labeling" pipeline (LLM + lexical matcher + skeptic review) to augment training data for underrepresented domains.

B. Pipeline Architecture

The pipeline consists of three sequential stages:

1. Stage 1: Candidate Extraction (High-Recall Encoder)

   • Model: Fine-tuned Bio-ClinicalBERT.
   • Task: Token-level classification using BIO tagging to identify candidate 4M spans.
   • Strategy: Designed for high recall to ensure no potential entity is missed.
   • Filtering: Only messages containing at least one candidate span are passed to the next stage, significantly reducing computational load.

2. Stage 2: In-Context Exemplar Retrieval

   • Mechanism: A semantic retriever (using all-MiniLM-L6-v2) embeds candidate spans and retrieves the top 3 most similar "gold" examples from the training set.
   • Purpose: Provides the LLM with domain-specific context and few-shot examples to guide its revision.

3. Stage 3: LLM Revision (Precision Refinement)

   • Models: Four instruction-tuned, open-source LLMs were evaluated locally: Gemma-2-9B, Phi-3-Medium, Qwen-2.5-14B, and Mistral-Nemo.
   • Task: The LLM receives the original message, the candidate spans, and the retrieved exemplars. It performs:
     • Boundary Correction: Refining start/end indices.
     • Label Evaluation: Correcting misclassified entities.
     • Selection: Accepting valid spans and rejecting false positives.
     • Merging: Combining adjacent spans with the same label.
   • Constraint: High-confidence spans from Stage 1 are treated as "anchors" unless clearly contradicted by context.

C. Evaluation Metrics

• Primary Metric: F₁ score (ent_type). This metric tolerates minor boundary variations (common in short text) as long as the entity label and semantic overlap are correct.
• Baselines:
  • Zero-shot LLMs (Gemma, Phi, Qwen, Mistral).
  • Standalone fine-tuned Bio-ClinicalBERT.
  • A previously fine-tuned Gemma LLM (from prior work).

3. Key Results

Performance Improvements

• vs. Fine-tuned Gemma: The 4M-ER pipeline outperformed the fully fine-tuned Gemma baseline across all domains, achieving F₁ improvements of +2 to +11 percentage points without any additional model fine-tuning.
  • Example: Mobility F₁ improved from 0.55 (Fine-tuned Gemma) to 0.64 (4M-ER + Phi).
• vs. Zero-shot LLMs: Zero-shot models performed poorly (avg F₁ ~0.23), confirming that general LLMs cannot handle clinical TMs without domain adaptation or a structured pipeline.
• vs. Standalone Bio-ClinicalBERT: The pipeline improved precision significantly while maintaining high recall.
  • Mobility Precision: Increased from 0.63 (BERT) to 0.81 (BERT + Phi revision).
  • What Matters Precision: Increased from 0.46 to 0.53.

Error Analysis

• False Positives (FP): LLM revision reduced FPs by 25–35%, particularly in "What Matters" and "Medication," by correcting misclassifications caused by conversational ambiguity (e.g., distinguishing "DNS office" from "DNR").
• False Negatives (FN): The pipeline retained the high recall of the Bio-ClinicalBERT encoder, capturing subtle entities (e.g., implied patient refusals) that the standalone fine-tuned Gemma missed.
• Remaining Errors: The primary remaining error type was label-level misclassification (e.g., confusing "no energy" as Mobility vs. Mentation), reflecting genuine clinical ambiguity.

Efficiency and Robustness

• Resource Usage: The pipeline required roughly half the GPU memory (12 GB vs. 24 GB) compared to fine-tuning the Gemma model.
• Speed: Message filtering reduced the number of messages processed by LLMs by ~35%.
• Stability: The system demonstrated high reproducibility across 5 repeated runs (F₁ variation < ±0.02).

Impact of Silver Data Augmentation

Using silver-labeled data to pre-train the Bio-ClinicalBERT encoder further improved performance, especially in the hardest domains:

• What Matters: F₁ increased from 0.59 to 0.67.
• Mobility: F₁ increased from 0.64 to 0.67.

4. Key Contributions

1. Novel Architecture: Demonstrates that a hybrid encoder-decoder approach (Encoder for extraction + Inference-only LLM for revision) outperforms both standalone encoders and fully fine-tuned generative LLMs for clinical NER in informal text.
2. Resource Efficiency: Proves that high-performance clinical extraction is achievable with locally deployed, open-source models and significantly lower computational costs than full LLM fine-tuning.
3. Domain Adaptation: Successfully applied the 4M framework to secure text messages, a previously underutilized data source in long-term care, converting unstructured chat into structured, auditable data.
4. Methodological Validation: Validated the use of silver data augmentation and message filtering strategies to optimize the trade-off between accuracy and computational cost.

5. Significance and Implications

• Clinical Surveillance: Enables real-time monitoring of patient status across shifts and disciplines, allowing for early detection of decline in the 4M domains.
• Regulatory Compliance: Provides a mechanism to automatically aggregate 4M data for reporting to CMS and other quality bodies, turning unstructured chat logs into auditable evidence.
• Scalability: The pipeline's efficiency makes it feasible for nursing homes with limited IT infrastructure to adopt advanced NLP for quality improvement.
• Future Applications: The structured data output serves as a foundation for predictive modeling (e.g., predicting hospital transfers) and the construction of 4M-specific ontologies.

In conclusion, the study establishes that combining a high-recall token classifier with a lightweight, context-aware LLM revision step is a superior, cost-effective strategy for extracting critical clinical insights from the noisy, informal text messages of nursing home care teams.