💬 NLP

Reliable Multilingual Orthopedic Decision Support from Clinical Narratives: Language-Aware Adaptation and Verification-Guided Deferral

This paper presents a reliability-oriented framework for multilingual orthopedic decision support across English, Hindi, and Punjabi that leverages a domain-adaptive IndicBERT-HPA model and a verification-guided deferral mechanism to achieve superior classification performance and robustness compared to zero-shot LLMs and standard baselines.

Original authors: Danish Ali, Li Xiaojian, Sundas Iqbal, Farrukh Zaidi

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Danish Ali, Li Xiaojian, Sundas Iqbal, Farrukh Zaidi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Multilingual Translator with a Safety Net

Imagine a busy hospital in South Asia where patients speak three different languages: English, Hindi, and Punjabi. The doctors need to quickly sort through thousands of handwritten or typed notes to figure out what kind of orthopedic (bone and joint) problem a patient has.

Currently, most computer systems are like monolingual librarians: they are great at reading English books but get confused or make mistakes when the story is written in Hindi or Punjabi. They also struggle when the notes are messy, incomplete, or use local slang.

This paper introduces a new system designed to be a reliable, multilingual assistant that doesn't just guess, but knows when to say, "I'm not sure, please ask a human doctor."

1. The Problem: The "One-Size-Fits-All" Trap

The authors found that standard AI models are like generalist chefs. They can cook a basic meal (classify a text) in English, but when you give them ingredients in Hindi or Punjabi, or ask for a very specific regional dish, the quality drops.

The Challenge: Medical notes are messy. They might mix scripts (writing English words in Hindi letters), use incomplete sentences, or have unbalanced data (e.g., lots of notes about hip pain in English, but lots of notes about back pain in Punjabi).
The Risk: If an AI confidently guesses the wrong diagnosis, it could lead to bad care. The authors wanted a system that is not just "smart," but "safe."

2. The Solution: The "Specialized Adapter" (IndicBERT-HPA)

The team built a new AI model called IndicBERT-HPA.

The Analogy: Imagine a master translator (the base AI) who speaks all three languages fluently. However, this translator isn't a medical expert yet.
The Innovation: The authors added special "medical glasses" (called adapters) specifically for Hindi and Punjabi.
- When the AI reads English, it uses the master translator's standard knowledge.
- When it reads Hindi or Punjabi, it puts on the specialized "medical glasses" that help it understand local medical terms and sentence structures better.
The Result: This system became the best at sorting the notes. In their tests, it correctly categorized about 88% of the notes overall, outperforming other standard models and even the most advanced "chatbot" style AIs that were just asked to guess without any special training.

3. The "Zero-Shot" Chatbots vs. The Specialized Model

The researchers also tested popular, powerful AI chatbots (like DeepSeek, Mistral, and Zephyr) to see if they could do the job without any special training. They asked these chatbots to read a note and pick one of six categories (like "Spinal," "Hip," or "Bone").

The Analogy: Think of these chatbots as brilliant general knowledge students who have read millions of books but have never taken a medical exam.
The Result: When asked to do this specific medical task, they performed poorly (scoring around 61% accuracy). They were fluent and could write good sentences, but they were bad at making precise, structured medical decisions. The specialized model (IndicBERT-HPA) was much more reliable because it was specifically "trained" for this job.

4. The Safety Net: The "Verification Gatekeeper"

The most unique part of this paper is not just the AI that guesses, but the system that checks the guess.

The Analogy: Imagine a security checkpoint at an airport.
- Step 1: The AI (the traveler) makes a claim: "I am going to the Hip Department."
- Step 2: The Gatekeeper (the Verification Layer) checks three things:
  1. Confidence: "How sure are you?" (If the AI is only 50% sure, it gets stopped).
  2. Evidence: "Does the note actually mention hip symptoms?" (If the note is vague, it gets stopped).
  3. Language Risk: "Is the note written in a weird mix of scripts that looks suspicious?" (If yes, it gets stopped).
- Step 3: If everything looks good, the Gatekeeper says "Accept" (send to the doctor). If anything is shaky, it says "Defer" (send to a human doctor for review).
The Result:
- Without this gatekeeper, the system was right 71.5% of the time.
- With the gatekeeper, the system only "accepted" about 72% of the cases, but for those accepted cases, it was right 84.4% of the time.
- Crucially, this system reduced the number of wrong answers that were automatically accepted by about 60%. It essentially said, "I'm not confident enough to let this one through; let a human handle it."

5. What They Did NOT Claim

It is important to note what the paper doesn't say:

They did not say this system is ready to replace doctors.
They did not test this in a live hospital with real patients walking through the doors today.
They did not claim that the "chatbot" AIs would fail if they were trained differently (they only tested them in a "zero-shot" mode, meaning they were just asked to guess without learning first).

Summary

The paper presents a reliable, multilingual tool for sorting orthopedic notes in English, Hindi, and Punjabi. It uses a specialized model that adapts to local languages and a safety gatekeeper that refuses to make a decision if it's not sure. This approach ensures that when the computer does make a suggestion, it is highly likely to be correct, and when it isn't sure, it politely passes the task to a human.

Technical Summary: Reliable Multilingual Orthopedic Decision Support from Clinical Narratives

1. Problem Statement

Multilingual clinical decision support (CDS) faces significant hurdles in linguistically diverse, low-resource healthcare settings, particularly in South Asia where clinical interactions occur in regional languages (e.g., Hindi, Punjabi) while AI tools remain predominantly English-centric. Orthopedic decision support is a critical use case due to the nature of patient narratives, which often contain overlapping symptoms, incomplete evidence, heterogeneous terminology, and language-dependent documentation patterns.

Existing approaches struggle with several specific challenges:

Language-Conditional Instability: Model performance and calibration vary significantly across languages, especially under natural clinical prevalence distributions where class imbalance is severe.
Domain Specialization: General-purpose multilingual encoders and zero-shot Large Language Models (LLMs) often lack the specific grounding required for safety-sensitive, closed-set clinical categorization.
Reliability Gaps: Aggregate accuracy metrics obscure clinically meaningful failure modes, such as overconfident predictions on uncertain cases or language-specific biases.
Lack of Deferral Mechanisms: Current systems often lack explicit mechanisms to defer uncertain predictions for human review, a crucial requirement for safety-oriented CDS.

2. Methodology

The study proposes a reliability-oriented framework comprising a curated dataset, a domain-adaptive model architecture, and a deterministic post-prediction verification layer.

2.1 Dataset Construction

The authors constructed a large-scale, curated multilingual orthopedic clinical corpus containing 134,979 records in English, Hindi, and Punjabi. The data was sourced from a tertiary-care hospital in Pakistan (with collaboration from Indian hospitals) and includes naturally authored notes, not translations.

Diagnostic Labels: Six categories: Spinal, Musculoskeletal, Bone, Hip, Other, and Unknown (denoting insufficient evidence).
Evaluation Settings: Two distributions were defined:
1. Controlled Balanced: 18,000 records (1,000 per class per language) to enable fair cross-language comparison.
2. Natural-Prevalence: The full refined corpus reflecting realistic clinical label skew (e.g., Hip complaints dominate English/Hindi; Spinal dominates Punjabi).

2.2 Proposed Architecture: IndicBERT-HPA

To address domain and language mismatches, the authors introduced IndicBERT-HPA (Hindi–Punjabi Adapters).

Base: A shared IndicBERT encoder provides a common multilingual representation space.
Adaptation:
- English: Follows the direct shared-encoder path.
- Hindi & Punjabi: Pass through language-specific, lightweight orthopedic adapter heads. These adapters use a bottleneck residual transformation to project shared representations into a clinically discriminative space without duplicating the full encoder.
Training: The model is trained end-to-end using cross-entropy loss, jointly optimizing the shared encoder, adapters, and the classification head.

2.3 Baselines and Comparisons

The study compares IndicBERT-HPA against:

Supervised Encoders: XLM-RoBERTa, mDeBERTa, standard IndicBERT, and DistilBERT (fine-tuned on the task).
Zero-Shot LLMs: DeepSeek Open, Mistral-7B Instruct, and Zephyr-7B evaluated in a zero-shot, closed-label setting without task-specific fine-tuning or retrieval augmentation.

2.4 Deterministic Selective-Verification Layer

A post-prediction layer is implemented to separate diagnostic prediction from decision authorization. It does not modify model parameters but determines whether a prediction should be automatically accepted or deferred. The decision function $g$ relies on three signals:

Confidence Gating: Rejects predictions below a threshold $\tau$ (set to 0.60).
Evidence-Consistency Checking: Verifies if the predicted category is supported by symptom-diagnosis evidence rules.
Language-Risk Screening: Flags records with script mismatches, abnormal mixing, or excessive transliteration noise.
If any condition fails ( $s_i < \tau$ OR evidence $\neq$ supported OR risk = high), the case is deferred for human review.

3. Key Contributions

Curated Multilingual Orthopedic Corpus: A large-scale dataset spanning English, Hindi, and Punjabi, supporting both balanced and natural-prevalence evaluation to analyze class imbalance and language heterogeneity.
Domain-Adaptive Encoder (IndicBERT-HPA): An architecture that augments IndicBERT with language-aware orthopedic adapters, preserving shared representations while enabling targeted specialization for low-resource languages.
Reliability-Oriented Evaluation Protocol: A comprehensive evaluation extending beyond aggregate accuracy to include per-class performance, ROC-AUC, AUPRC, Expected Calibration Error (ECE), and cross-language stability under natural prevalence.
Deterministic Selective-Verification Layer: A rule-based system that combines confidence, evidence consistency, and language risk to authorize acceptance or defer uncertain cases, explicitly addressing safety and human-in-the-loop requirements.

4. Experimental Results

4.1 Supervised Encoder Performance

Controlled Setting: IndicBERT-HPA achieved the highest overall performance, with an averaged Macro-F1 of 0.8177, Macro-AUROC of 0.9302, and AUPRC of 0.9147. It consistently outperformed baselines in Musculoskeletal and "Unknown" (insufficient evidence) classification.
Natural-Prevalence Setting: IndicBERT-HPA maintained robustness, achieving an averaged Macro-F1 of 0.8792, Macro-AUROC of 0.894, and AUPRC of 0.902. It significantly outperformed all baselines, including XLM-RoBERTa and DistilBERT.
Calibration: While IndicBERT-HPA showed strong discrimination, it exhibited higher Expected Calibration Error (ECE) than some baselines in specific languages (e.g., English), motivating the need for the verification layer rather than relying on confidence alone.

4.2 Zero-Shot LLM Performance

Zero-shot LLMs performed substantially worse than task-adapted encoders for closed-set classification:

DeepSeek Open (the strongest LLM) achieved an averaged F1-score of 0.6121 under natural prevalence.
IndicBERT-HPA outperformed the best LLM by a large margin (0.8792 vs. 0.6121).
LLMs exhibited significant language-dependent instability and lacked the task-specific grounding required for structured clinical categorization without fine-tuning.

4.3 Selective Verification Outcomes

Evaluated on a held-out subset of 5,000 records:

Accept-All Baseline: 71.5% accuracy, 0.65 Macro-F1.
With Verification: The system accepted 72.3% of cases (deferring 27.7%) and achieved 84.4% selective accuracy and 0.76 selective Macro-F1.
Safety Impact: The layer reduced unsafe accepted errors by 60.4% compared to the accept-all policy, effectively routing uncertain cases for human review without modifying the underlying classifier.

5. Significance and Claims

The paper claims that multilingual orthopedic decision support requires a shift from autonomous diagnosis to reliability-oriented assistance. The study demonstrates that:

Domain Adaptation is Critical: Explicit domain adaptation (IndicBERT-HPA) significantly outperforms both general multilingual encoders and zero-shot LLMs in low-resource, multilingual clinical settings.
Natural Prevalence Matters: Evaluation under natural clinical distributions reveals failure modes and calibration issues that balanced benchmarks obscure.
Deferral Enhances Safety: Deterministic selective verification effectively improves the reliability of automatically accepted predictions by filtering out low-confidence or inconsistent cases, thereby reducing the risk of automated errors in clinical workflows.
Human-in-the-Loop Necessity: The results support a workflow where AI provides preliminary categorization with explicit uncertainty signals, deferring complex or risky cases to clinicians, rather than attempting fully autonomous diagnosis.

The authors emphasize that these findings are retrospective and subset-specific; prospective clinician-in-the-loop validation is required before deployment in real-world workflows. The study does not claim that zero-shot LLMs are ready for this specific task without adaptation, nor does it claim the verification layer replaces clinical judgment.