Reliability-Oriented Multilingual Orthopedic Diagnosis: A Domain-Adaptive Modeling and a Conceptual Validation Framework

Technical Summary: Reliability-Oriented Multilingual Orthopedic Diagnosis

Problem Statement
Multilingual clinical decision support faces significant challenges in linguistically diverse, low-resource healthcare environments, particularly where electronic health records and AI tools are predominantly English-centric while clinical interactions occur in regional languages like Hindi and Punjabi. This language mismatch increases the risk of misinterpretation and inconsistent care delivery. Orthopedic diagnosis serves as a high-risk testbed for this issue due to overlapping symptoms, heterogeneous terminology, and the critical need for reliable, structured classification rather than open-ended generation. Existing approaches often rely on general-purpose multilingual encoders or zero-shot Large Language Models (LLMs), which the authors argue lack sufficient domain specialization, exhibit language-conditional instability, and suffer from calibration gaps in safety-critical contexts.

Methodology
The study proposes a system-level analysis comparing three modeling regimes against zero-shot LLMs for orthopedic diagnosis across English, Hindi, and Punjabi.

1. Dataset Construction: The authors curated a large-scale multilingual orthopedic corpus from Bahawalpur Victoria Hospital and Indian partners, comprising approximately 135,000 clinical notes. The data underwent rigorous refinement, including deduplication, noise filtering, and label alignment by licensed orthopedic physicians. Two evaluation settings were established:

   • Controlled (Balanced): 18,000 notes evenly distributed across languages and six diagnostic categories (Spinal, Musculoskeletal, Bone, Hip, Other, Unknown).
   • Natural (Real-World): ~135,000 notes retaining original class skew and prevalence to simulate deployment conditions.

2. Modeling Regimes:

   • Task-Only Fine-Tuning: Evaluation of general-purpose multilingual encoders (XLM-RoBERTa, IndicBERT, mDeBERTa) and a lightweight English-centric baseline (DistilBERT) fine-tuned end-to-end on the orthopedic task.
   • Domain-Adaptive Architecture (IndicBERT-HPA): A proposed extension of IndicBERT that integrates language-specific orthopedic adapter heads. This architecture uses bottleneck projections to map general linguistic representations into a clinically discriminative orthopedic subspace while preserving cross-lingual transfer capacity.
   • Zero-Shot LLMs: Evaluation of instruction-tuned LLMs (DeepSeek Open, Mistral-7B Instruct, Zephyr-7B) in a zero-shot setting to assess their generative fluency versus structured diagnostic reliability without fine-tuning.

3. Evaluation Protocol: Performance was measured using precision, recall, F1-score, accuracy, ROC-AUC, and AUPRC. Crucially, the study incorporated Expected Calibration Error (ECE) to assess confidence reliability.

4. Conceptual Validation Framework: Building on empirical failure modes, the authors propose a deterministic, agent-based validation layer. This framework separates diagnosis generation from decision authorization, employing evidence-checking agents, language-consistency checks, and conservative human-in-the-loop gating to defer low-confidence or contradictory predictions.

Key Contributions

1. Curated Multilingual Dataset: The creation of a large-scale, physician-validated orthopedic dataset spanning English, Hindi, and Punjabi, supporting both balanced and natural distribution analysis.
2. IndicBERT-HPA Architecture: The introduction of a domain-adaptive model that explicitly models domain structure via language-specific adapters, demonstrating that architectural specialization can outperform scale in low-resource clinical settings.
3. Reliability-Oriented Evaluation: A comprehensive protocol that moves beyond aggregate accuracy to analyze per-class performance, calibration behavior (ECE), and cross-lingual instability, highlighting that high discrimination does not guarantee safe deployment.
4. Conceptual Validation Framework: A theoretical design for a deterministic agent-based system that enforces evidence consistency and conservative gating, positioning the separation of prediction and authorization as a core requirement for safety-critical systems.

Results

• Zero-Shot LLMs: Despite strong linguistic fluency, zero-shot LLMs exhibited unstable calibration, weak label grounding, and significant cross-lingual variability. Their performance was consistently lower than supervised encoders, particularly in low-resource languages (Punjabi) and for structured classification tasks.
• Task-Only vs. Domain-Adaptive: While task-fine-tuned models (like DistilBERT) achieved strong aggregate scores, they showed brittle calibration and uneven performance across languages. IndicBERT-HPA achieved the strongest overall discrimination (Macro-F1 ~0.88–0.89 in real-world settings) and the most consistent cross-lingual behavior. It demonstrated superior ranking quality (AUROC/AUPRC) and more predictable confidence signals compared to baselines.
• Calibration and Robustness: The study found that models with high AUROC could still suffer from overconfidence (high ECE). IndicBERT-HPA, while showing moderate calibration error, maintained stable separability, which the authors argue is more critical for safety than raw accuracy, as miscalibration can be corrected post-hoc while poor discrimination cannot.
• Real-World Performance: Under natural class imbalance, supervised encoders significantly outperformed zero-shot LLMs. However, all models showed sensitivity to distributional shifts, reinforcing the need for explicit validation layers.

Significance and Claims
The paper argues that reliable multilingual clinical decision support cannot rely solely on model scale or generative fluency. Instead, it requires a systems-level approach that integrates:

1. Domain-Adaptive Specialization: Explicit architectural alignment with medical taxonomy and language constraints is more effective than general-purpose pretraining for structured tasks.
2. Reliability-Aware Evaluation: Metrics must include calibration and per-class stability, not just aggregate accuracy, to identify failure modes in safety-critical contexts.
3. Governance and Validation: The separation of prediction generation from decision authorization via deterministic, rule-based validation layers is essential for managing risk, ensuring auditability, and enabling human-in-the-loop oversight.

The authors conclude that while LLMs show promise in generative tasks, their zero-shot application to structured, high-risk multilingual diagnosis is currently insufficient. The proposed IndicBERT-HPA architecture and the conceptual validation framework offer a pathway toward safer, more reliable clinical AI systems that prioritize accountability and risk mitigation over autonomous decision-making.