TreeProbe : A Tibetan Medicine Benchmark for Cultural Bias in LLMs

Technical Summary: TreeProbe – A Tibetan Medicine Benchmark for Cultural Bias in LLMs

Problem Statement
Large Language Models (LLMs) are increasingly proposed as tools to mitigate global health inequities. However, current models are predominantly trained on Western-centric, high-resource corpora, leading to a systematic underrepresentation of localized cultural contexts and region-specific medical knowledge. This issue is particularly acute in Tibetan medicine, one of the world's four major traditional medical systems, which possesses an independent and highly structured theoretical framework. When LLMs lack a grounded understanding of Tibetan medical epistemology, they tend to default to dominant external systems—specifically biomedicine or Traditional Chinese Medicine (TCM)—during reasoning. This "ontology drift" distorts native knowledge structures, potentially reinforcing existing hierarchies of knowledge and exacerbating healthcare inequities. A critical gap exists in the lack of quantitative tools to evaluate and diagnose this specific form of cultural bias within Tibetan medical contexts.

Methodology: The TreeProbe Benchmark
To address this gap, the authors introduce TreeProbe, the first cultural-bias benchmark grounded in the native "Medical Tree" framework of Tibetan medicine, as codified in the Four Tantras. The benchmark is structured around three roots: Physiopathology, Diagnosis, and Therapy.

• Dataset Construction: TreeProbe comprises 4,719 expert-adjudicated items covering 467 diseases and 10 subtasks (including both Multiple-Choice Questions and Generative QA). The construction process follows a four-stage pipeline:

  1. Knowledge Unit Extraction: Atomic knowledge units are extracted from authoritative sources (the Four Tantras and the national standard GB/T 46946–2025) by native Tibetan-speaking annotators. Each unit is a structured quadruple linking source evidence, disease codes, and attribute vectors covering etiology, pathogenesis, diagnostics, and treatment.
  2. Operator-Guided Instantiation: Items are derived using a family of probing operators that map specific attribute slots to target questions. This ensures difficulty arises from reasoning rather than information leakage.
  3. Cross-Ontology Distractor Authoring: To rigorously test for bias, the benchmark employs a specific distractor taxonomy:
     • Faithful Answer (FA): The correct answer under Tibetan medical reasoning.
     • Intra-Tibetan Distractor (ITD): Uses correct Tibetan terminology but violates the specific relational logic required by the question.
     • TCM-drift Distractor (TCD): Plausible under TCM but incorrect under Tibetan theory.
     • Biomedical-drift Distractor (BID): Aligns with biomedical classification but is incorrect in the Tibetan context.
       Crucially, distractors are authored by domain experts (TCM and biomedical physicians) using ontology-neutral clinical vignettes to prevent model self-preference bias.
  4. Adjudication: Items undergo strict filtering for ontological separability and safety, followed by blind adjudication by a panel of five independent experts (three Tibetan, one TCM, one biomedical). An item is retained only if at least four experts agree.

• Evaluation Metrics: The paper defines two log-odds metrics to quantify bias:

  • Ontology Drift Log-Odds (ODLO): Measures the overall departure from the Tibetan ontology (drift toward external systems).
  • Biomedical–TCM Drift Log-Odds (BTD-LO): Characterizes the direction of external drift (positive for biomedical, negative for TCM).

Experimental Results
The authors evaluated six representative frontier LLMs (including GPT-5.4, Claude Sonnet 4.6, Gemini 3.1, GLM-4.7, DeepSeek-V3.2, and Qwen3.5-397B-A17B) on TreeProbe.

• Performance Limitations: Current models exhibit significant limitations in native Tibetan medical contexts. Even the strongest zero-shot model, Qwen3.5-397B-A17B, achieved an average accuracy of only 60.55% on MCQs, while performance on generative QA was substantially lower (average scores around 0.47–0.55 on a normalized [0, 1] scale).
• Systematic Ontology Drift: Analysis of error patterns reveals that models do not merely fail due to a lack of knowledge; they systematically drift toward external ontologies. Every model tested showed an ODLO > 0.48, indicating that errors consistently take the form of substitution with external medical systems rather than confusion within Tibetan internal alternatives.
• Directional Bias: Models diverge in their drift direction based on pretraining data composition and surface semantic similarity:
  • Biomedical-leaning: Models like Claude Sonnet 4.6 and GPT-5.4 showed a strong tendency to drift toward biomedical reasoning.
  • TCM-leaning: Models like GLM-4.7, DeepSeek-Chat, and Gemini showed a stronger drift toward TCM. The authors attribute this to the high surface semantic similarity between Tibetan and Chinese medical concepts, which lowers the "cost" of substitution even for models without disproportionate TCM exposure.
• Linguistic vs. Ontological Competence: In generative tasks, models demonstrated high "Linguistic Quality" (fluency in Tibetan) but low "Ontology Fidelity." This indicates that models can generate grammatically correct Tibetan text while failing to preserve the underlying medical reasoning or substituting it with external paradigms.
• Controlled Diagnostic Tests: Experiments where models were forced to choose between correct Tibetan, TCM, and biomedical answers revealed that even when models demonstrated access to the correct Tibetan answer, they frequently reverted to their preferred external paradigm. In some cases, models selected clinically incorrect answers from their preferred external system over the correct Tibetan answer, suggesting a deep-seated preference for external ontologies.

Key Contributions

1. TreeProbe Benchmark: The introduction of the first cultural-bias benchmark specifically for Tibetan medicine, organized around the native Medical Tree framework, containing 4,719 expert-adjudicated items.
2. Construction Paradigm: A reusable methodology for low-resource traditional medicine evaluation, utilizing structured knowledge units and expert-authored cross-system distractors to isolate ontology drift from knowledge gaps.
3. Systematic Characterization of Bias: A comprehensive evaluation of frontier LLMs that reveals the mechanisms of cultural bias, demonstrating that errors are not random but follow structured patterns of ontology drift toward biomedicine or TCM, shaped by pretraining data and surface semantic similarities.

Significance and Claims
The paper positions TreeProbe as a diagnostic tool essential for developing medical AI systems that are both linguistically inclusive and epistemically fair. The authors claim that without such rigorous evaluation, unchecked cultural bias in LLMs risks transforming these models into mechanisms that overwrite or replace vulnerable knowledge systems with dominant medical paradigms. By quantifying and tracing these biases, TreeProbe provides a foundation for model diagnosis, bias attribution, and alignment, ensuring that AI applications in healthcare do not inadvertently marginalize traditional medical traditions. The authors explicitly state that TreeProbe is a research benchmark for evaluation and bias diagnosis, not a clinical decision-support system.