💻 computer science

Auditable Multi-LLM Evidence Synthesis for Contested Heritage Origins: Comparing Competing Hypotheses in the Turpan Karez Debate

This paper proposes an auditable, multi-LLM evidence synthesis framework that resolves the contested origins of Turpan's Karez system by quantitatively evaluating competing hypotheses, ultimately identifying the Central Asia Introduced hypothesis as the most supported explanation based on a comprehensive analysis of fragmented multilingual literature.

Original authors: Xiangping Qi, Muhammad Irfan Malik, Zhongjing Wang, Jingping Zhang, Shaolan Wang, Hongjian Gan, Kanghui Li, Yixuan Shi

Published 2026-11-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiangping Qi, Muhammad Irfan Malik, Zhongjing Wang, Jingping Zhang, Shaolan Wang, Hongjian Gan, Kanghui Li, Yixuan Shi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the arid landscapes of Central Asia, where water is the most precious resource, a remarkable engineering feat has sustained life for centuries: the karez. These are not simple wells, but complex underground systems that tap into mountain aquifers and channel water through gently sloping tunnels to the plains below, bringing life to oases without the loss of water to evaporation. For generations, scholars have debated the true origin of this technology in the Turpan Basin of China. The question is not merely academic; it touches on how human societies have historically adapted to harsh environments and how ideas and technologies have moved across the world. Three main theories have long competed for acceptance. One suggests the technology was brought from the Central Plains of China; another argues it arrived from Central Asia, carried along ancient trade routes; and a third posits that the local people of Turpan invented it independently, perfectly adapted to their unique geography. The problem is that the historical record is a tangled web of fragmented documents, repeated citations, and conflicting interpretations, making it nearly impossible to determine which story is the most accurate.

To cut through this confusion, a team of researchers from universities in China, Pakistan, and the United States devised a new way to sift through the evidence. They treated the debate not as a battle of opinions, but as a data problem, building a digital framework to organize thousands of pages of historical literature written in six different languages. Instead of relying on a single expert to read every book and form an opinion, they used a team of advanced artificial intelligence models to read the texts simultaneously. These models were instructed to act like careful archivists, pulling out specific sentences that mentioned the origin of the karez, checking exactly where those sentences came from, and verifying that the text actually supported the claim being made. Crucially, the system was designed to be auditable, meaning every piece of evidence could be traced back to its original page and paragraph, ensuring that no information was lost or invented.

The researchers gathered 275 full-text publications, ranging from ancient travelogues to modern scientific studies, and fed them into their system. The artificial intelligence models identified hundreds of potential clues, which were then cross-checked against the original source texts to ensure accuracy. The system grouped these clues into "families," recognizing that when ten different books all cite the same original document, they represent only one piece of evidence, not ten. This prevented the debate from being skewed by the sheer volume of repeated citations. After this rigorous cleaning and verification process, the team was left with a solid set of verified evidence to evaluate the three competing theories.

The results of this massive digital synthesis revealed a clear, though nuanced, picture. The theory that the karez technology was introduced from Central Asia emerged as the most strongly supported hypothesis based on the available literature. It accumulated the largest number of independent evidence families, suggesting that the idea of an external origin has the broadest backing in the historical record. However, the study also found that the theory of local invention remained a very strong contender. While it had fewer total pieces of evidence, the evidence it did have was more diverse and faced less opposition from other scholars. The third theory, which claimed the technology came from China's Central Plains, received the least support in this specific analysis.

What makes this finding significant is not just the ranking of the theories, but the clarity with which the study presents the uncertainty. The researchers did not declare the Central Asian origin as an absolute, unchangeable fact. Instead, they showed that while it currently holds the lead, the gap between it and the local invention theory is narrow enough that new discoveries could shift the balance. The study highlights that the debate has been hindered not by a lack of information, but by a lack of a systematic way to organize that information. By creating a transparent, step-by-step process that anyone can follow and verify, the team has provided a new standard for how to investigate the origins of cultural heritage. They have shown that even when a "smoking gun" piece of evidence is missing, a careful, structured synthesis of all available knowledge can reveal which story is currently the most plausible, offering a clear path forward for future historians and archaeologists to build upon.

Technical Summary: Auditable Multi-LLM Evidence Synthesis for Contested Heritage Origins

Problem Statement
The origin of the Turpan Karez (underground irrigation systems) in China remains a subject of intense scholarly debate, centered on three competing hypotheses: the Central Asia Introduced (CAI) hypothesis, the Central Plains Introduced (CPI) hypothesis, and the Local People Innovated (LPI) hypothesis. This debate is hindered by fragmented multilingual literature (spanning Chinese, English, Russian, French, Persian, and Arabic), circular citation practices where the same primary source is repeatedly cited across multiple publications, and inconsistent evidentiary standards. Crucially, no single "golden evidence" exists to definitively confirm or veto any hypothesis. Traditional computational tools (e.g., CiteSpace, VOSviewer) are effective for bibliometric mapping but lack the capacity to extract substantive scholarly claims, distinguish between original interpretations and cited viewpoints, or validate evidence against source texts. Furthermore, existing Large Language Models (LLMs) often suffer from hallucinations and weak source traceability when processing complex historical corpora.

Methodology
The authors propose an auditable, multilingual, multi-LLM evidence-synthesis framework designed to evaluate competing historical hypotheses through a structured, human-in-the-loop workflow. The methodology consists of five primary stages:

Multilingual Corpus Construction: A three-stage progressive search strategy was employed across eight databases (including Web of Science, Scopus, CNKI, and eLIBRARY.RU) using Boolean logic to broaden and then narrow search terms (Core terms $\rightarrow$ Thematic terms $\rightarrow$ Geographic terms). This yielded a final corpus of 275 full-text publications in six languages, processed according to PRISMA 2020 guidelines.
Traceable RAG Retrieval: Full texts were cleaned, normalized, and hierarchically chunked (targeting ~1,000 characters). A Retrieval-Augmented Generation (RAG) system utilizing Reciprocal Rank Fusion (RRF) combined BM25 and TF-IDF retrieval scores to identify candidate passages. Eight multilingual queries were executed, resulting in 141 unique target chunks (plus context) selected for evidence extraction.
Parallel Multi-LLM Evidence Extraction: Three distinct LLMs (DeepSeek V4 Pro, GLM-5.2, and Qwen3.7-Plus) independently processed the same input chunks. They were instructed to extract structured evidence (origin location, date, stance, evidence type) strictly from the provided text, prohibiting the use of parametric knowledge or external tools. This generated 727 unique candidate evidence records.
Cross-Model Validation and Human Adjudication: A rigorous validation pipeline was implemented:
- Source-Constrained Validation: The three LLMs re-evaluated candidates against the source text to verify if claims were "entailed," "contradicted," or "insufficient."
- Human Review: Records receiving $\ge$ 2 model confirmations underwent blind review by two human researchers. Records were classified as Verified (both agree), Uncertain (disagreement), or Rejected.
- Evidence Family Consolidation: To prevent evidence inflation from repeated citations, verified records were grouped into 484 Evidence Families based on shared primary sources, historical events, or citation lineages.
Hypothesis Evaluation and Robustness Analysis: The three hypotheses were scored across four dimensions: Evidence Sufficiency (number of supporting families), Net Evidence Balance (support vs. opposition), Scholarly Consensus (authorial stance distribution), and Evidence-Type Diversity (Shannon index across documentary, material, and legendary evidence). Robustness was tested via weight perturbation, leave-one-model-out analysis, and 10,000 family-level bootstrap iterations.

Key Results

Evidence Volume: From 275 documents, the framework extracted 633 Verified evidence records (87.1% of candidates), 85 Uncertain, and 9 Rejected.
Model Complementarity: The three LLMs demonstrated significant complementarity; 75.65% of unique candidate records were identified by only one model, highlighting the necessity of a multi-model approach.
Hypothesis Scoring:
- CAI (Central Asia Introduced): Achieved the highest Evidence Sufficiency (116 supporting families) and the highest Composite Score (0.5735).
- LPI (Local People Innovated): Achieved the highest Net Evidence Balance (0.862), Scholarly Consensus (0.778), and Evidence-Type Diversity (0.410), resulting in a Composite Score of 0.5125.
- CPI (Central Plains Introduced): Received the lowest scores across all dimensions, with a Composite Score of 0.3321.
Robustness: In 82.6% of 10,000 bootstrap iterations, CAI ranked first. However, the ranking was sensitive to specific conditions: when uncertain evidence was recoded against the leading hypothesis, or when the DeepSeek model was excluded, LPI occasionally ranked first. The ranking was not driven by any single Evidence Family.

Significance and Claims
The paper claims to offer a new paradigm for investigating heritage origins in the absence of decisive archaeological evidence. Its primary contributions are methodological rather than definitive historical conclusions:

Auditability and Traceability: The framework makes the entire evidence synthesis process transparent, linking final scores back to specific page numbers, text chunks, and source documents, thereby addressing the "black box" nature of AI in historical research.
Handling Contested Origins: It provides a structured mechanism to compare competing hypotheses by quantifying evidence volume, balance, consensus, and diversity, moving beyond simple citation counts.
Human-LLM Collaboration: The study demonstrates that LLMs are effective for scalable evidence discovery and extraction but require human adjudication for historical interpretation and validity checks.
Modest Historical Conclusion: The authors explicitly state that their results indicate relative evidentiary support within the current corpus and framework, not a definitive historical truth. They conclude that CAI currently holds a relative advantage in the evidence base, but LPI remains a strong competing explanation, and the debate likely involves different historical stages (transmission vs. local adaptation) that are not mutually exclusive.

The framework is presented as a tool to enhance the systematic organization and reproducibility of large-scale literature-based evidence synthesis, offering a transparent basis for reassessing interpretations as new evidence emerges.

Technical Summary: Auditable Multi-LLM Evidence Synthesis for Contested Heritage Origins

More like this