Disagreement as a signal: an auditable dual-LLM and human-arbitrated workflow for methodological quality assessment in preclinical complex-intervention evidence

Technical Summary: Disagreement as a Signal in Auditable Dual-LLM Workflows

Problem Statement
The integration of Large Language Models (LLMs) into evidence synthesis workflows is accelerating, yet their application to methodological quality assessment remains challenging. Unlike simple information extraction, quality assessment involves complex judgments regarding non-specific factors, domain-specific controls, and context-dependent scoring rules. Current literature suggests that while LLMs can assist in screening or extraction, they cannot autonomously replace human assessors in high-stakes judgment tasks without risking reliability. A critical gap exists in the lack of a clearly defined, auditable workflow that leverages LLMs for pre-scoring while maintaining human oversight, particularly for complex interventions like preclinical animal studies where control conditions must match specific non-specific factors (e.g., heat, smoke, restraint).

Methodology
The authors developed and piloted an auditable, human-arbitrated dual-LLM pre-scoring workflow. The process was structured as follows:

1. Corpus Preparation: 16 full-text animal studies on moxibustion for exercise-induced fatigue were selected. PDFs were converted to Markdown using MinerU to enhance machine readability.
2. Structured Evidence Extraction: Evidence was extracted into structured tables containing source files, original line numbers, verbatim excerpts, and extraction rationales to ensure provenance.
3. Dual-LLM Pre-scoring: Two independent interactive LLM environments processed the same evidence packages and prespecified scoring rules. They generated initial scores, evidence locations, and rationales.
4. Arbitration Triggers:
   • Discrepancy Route: Items with score differences ($\Delta \ge 1$), ambiguous evidence, or boundary judgments were routed to human arbitration.
   • Agreement Route: Items where both LLMs agreed were not accepted as ground truth but were subjected to sampled review by human researchers to detect shared misclassifications.
   • Third-Party Review: Key boundary items underwent additional independent review.
5. Scoring Frameworks: The workflow applied three architectures:
   • SYRCLE: 9 items (Risk of Bias).
   • ARRIVE 2.0: 12 items (Reporting Quality).
   • Moxibustion-Specific Framework (MSF): 10 items (Complex-intervention operational quality, focusing on control design, dose reproducibility, and safety).

Key Results

• Workflow Metrics: The workflow generated 496 item-level assessments. The two LLMs achieved 79.2% exact agreement (393 items) and 20.8% disagreement (103 items). All discrepant items were arbitrated by humans.
• Human Intervention Value:
  • Among 50 sampled agreed items, 48 were confirmed and 2 were corrected (96.0% confirmation rate within the sample, not overall accuracy).
  • Human arbitration resolved discrepancies by returning to the evidence chain; the final decision matched LLM A in 73.8% of cases, LLM B in 23.3%, and differed from both in 2.9%.
  • Third-party review identified and corrected 8 systematic scoring inconsistencies.
• Disagreement Patterns: Disagreements concentrated on context-dependent and domain-boundary items, including inclusion/exclusion criteria, outcome reporting, baseline characteristics, specificity interpretation, and the matching of fixation/environment and sham controls.
• Methodological Insights: The structured matrix revealed recurrent weaknesses in the corpus: insufficient sham or non-acupoint controls, inadequate thermal/fixation matching, incomplete dose reproducibility (temperature, distance), and weak safety reporting.
• Construct Comparison: MSF percentage scores exceeded SYRCLE scores in all 16 studies (mean $\Delta = +23.9$ percentage points). This difference was driven primarily by the MSF's assessment of dose and operational reproducibility, constructs not captured by generic risk-of-bias tools.

Key Contributions

1. Auditable Workflow Design: The study proposes a transferable workflow where LLMs are constrained to evidence localization, pre-scoring, and discrepancy exposure, rather than autonomous assessment. The core components—structured provenance, dual-model discrepancy-triggered arbitration, sampled review of agreed items, and independent third-party review—are designed to be domain-agnostic.
2. Disagreement as a Signal: The paper reframes LLM disagreement not as noise or failure, but as a high-value signal indicating complex judgment requirements or insufficient evidence, effectively routing human effort to the most critical items.
3. Domain-Specific Framework (MSF): The development of the MSF demonstrates how generic tools (SYRCLE/ARRIVE) can be complemented by intervention-specific frameworks to capture operational blind spots (e.g., thermal dose reproducibility in moxibustion) that generic tools miss.

Significance and Claims
The authors position this study as a methodological pilot rather than a definitive validation of LLM accuracy.

• Role of LLMs: The study claims that LLMs are effective tools for structuring dispersed methodological clues and exposing uncertainty within a traceable evidence chain, but they must remain auxiliary to human arbitration.
• Limitations: The authors explicitly state that the 96% confirmation rate in sampled agreed items does not equate to overall model accuracy, as no fully independent dual-human gold standard was established. The MSF is in a pilot stage and requires external validation for reliability and validity.
• Generalizability: The workflow components are claimed to be transferable to other complex interventions (e.g., acupuncture, surgery), provided domain experts define specific scoring items and boundary rules before deployment.
• Conclusion: The primary contribution is the demonstration that a dual-LLM workflow, when coupled with rigorous human arbitration and provenance tracking, can generate auditable, comparable item matrices that reveal recurrent methodological patterns in complex-intervention research which might otherwise remain obscured.