💬 NLP

Using Human-LLM Disagreement to Improve Checklist-Based Quality Appraisal

This paper demonstrates that analyzing human-LLM disagreement on checklist-based quality appraisal can identify ambiguous criteria, and that revising these items significantly improves agreement while preserving the relative ranking of studies, thereby supporting more reliable LLM-assisted research synthesis workflows.

Original authors: Timo van der Kuil (Methodology and Statistics Utrecht University), Bruno Messina Coimbra (Methodology and Statistics Utrecht University), Mirjam van Zuiden (Clinical Psychology Utrecht University), Ro

Published 2026-08-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Timo van der Kuil (Methodology and Statistics Utrecht University), Bruno Messina Coimbra (Methodology and Statistics Utrecht University), Mirjam van Zuiden (Clinical Psychology Utrecht University), Robert A. Bagheri (Methodology and Statistics Utrecht University), Rens van de Schoot (Methodology and Statistics Utrecht University), Klaas Dieleman (Methodology and Statistics Utrecht University), Berend Greijn (Methodology and Statistics Utrecht University), Stefan Houkes (Methodology and Statistics Utrecht University), Sebastiaan Rodenhuis (Methodology and Statistics Utrecht University), Elizabeth M. Grandfield (Methodology and Statistics Utrecht University)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every year, the world of science produces a flood of new research, creating a challenge for those who try to make sense of it all. When scientists want to understand the full picture of a topic, they conduct a systematic review, a process that involves gathering every relevant study on a subject and checking how well each one was done. This quality check is crucial because it separates reliable findings from flawed ones, but it is also incredibly slow and mentally exhausting. Experts must read through dense reports and decide if the researchers followed the rules, a task that can take an hour per study and is prone to mistakes when people get tired. Recently, powerful computer programs known as large language models have emerged as potential helpers for this work. These programs can read and understand text, leading to a hopeful idea: could a computer take over the tedious job of checking study quality? However, simply handing a checklist to a computer is not enough. If the questions on the checklist are vague or confusing, even a smart computer will struggle to give the right answer, just as a human would.

A team of researchers set out to test this idea and, more importantly, to see how they could make the computer and the human experts agree better. They focused on a specific set of rules called the GRoLTS checklist, which is used to judge studies that track how groups of people change over time. The researchers first asked several different computer models to apply these rules to a collection of real studies about trauma, while human experts had already graded the same studies. They found that the computers were not perfect; they agreed with the humans on some questions but often disagreed on others. The disagreements were not random. They happened most often when a checklist item was worded in a way that allowed for multiple interpretations, such as asking if something was done for "all" models when the study only reported on some, or using broad terms that were hard to pin down. The computer would often find a piece of evidence and say "yes," while the human expert, looking for a stricter standard, would say "no."

Instead of just accepting these errors, the researchers used the disagreements as a map to fix the checklist itself. They took the confusing questions and rewrote them to be clearer and more specific. They broke down complex questions that asked two things at once into separate, simpler questions. They removed "if-then" scenarios that required guessing and replaced vague terms with concrete examples of what to look for. Once they had created this improved, second version of the checklist, they asked the computers and the human experts to grade the studies again. The results showed a clear improvement. The computers and humans agreed much more often on the new version. The confusion that had caused the computers to stumble was largely gone. Even more importantly, the researchers found that the computers were still very good at sorting the studies from best to worst, even if they made a few small mistakes on individual questions. This means that a computer can reliably help researchers prioritize which studies to look at first, as long as the rules it follows are written clearly.

The study suggests that the key to making artificial intelligence useful for scientific review is not just choosing a smarter computer program, but designing better questions for it to answer. When the rules are ambiguous, both humans and machines struggle, but when the rules are precise, the machines become reliable partners. The researchers did not find that computers could replace human experts entirely, as some difficult questions still required human judgment. Instead, they showed that by analyzing where the computer and the human disagreed, they could identify the weak spots in their tools and fix them. This approach turns a mismatch into a tool for improvement, suggesting that the future of scientific review lies in a partnership where computers handle the clear-cut tasks and humans focus on the complex ones, guided by checklists that are designed for both to understand.

Technical Summary: Using Human-LLM Disagreement to Improve Checklist-Based Quality Appraisal

Problem Statement
Systematic reviews require rigorous quality appraisal of included studies, a process that is labor-intensive, cognitively demanding, and prone to fatigue-related errors. While Large Language Models (LLMs) offer potential for automation, their application in this domain is hindered by the ambiguity inherent in checklist criteria. Existing research typically treats appraisal checklists as fixed inputs, focusing on prompt engineering or model selection without addressing whether the checklist design itself contributes to disagreement between human experts and LLMs. Furthermore, it remains unclear whether LLM-generated scores preserve the relative ranking of studies by quality, a critical requirement for prioritization in evidence synthesis.

Methodology
The study utilized the Guidelines for Reporting on Latent Trajectory Studies (GRoLTS) checklist, a 21-item binary instrument designed for human experts. The research employed a two-stage experimental design across three distinct domains: Post-Traumatic Stress Disorder (PTSD), Educational Achievement, and Adolescent Delinquency.

Pipeline Implementation: A Retrieval-Augmented Generation (RAG) pipeline was constructed to apply the checklist to full-text articles. This pipeline involved:
- Preprocessing: Converting PDFs to Markdown to preserve structural elements.
- Segmentation: Dividing text into overlapping chunks (1,000 words with 50-word overlap).
- Embedding & Retrieval: Using the Qwen3-Embedding-8B model to generate vector representations for both checklist items and text chunks. For each item, the top-10 semantically relevant chunks were retrieved via cosine similarity.
- Generation: A standardized prompt instructed five different LLMs (GPT-5-mini, LLaMA 3.3 70B, Qwen3 30B, Qwen3-Next 80B, and Magistral Small) to generate binary (Yes/No) judgments based solely on the retrieved context.
Evaluation Strategy:
- Stage 1 (Baseline): Human annotations (from the original GRoLTS study) were compared against LLM outputs using the original checklist (v1) in the PTSD domain.
- Stage 2 (Revision & Validation): Based on Stage 1, the checklist was revised (v2) to address sources of disagreement. The revised checklist was applied to all three domains.
- Metrics: Agreement was measured using exact-match accuracy, chance-corrected metrics (Cohen's $\kappa$ and Fleiss' $\kappa$ ), and Spearman's rank correlation to assess the preservation of study-level ordering.

Key Contributions
The primary contribution of this work is the demonstration that analyzing human-LLM disagreement serves as a diagnostic tool for improving appraisal instruments, rather than merely evaluating model performance. The study identified specific structural flaws in checklist design—such as conditional phrasing, double-barreled questions, and ambiguous scope—that drive disagreement. By systematically revising these items (e.g., splitting items to distinguish partial vs. complete reporting and removing conditional sub-items), the authors developed a revised checklist (v2) that significantly improved alignment between human and automated judgments.

Results

Item-Level Performance: In the original checklist (v1), agreement varied drastically, with some items showing near-perfect accuracy and others as low as 0.10. Items with low accuracy were characterized by ambiguity and conditional logic.
Impact of Revision: The revised checklist (v2) yielded a marked improvement. Mean accuracy for the lowest-performing items increased from ~0.10–0.20 (v1) to 0.40–0.60 (v2). The proportion of items achieving high accuracy (0.7–1.0) increased, and variability across different LLMs decreased.
Chance-Corrected Agreement: Cohen's $\kappa$ values for v2 ranged from 0.41 to 0.72 (moderate to substantial agreement), an improvement over the 0.31–0.52 range observed in v1. Fleiss' $\kappa$ across all raters (five LLMs and one human) improved from 0.41 (v1) to approximately 0.60 (v2).
Rank Preservation: While item-level misclassifications persisted, LLM-generated total scores maintained a moderate-to-strong correlation with human rankings (Spearman $\rho$ ranging from 0.36 to 0.72). Crucially, the study found that rankings derived from high-agreement items alone showed strong correlation with human rankings ( $\rho \approx 0.58–0.93$ ), whereas low-agreement items introduced instability.

Significance and Claims
The paper claims that the reliability of LLM-assisted quality appraisal is not solely a function of model capability but is heavily dependent on the clarity and structure of the appraisal instrument. The authors posit that human-LLM disagreement is a useful signal for identifying problematic checklist items, enabling an iterative process to refine research synthesis workflows.

The study suggests a "selective-automation" strategy: LLMs can effectively handle well-specified criteria to preserve the relative ranking of studies for prioritization, while human reviewers should focus on resolving ambiguous or conceptually complex items. This approach allows for the integration of LLMs into systematic reviews without requiring perfect item-level accuracy, provided the checklist is designed to minimize ambiguity. The findings support the broader utility of LLMs in evidence synthesis, provided that the tools they use are co-designed with both human and automated constraints in mind.

Technical Summary: Using Human-LLM Disagreement to Improve Checklist-Based Quality Appraisal

More like this