SPIRIT-CONSORT-ELM: Element-Level Assessment of Randomized Controlled Trial Reporting Using Large Language Models

Technical Summary: SPIRIT-CONSORT-ELM

Problem Statement
Randomized controlled trials (RCTs) are fundamental to assessing intervention benefits and harms, yet their utility is frequently compromised by incomplete reporting. While reporting guidelines such as SPIRIT (for protocols) and CONSORT (for results) provide checklists to ensure completeness, existing automated assessment methods operate primarily at the "item level." This binary approach—determining only whether a checklist item is present or absent—fails to capture the nuance of reporting completeness. Many checklist items consist of multiple distinct "elements" (specific details required for full compliance). Consequently, an item may be marked as "reported" even if critical sub-elements are missing, or marked "incomplete" without distinguishing between a single missing detail and a total absence of information. This lack of granularity hinders the ability to provide actionable feedback to authors and limits the reliability of meta-research into reporting trends.

Methodology
The authors address this gap by developing SPIRIT-CONSORT-ELM, a framework and dataset for element-level assessment. The methodology involves three primary components:

1. Framework Construction: Three experts in epidemiology and meta-research derived 119 specific questions from the 83 checklist items of SPIRIT 2013 and CONSORT 2010. These questions target constituent elements within checklist items. The framework includes single-choice ("check-one"), multiple-choice ("check-all-that-apply"), and conditional questions. Conditional questions are triggered only when specific elements are present or absent, requiring further detail or rationale.
2. Dataset Curation: The framework was applied to the existing SPIRIT-CONSORT-TM corpus, comprising 200 articles (100 protocol-results pairs). Two experts independently annotated 50 articles (25 pairs) to establish inter-annotator agreement, while the remaining 150 articles were annotated by a single expert. Annotations were based on relevant text snippets previously identified for the corresponding checklist items.
3. Automated Assessment Pipeline: The authors formulated element-level assessment as a machine reading comprehension (MRC) task. The pipeline consists of:
   • Evidence Retrieval: A PubMedBERT-based model (fine-tuned on SPIRIT-CONSORT-TM) identifies sentences containing item-level reporting information.
   • Question Answering: A generative Large Language Model (LLM), specifically GPT-5 and the open-weight Qwen-2.5, answers the 119 questions based on the retrieved text.
   • Prompting Strategies: The system employs hand-crafted, question-agnostic prompt templates. The authors evaluated the impact of Chain-of-Thought (CoT) reasoning, LLM-generated prompts, and in-context examples (selected via random or semantic similarity strategies).

Key Contributions
The paper presents three main contributions:

1. Element-Level Framework: A formalized set of 119 questions that operationalize reporting guidelines at the element level, moving beyond binary item presence to assess specific reporting details.
2. SPIRIT-CONSORT-ELM Dataset: An expert-annotated dataset containing judgments for 119 questions across 200 RCT publications, providing a benchmark for fine-grained assessment.
3. Automated Assessment Pipeline: An evaluation of LLMs for this task, demonstrating a pipeline that combines sentence-level evidence retrieval with generative LLM reasoning to answer structured questions.

Results

• Human Annotation: Inter-annotator agreement was high, with a mean Gwet's AC1 of 0.782, indicating that humans can reliably perform element-level assessment when provided with relevant text snippets.
• Model Performance: The automated pipeline using GPT-5 achieved an F1 score of 0.822, accuracy of 0.852, and Gwet's AC1 of 0.796. The open-weight Qwen-2.5 model achieved comparable results (F1: 0.803), though with higher inference time. Both models outperformed a majority-class baseline.
• Evidence Retrieval Impact: Using expert-annotated text snippets (an upper bound) improved performance significantly (F1: 0.893) compared to the automated retrieval method (F1: 0.822), suggesting that improving evidence retrieval is a key area for future enhancement.
• Prompting and Examples: Chain-of-Thought reasoning provided a modest performance boost (+1.0 pp F1) but increased processing time and cost by more than fivefold. In-context examples (one-shot) yielded slight improvements (+0.8 pp F1) depending on the selection strategy, but adding more examples (3-shot, 5-shot) degraded performance.
• Guideline Differences: Performance was higher for SPIRIT-related questions (F1: 0.840) than CONSORT-related questions (F1: 0.780), likely because CONSORT questions often require interpreting longer, more diffuse text spans regarding results.

Significance and Claims
The authors claim that SPIRIT-CONSORT-ELM provides the first benchmark for evaluating reporting guideline completeness at the element level. They posit that this granularity enables a more valid and reliable evaluation of RCT transparency than simple item-level checks. The framework allows for the identification of specific missing elements, offering actionable feedback for authors, reviewers, and editors. While the automated system demonstrates feasibility, the authors remain modest, noting that current performance levels, while encouraging, may require further improvement for widespread real-world adoption. They emphasize that the framework complements existing guidelines and supports meta-research by providing higher-resolution data on reporting practices. The dataset and code are publicly available to facilitate further development of automated systems for improving RCT reporting.