Technical Summary: RegCheck – A Tool for Structured Comparisons Between Study Registrations and Papers

Problem Statement

Despite the recognized benefits of study registration for scientific transparency and rigour, the practice of comparing study registrations against their final published papers remains rare. This gap exists because manual comparison is a labour-intensive, time-consuming process requiring careful cross-format reading and domain expertise. Consequently, deviations from registered plans (e.g., outcome switching, changes in sample size, or analysis deviations) frequently go undetected by authors, reviewers, and editors.

While Generative Large Language Models (LLMs) offer a potential solution for automation, the ad hoc use of commercial chat interfaces is insufficient for formal research evaluation. The authors identify six critical limitations of direct LLM prompting:

1. Analytic Flexibility: Output varies significantly based on prompt phrasing, formatting, and model parameters, hindering standardization and reproducibility.
2. Interface Limitations: Commercial chat interfaces gate off critical hyperparameters (e.g., temperature, system prompts) and lack stability, as providers may change underlying models without notice.
3. Data Privacy: Uploading unpublished manuscripts to commercial interfaces risks data retention and training usage, violating confidentiality expectations of peer review.
4. Unwanted Dependencies: Sequential generation in chat contexts creates dependencies between dimensions (e.g., an error in judging hypotheses may propagate to statistical analysis), preventing independent evaluation of specific dimensions.
5. Task Conflation: Delegating ingestion, definition, extraction, and adjudication to a single LLM call obscures the performance of individual steps.
6. Verifiability: LLMs may "hallucinate" or provide claims without systematic citations, making it difficult for users to verify the source of a judgement without manual re-reading.

Methodology: The IDEA Framework and Parity Architecture

To address these limitations, the authors introduce RegCheck, an open-source, modular tool built upon a broader architecture called Parity. Parity implements the IDEA framework, a four-step conceptual workflow designed to keep inputs, evidence retrieval, and judgement steps distinct:

1. Ingestion: Documents (registrations and papers) are collected and parsed using deterministic methods (e.g., JSON APIs for registries, specialized PDF parsers like GROBID or DPT-2 for manuscripts). This ensures text integrity and handles multi-study papers by isolating relevant sections.
2. Definition: Users explicitly define the dimensions of comparison (e.g., sample size, primary outcomes). This step is human-driven, allowing for discipline-specific presets (e.g., clinical/medical, psychology) or custom definitions.
3. Extraction: Relevant text is retrieved using semantic embeddings. Documents are chunked (aligned with sentence/section boundaries), embedded, and queried against the dimension definitions using cosine similarity. A hybrid keyword search supplements this to ensure high-value chunks are not missed.
4. Adjudication: A generative LLM is invoked only at this stage. It receives the extracted evidence excerpts and the dimension definition to produce a structured judgement (Deviation, No Deviation, or Insufficient Evidence) with a rationale. Crucially, each dimension is adjudicated in a separate call to prevent cross-contamination of errors.

RegCheck Workflow Implementation:
The tool operationalizes IDEA into a six-stage pipeline:

• Ingestion & Preparation: Parsing and normalizing text, stripping references, and handling cross-references within papers.
• Embedding & Indexing: Creating vector representations of text chunks using provider-neutral models (supporting local deployment).
• Definition: Selecting or creating dimension sets.
• Extraction & Adjudication: Retrieving evidence, summarizing it, and generating verdicts.
• Reporting: Generating interactive, shareable reports (via unique IDs) featuring an "Overview" view (colour-coded verdicts with rationales) and an "Evidence" view (side-by-side source text with highlighted excerpts).

The system supports both a web interface and a Command Line Interface (CLI). The CLI allows for fully local execution (using local embedding and LLM models) to maximize privacy and security, and includes a "consensus mode" that runs multiple adjudication calls to mitigate stochasticity.

Key Contributions

1. RegCheck Software: A functional, open-source tool (available at regcheck.app and GitHub) that automates structured registration-paper comparisons while maintaining human oversight.
2. The IDEA Framework: A conceptual model for decomposing meta-scientific comparison tasks into distinct, modular steps (Ingestion, Definition, Extraction, Adjudication), clarifying where LLMs are and are not appropriate.
3. Verifiable Workflow: A design that ensures all LLM judgements are directly supported by deterministic, verbatim text excerpts, allowing users to trivially verify claims.
4. Privacy-Preserving Architecture: Support for local deployment and open-weight models, addressing the confidentiality concerns inherent in peer review.
5. Discipline-Specific Presets: Pre-configured dimension sets for various fields (clinical, psychology, economics) that align with domain-specific reporting norms.

Results and Evaluation Status

The paper does not present final, large-scale empirical results quantifying RegCheck's accuracy against a ground truth, as systematic evaluation is ongoing. Instead, the authors provide:

• A Worked Example: A hypothetical clinical trial comparison demonstrating the tool's ability to identify clear deviations (e.g., outcome switching from continuous to dichotomous), subtle deviations (e.g., renal function threshold changes from 30 to 45 mL/min), and cases of insufficient evidence.
• Anecdotal Testing: Preliminary testing by the research team suggests the tool is accurate in extraction and adept at catching deviations missed by humans, including subtle numeric changes and undisclosed outcome switches.
• Evaluation Strategy: The authors outline a rigorous evaluation plan focusing on:
  • Interrater Agreement: Comparing RegCheck's performance against human raters to establish "non-inferiority" (i.e., RegCheck should agree with humans as much as humans agree with each other).
  • Artificial Benchmarks: Using LLM-generated texts with injected inconsistencies to test accuracy against known ground truths.

Significance and Claims

The authors position RegCheck not as a replacement for human expertise, but as a tool to lower the practical barrier to registration-paper comparison. Its significance lies in:

• Routine Integration: Making consistency checks a feasible part of author workflows, peer review, and post-publication evaluation, thereby maximizing the utility of study registrations.
• Standardization: Providing a reproducible, transparent, and verifiable method for comparing research artefacts, addressing the inconsistencies of ad hoc LLM use.
• Trustworthiness: Enhancing the ability of the scientific community to distinguish between pre-registered plans and post-hoc decisions, which is critical for assessing the risk of bias and the robustness of scientific claims.

The paper concludes that by decomposing the comparison task and keeping human judgement "in the loop" for definition and final verification, RegCheck aims to transform a rarely performed, laborious task into a routine, efficient component of scientific practice.