🤖 AI

Principles and Guidelines for Randomized Controlled Trials in AI Evaluation

This paper establishes a foundational framework for standardizing Randomized Controlled Trials in AI evaluation by adapting the Shadish et al. four-validity model and TOP Guidelines into five principles and 33 operational guidelines that prioritize human performance, causal inference, and transparency to address unique challenges in human-AI interaction studies.

Original authors: Christopher Kelly, Angelica Chowdhury, Alexandra Campili, Bimpe Ayoola, Devin Barbour, Thomas Chen Dawson, Ze Shen Chin, Rokas Gipiškis

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Christopher Kelly, Angelica Chowdhury, Alexandra Campili, Bimpe Ayoola, Devin Barbour, Thomas Chen Dawson, Ze Shen Chin, Rokas Gipiškis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out if a new, super-smart calculator actually helps people solve math problems better, or if it just makes them feel like they are doing a better job.

For a long time, scientists testing AI have been like chefs tasting their own soup without a recipe. They might say, "This soup tastes great!" but they haven't told you what ingredients they used, how much salt was added, or if they tasted it while wearing a blindfold. Sometimes, the "soup" (the AI) changes flavor halfway through the tasting, or the people tasting it are all professional chefs rather than regular diners.

This paper is a new, strict recipe book for testing AI. It says: "Stop guessing. Let's run a proper, scientific experiment to see if AI actually helps humans."

Here is the simple breakdown of their "recipe," using some everyday analogies:

1. The Core Idea: The "Human Uplift" Test

Most AI tests today are like a car crash test where they just crash the car into a wall and see how much the car breaks. This paper says we need to test the driver, not just the car.

The Old Way: "Look, the AI wrote this code!"
The New Way: "We gave half the people the AI and half didn't. Did the people with the AI actually write better code, learn faster, or make fewer mistakes than the people without it?"

2. The Five Rules of the Game (The Principles)

The authors borrowed rules from medicine and psychology and added a new one for AI. Think of these as the five pillars holding up a bridge. If one is weak, the bridge (the study) collapses.

Rule 1: Are We Measuring the Right Thing? (Construct Validity)
- Analogy: Imagine you want to test if a new running shoe makes you faster. If you measure "how loud the shoe squeaks," you aren't measuring speed.
- The Paper's Point: Don't just count how many words an AI helped someone write (that's easy to fake). Measure if the quality of the thinking actually improved.
Rule 2: Did the AI Actually Cause the Change? (Internal Validity)
- Analogy: If you give a group of people a new vitamin and they get healthier, was it the vitamin? Or was it that they also started eating salad and sleeping more?
- The Paper's Point: You must make sure the only difference between the two groups is the AI. If the "AI group" also got better instructions or a nicer computer, you can't blame the AI for the success. Also, you have to stop the "no-AI" group from secretly sneaking a peek at the AI.
Rule 3: Does This Work for Everyone? (External Validity)
- Analogy: If a medicine works on 20-year-old men in a lab, does it work on 70-year-old women in a hospital?
- The Paper's Point: Don't test AI only on computer experts or only on university students. If you do, you can't claim it works for everyone. You need to say clearly: "This works for these people, in this situation."
Rule 4: Are the Numbers Real? (Statistical Conclusion Validity)
- Analogy: If you flip a coin 3 times and get heads every time, you might think the coin is magic. But if you flip it 1,000 times, it's probably just luck.
- The Paper's Point: Don't get excited about tiny improvements that could just be luck. The authors say we need to be super strict with our math (using a stricter "p-value" of 0.005 instead of the usual 0.05) because AI claims are high-stakes. We need to know how much better the AI made things, not just if it made things "better."
Rule 5: Show Your Work! (Transparency, Repeatability, and Verification)
- Analogy: If a magician says, "I made a rabbit appear," but won't let you see the hat or the rabbit, you don't believe him.
- The Paper's Point: AI changes fast. If you don't write down exactly which version of the AI you used, what prompts you typed, and share your data, nobody can check your work later. The paper creates a 4-level "trust ladder":
  1. Disclosure: "We told you what we did."
  2. Sharing: "We gave you the data and code."
  3. Verification: "An independent person checked that the code actually runs."
  4. Repeatability: "Another team tried it with new people and got the same result."

3. The 33-Step Checklist

The paper doesn't just give rules; it gives a 33-step checklist for researchers.

Example: "Before you start, write down exactly what you are testing so you can't change the rules halfway through."
Example: "Make sure the people in the 'no AI' group aren't secretly using AI on their own phones."
Example: "Check if the AI makes the work faster but worse, or slower but better."

4. Why Do We Need This?

Right now, the world is full of AI studies that are confusing. Some say AI is amazing; others say it's useless. The authors say this is because everyone is playing by different rules.

The Problem: If we can't trust the studies, we might deploy dangerous AI, or we might ignore helpful AI.
The Solution: This paper is a "standard operating procedure." It's a blueprint for building a bridge of trust. It helps researchers plan their studies, helps reviewers check if a study is good, and helps governments make laws based on real facts, not guesses.

Summary

Think of this paper as the FDA (Food and Drug Administration) for AI experiments. Just as you wouldn't eat a new drug without a rigorous, double-blind clinical trial, we shouldn't trust claims about AI helping humans without these strict, standardized tests. It's about moving from "We think AI is cool" to "We have proof that AI helps these people do this task better."

Technical Summary: Principles and Guidelines for Randomized Controlled Trials in AI Evaluation

Problem Statement

The credibility of empirical research across disciplines has faced sustained scrutiny due to replication crises, heterogeneous trial designs, and inadequate transparency. In the specific domain of advanced AI system evaluation, these issues are compounded by a near-total absence of established best practices. Current evaluation ecosystems rely heavily on benchmark evaluations, leaderboards, and audits, which fail to provide robust assurances regarding AI performance across different domains or user groups.

A critical gap exists in the methodology for assessing "human uplift"—the causal effect of AI access on human performance. Terminology is inconsistent; "uplift" is often confused with "treatment effect," "incremental impact," or simple "performance evaluation," leading to conflicting definitions among leading AI organizations (e.g., RAND, UK AI Safety Institute, Anthropic, OpenAI). Furthermore, existing reporting standards (such as CONSORT-AI, SPIRIT-AI, and STARE-HI) were designed for static clinical or software interventions and fail to address AI-specific challenges, including:

Rapid model evolution: Static intervention assumptions do not hold for systems undergoing frequent updates, prompt changes, or fine-tuning.
Contamination and spillover: In multi-user environments, control groups may access models unsanctioned by the protocol, violating Randomized Controlled Trial (RCT) assumptions.
Human-AI interaction dynamics: Existing frameworks lack guidance on documenting user instructions, autonomy levels, and output modification.
Construct validity: Standard metrics (e.g., speed, token counts) may not reflect genuine capability improvements like reasoning quality or trust.
Reproducibility barriers: Closed-source models and API-based systems render traditional transparency (sharing weights/data) insufficient; reproducibility now depends on logs, prompts, and metadata.

Without clear standards, policymakers and governance bodies cannot aggregate findings, and technical decisions regarding deployment thresholds and safety rely on potentially underpowered or poorly designed studies.

Methodology

This work establishes a foundational framework by adapting the Shadish-Cook-Campbell four-validity framework (Shadish et al., 2002) and extending it with a fifth principle derived from the Transparency and Openness Promotion (TOP) Guidelines (Center for Open Science, 2025). The authors operationalize these five principles into 33 concrete guidelines tailored specifically for AI evaluation RCTs.

The framework defines an AI evaluation RCT as a randomized controlled trial measuring the causal effect of AI access or use on a given human outcome (e.g., behavior, performance, wellbeing), distinct from benchmark evaluations that assess model outputs against fixed test sets.

The methodology involves:

Adapting Validity Principles: Extending Construct, Internal, External, and Statistical Conclusion Validity to address AI-specific threats (e.g., model version drift, interference).
Integrating Transparency: Developing a four-level hierarchy for Transparency, Repeatability, and Verification (TRV) that scales the strength of causal claims based on verifiability.
Operationalization: Translating abstract principles into actionable requirements covering study design, analysis, reporting, and pre-registration.
Evidence Base: Drawing on established standards from software engineering, economics, clinical sciences, and psychology, while addressing gaps identified in recent practitioner interviews (Paskov et al., 2026).

Key Contributions

1. A Five-Principle Framework for AI RCTs

The paper formalizes five core principles that serve as interpretive tools for determining the validity of claims:

Construct Validity: Ensures outcome measures capture underlying constructs (e.g., coding competence) rather than narrow proxies (e.g., speed) that AI can optimize without genuine skill enhancement.
Internal Validity: Ensures observed differences are attributable to the AI intervention, addressing threats like control group contamination, model updates during the study, and differential compliance.
External Validity: Defines the scope of generalization regarding population (e.g., AI literacy levels), tasks, and settings, explicitly warning against over-generalizing from WEIRD (Western, Educated, Industrialized, Rich, Democratic) samples or artificial tasks.
Statistical Conclusion Validity: Addresses evidentiary thresholds, power, and precision. The framework recommends $\alpha = 0.005$ for novel causal claims to reduce false positives, emphasizes effect size estimation with confidence intervals, and distinguishes between exploratory and confirmatory analyses.
Transparency, Repeatability, and Verification (TRV): A graded framework where causal claim strength scales with verifiability. It distinguishes between:
- Level 1 (Disclosure): Reporting what was done.
- Level 2 (Sharing): Depositing data/code/materials.
- Level 3 (Verification): Independent confirmation that materials are accessible and code runs.
- Level 4 (Repeatability): Independent reproduction, robustness checks, and replication with new data.

2. Thirty-Three Operationalized Guidelines

The authors provide a comprehensive checklist (Appendix A) covering the entire research lifecycle. Key guidelines include:

Design: Pre-registering protocols, defining the Overall Evaluation Criterion (OEC) before data collection, and using stratified randomization based on AI literacy and domain expertise.
Implementation: Monitoring compliance via objective tracking (API logs, screen recordings), ensuring UI equivalence between treatment and control, and controlling for participant effort.
Analysis: Using intention-to-treat (ITT) as the primary analysis, conducting sensitivity analyses for compliance, and reporting heterogeneity metrics (e.g., $\tau^2$ ).
Reporting: Using an evidentiary continuum for p-values (avoiding binary "significant/non-significant" language), reporting harms and unintended effects, and distinguishing between statistical, practical, and scientific significance.

3. Addressing AI-Specific Challenges

The framework explicitly introduces mechanisms to handle:

Model Versioning: "Pinning" model versions and monitoring for drift.
Spillover Effects: Using cluster randomization or monitoring to prevent unsanctioned model use from contaminating control groups.
Heterogeneity: Pre-specifying subgroups (e.g., by AI literacy) and calculating power for interaction effects, which require significantly larger sample sizes than main effects.
Equitable Impact: Assessing whether AI benefits or harms specific subpopulations, rather than relying solely on average treatment effects.

Results and Claims

The paper does not present new empirical data or experimental results. Instead, its "results" are the framework and guidelines themselves. The authors claim that this work:

Standardizes the Field: Provides a shared infrastructure for planning, reviewing, and setting standards for AI evaluation RCTs.
Formalizes Causal Inference: Adapts RCT methodology to the dynamic nature of AI, moving beyond simple benchmarking to causal inference on human performance.
Mitigates Validity Threats: Systematically catalogs threats to validity specific to AI (e.g., "Model version drift," "Construct confounding with AI assistance") and provides design-based solutions.
Elevates Evidentiary Standards: Proposes stricter statistical thresholds ( $\alpha = 0.005$ ) and a graded transparency framework to counter the "false positive" risks in a nascent field with high researcher degrees of freedom.

Significance and Intended Impact

The authors position this framework as serving three key roles:

Design Tool: A pre-study checklist for researchers to ensure methodological rigor in planning AI uplift studies.
Evaluation Rubric: A review standard for funders, journals, and governance bodies to assess the credibility of existing work and understand the inferential limits of reported findings.
Blueprint for Standard Setting: A roadmap for the field to converge on norms, potentially elevating specific guidelines into mandatory requirements (e.g., mandatory pre-registration) as the field matures.

The paper concludes that rigorous, transparent, and repeatable evaluation practices are essential safeguards. They ensure that AI development proceeds on credible evidence, that benefits are distributed equitably, and that claims about AI capabilities align with empirical reality. The authors acknowledge limitations, noting that the framework focuses exclusively on experimental designs (excluding observational studies) and imposes substantial resource demands. They call for future work to validate the framework through systematic community review and pilot implementation.