📊 statistics

The Judge Knows When It Knows: Calibrated Abstention for LLM-Based A/B-Test Prediction

This study demonstrates that while multimodal LLMs cannot reliably predict real A/B test outcomes from screenshots alone due to a shared bias toward unreliable labels, they can achieve meaningful accuracy by calibrating their predictions to identify and abstain from uncertain cases, a limitation that mirrors the inability of human experts to outperform chance despite high inter-rater consensus.

Original authors: Tyler Dooskin, Squoosh Technical Staff

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tyler Dooskin, Squoosh Technical Staff

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: "Which of these two suspects is guilty?" In the world of websites, the "suspects" are two slightly different versions of a page, and the "guilty" one is the version that gets more people to buy something or sign up. This is called an A/B test. Usually, to find the winner, you have to wait and watch real people interact with the site for days or weeks. But recently, a new kind of detective has arrived: Artificial Intelligence. The big question everyone is asking is, "Can an AI look at a picture of the two websites and instantly tell us which one will win, saving us all that waiting time?"

To understand this paper, you need to know a few things about how we measure "good" in science. First, there's the difference between being right by luck and being right by skill. If you guess "Heads" on a coin flip, you'll be right 50% of the time just by chance. If a detective claims to be a genius but only gets 51% right, they aren't really doing anything special. Scientists use a special score called "Cohen's kappa" (let's call it the "Skill Score") to strip away the luck and see if the detective actually has a brain. Second, there's the problem of "Shared Bias." Imagine a teacher and a student who both grew up in the same town. They might agree on everything, not because the student is smart, but because they both learned the same wrong facts from their parents. If an AI and the people who made the test data both learned the same "folk wisdom," they might agree on the answer, but that doesn't mean the AI can predict the future.

This paper is a very honest, very strict report from a team called Squoosh who tried to answer that big question. They didn't just ask the AI to guess; they set up a trap. They locked their rules before they started, like a scientist writing down a recipe before cooking, so they couldn't change the rules to make the AI look better later. They used a massive library of real-world website tests to see if the AI could actually predict the winners.

Here is the twist: The AI is mostly a failure at its main job. When the team asked the AI to look at 330 real tests and pick a winner every single time, it barely did better than random guessing. In fact, the AI seemed to agree more with the "unreliable" tests (where the result wasn't clear) than the "reliable" ones. It turns out the AI and the people who created the test data were both repeating the same old myths about what makes a website good, rather than actually predicting what would happen. Even using a much smarter, more expensive AI didn't help; it was just as bad. The paper explicitly rules out the idea that we can just "buy" a better AI or write a better prompt to solve this. The "magic crystal ball" that predicts website winners from screenshots alone? It doesn't exist.

However, the paper doesn't say the AI is useless. It says the AI is a honest detective. The team discovered a special trick: if they tell the AI, "Only guess if you are absolutely sure, otherwise say 'I don't know'," the AI becomes surprisingly good at the times it does speak up. When the AI's internal team of judges all agreed strongly on an answer, it got the right winner about 31% of the time on the reliable tests. That sounds low, but remember, real human experts (who have years of experience) also get it right only about 29% of the time on average. The AI isn't a super-genius; it's just a tool that knows when it's guessing and when it's not.

The most important finding is about trust. The paper shows that when a group of AIs all agree with each other, it doesn't mean they are right; it often just means they are all repeating the same shared bias. It's like a jury where everyone is related; they will all vote the same way, but that doesn't make the verdict correct. The paper proves that this "shared bias" happens with humans, too. When they asked 15 real human experts to guess the winners, the experts agreed with each other strongly, but they were wrong just as often as the AI.

So, what is the final takeaway? The paper concludes that we cannot build a tool that predicts website winners with 100% certainty. Instead, the best tool is a "Calibrated Screening Instrument." It's a system that says, "I'm confident this one will win," or "I'm not sure, don't ask me." It's honest about its limits. The paper also shows that you don't need a super-expensive, huge AI to do this; a smaller, cheaper version works just as well if you use the "honesty" trick. The real value isn't in getting the answer every time; it's in knowing exactly which times you shouldn't trust the answer. The paper ends by releasing all its data, its failed experiments, and its rules, proving that sometimes, the most important discovery is admitting that the magic trick doesn't work, but a very careful, very honest version of it might just be useful enough.

=== SUMMARY ===

Technical Summary: Calibrated Abstention for LLM-Based A/B-Test Prediction

Problem Statement

The paper investigates whether multimodal Large Language Models (LLMs) can predict the winner of real-world A/B tests based solely on screenshots of web page variants. The authors challenge the prevailing industry narrative that LLMs can serve as "pre-test oracles" for conversion rate optimization (CRO). They identify a critical failure mode in existing evaluations: curated A/B-test catalogs are heavily skewed toward variant wins (often 75–85% B-wins), allowing a simple model that prefers the changed page to achieve high raw accuracy while contributing zero decision value. The core problem is distinguishing between a model that genuinely predicts experimental outcomes and one that merely exploits base-rate artifacts or shares "folk wisdom" priors with the label curators.

Methodology

The authors conducted a six-week, pre-registered experimental program against a proprietary corpus of 546 real A/B tests from a leading CRO agency. The study employed a rigorous statistical harness designed to prevent overfitting and marketing bias:

Corpus & Ground Truth: The dataset included 330–332 decisive tests with renderable screenshots. Crucially, the authors recomputed statistical significance ( $p < 0.05$ ) from raw per-arm counts, revealing that 44% of the "ground-truth" labels in the catalog came from non-significant tests.
Evaluation Metric: The primary metric is Cohen's $\kappa$ (kappa), a chance-corrected agreement measure, rather than raw accuracy. Confidence intervals were generated via percentile bootstrapping ( $B=10,000$ ).
Statistical Rigor:
- Pre-registration: Hypotheses, decision gates, and power analyses were locked before data collection.
- Gate-on-Resample: For gated metrics (where the model abstains), the confidence gate was re-applied within every bootstrap replicate to propagate coverage uncertainty into the $\kappa$ confidence interval.
- Equivalence Testing: TOST (Two One-Sided Tests) was used to prove that more expensive models were statistically equivalent to cheaper ones, rather than just failing to show a difference.
- Adversarial Verification: Headline results underwent independent re-derivation and falsification passes before publication.
Experimental Design: The study utilized a counterbalanced 16–20 vote panel of Gemini 3 Flash judges. It tested various "levers" including model capacity (Gemini 3.1 Pro), prompt mechanisms, stimulus fidelity (real vs. reconstructed screenshots), and change-type priors.

Key Results

1. Unconditional Prediction Fails the "Honesty Bar"

On the full dataset, a Gemini 3 Flash judge achieved $\kappa = 0.141$ [0.034, 0.248], which is detectably above chance but weak. However, when restricted to statistically significant labels (the trustworthy half of the data), the evidence became inconclusive ( $\kappa = 0.108$ [−0.049, 0.264]).

Shared Prior Artifact: The judge agreed more with unreliable (non-significant) labels than reliable ones. This suggests the model and the label curators share a "folk wisdom" prior about what should win, rather than the model predicting what did win.
Model Capacity: A frontier model (Gemini 3.1 Pro) was statistically equivalent to Flash ( $\Delta\kappa = -0.037$ within a $\pm0.20$ TOST bound) despite being 2.8x more expensive.
Prompt & Fidelity: Redesigning prompts, increasing stimulus fidelity, and using change-type priors all failed to produce statistically significant improvements.

2. Calibrated Abstention Reveals Signal

The only operating point that survived the authors' own label-validity audit was confident-call-or-abstain.

Mechanism: By gating predictions on internal panel agreement (specifically a vote-margin $\ge 0.6$ ), the model abstains on ~51% of tests.
Performance: On the subset of tests where the model makes a confident call (49% coverage), the $\kappa$ on significant-only labels rises to 0.311 [0.025, 0.555]. This is the only cell in the study where the confidence interval excludes zero for trustworthy labels.
Replication vs. Validity: The authors demonstrated that while the model's calls are perfectly reproducible (31/31 agreement across independent runs), reproducibility does not equal validity. A held-out confirmation on a smaller cohort (33 calls) returned $\kappa = -0.100$ [−0.200, 0.000], an inconclusive result that did not refute the finding but failed to confirm it, highlighting the fragility of selective prediction gains.

3. The Human Ceiling and Consensus Illusion

The paper benchmarks the LLM against human experts:

Human Baseline: A study of 15 source-naïve CRO practitioners on the same instrument scored at chance against real outcomes ( $\kappa \approx -0.03$ ).
Shared Prior in Humans: Like the LLMs, humans agreed strongly with each other (inter-rater $\kappa = 0.53$ ) but failed to predict reality. This confirms the "shared prior" artifact is not an LLM-specific pathology but a fundamental limitation of the task: the visual appearance of a page underdetermines its experimental outcome for both humans and machines.
Effective Sample Size: A 16-vote LLM panel carries only $\approx 2$ effective independent votes ( $n_{eff} \approx 1.9–2.6$ ). Similarly, 15 human experts acted statistically like fewer than two independent judgments. Consensus among correlated judges (or humans) is persuasive but not evidence of skill.

4. Economics and Deployment

Cost Efficiency: A 9-vote panel preserves the gated operating point ( $\kappa \approx 0.27$ ) at roughly half the inference cost of the full panel.
Deliverables: The paper supports deploying a calibrated screening instrument rather than a win-rate oracle. This includes:
- Confident-call-or-abstain winner reads.
- Screenshot-grounded UX audits with adversarial verification (12/13 findings verified).
- A tiered claims ledger where every number carries an evidence tier (e.g., CONFIRMED, EXPLORATORY, REFUTED).

Significance and Claims

The paper's primary contribution is a calibration discipline for LLM-based evaluation, moving away from "maximal" claims toward "honest" ones.

Refuted Claims: The paper explicitly refutes the possibility of unconditional winner prediction, mobile-validated accuracy (due to lack of mobile-rendered labeled corpora), and predicted conversion-lift percentages.
Supported Claims: The authors support the deployment of a calibrated screening instrument that abstains on roughly half of inputs. The value proposition is not a high-accuracy oracle, but a tool that identifies a small minority of cases where a signal exists and abstains on the rest to avoid hallucinated confidence.
Scientific Integrity: The paper serves as a counter-narrative to vendor whitepapers by publishing negative results, locked gates, and "inconclusive" held-out tests. It argues that a measurement system that cannot kill its owner's claims is advertising, not measurement.

The authors conclude that on the task of predicting A/B test winners from screenshots, consensus—whether human or model—is reproducible and persuasive, but not evidence. The only viable path forward is a system that knows when it knows, and explicitly admits when it does not.