📊 statistics

Optimal Inference with Black-box Predictions

This paper establishes the information-theoretic limits of statistical inference in high-dimensional Gaussian sequence models by combining observed data with black-box predictions, and proposes practical hypothesis tests that adapt to unknown prediction accuracies while leveraging strong alignment to achieve both validity and efficiency.

Original authors: Lucas Kania, Abhinav Chakraborty, Edward Kennedy, Larry Wasserman, Sivaraman Balakrishnan

Published 2026-08-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Lucas Kania, Abhinav Chakraborty, Edward Kennedy, Larry Wasserman, Sivaraman Balakrishnan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Is a suspect innocent, or is there a hidden pattern in the evidence that proves they are guilty? In the world of statistics, this is called hypothesis testing. Usually, detectives rely solely on the raw evidence they can see—fingerprints, footprints, or witness statements. But in modern science, we often have a second source of information: a "black box." Think of this as a super-smart, mysterious AI that looks at the data and whispers a guess about what's really happening. The problem is, we don't know if this AI is a genius or a complete hallucination. If we trust a bad guess too much, we might convict an innocent person. If we ignore a brilliant guess, we might let a guilty person go free.

The big question for statisticians is: How do we combine the hard evidence (the data) with these mysterious whispers (the predictions) to get the best possible answer? We want a method that is valid (it won't make mistakes just because the AI is wrong) but also efficient (it uses the AI's help when it's right to find the truth faster). This paper dives deep into that exact puzzle, using a mathematical playground called the "Gaussian sequence model" to figure out the absolute limits of how well we can do this.

The Mystery of the Whispering Oracle

Imagine you are playing a game of "Hot and Cold" to find a hidden treasure. You have a map (your data), but it's a bit blurry. Then, a mysterious Oracle (your black-box prediction) steps in and points in a direction, saying, "The treasure is this way!"

If the Oracle is perfect, you just follow the arrow and find the treasure instantly. If the Oracle is lying or confused, following the arrow might lead you off a cliff. The tricky part is that you don't know if the Oracle is telling the truth until after you've made your move.

This paper, written by a team of statisticians, asks: What is the smartest way to play this game when you don't know how good the Oracle is?

They found that the answer depends entirely on how the Oracles are arranged.

Scenario 1: The Lone Oracle

If you only have one Oracle, the game is simple. You can either follow its lead or ignore it. The paper shows that a smart detective can switch between these two strategies almost as well as a "super-detective" who knows exactly how accurate the Oracle is. There isn't much of a penalty for not knowing the Oracle's quality here.

Scenario 2: The Squad of Oracles (The Orthogonal Case)

Now, imagine you have a whole team of Oracles, and they are all pointing in completely different, unrelated directions (like one pointing North, one East, one Up, etc.). This is what the authors call "orthogonal" predictions.

Here, the paper discovers a surprising twist. If you don't know which Oracles are telling the truth, you have to be very careful. You can't just pick one; you have to check all of them. The authors prove that the more Oracles you have, the harder the game becomes if you don't know their accuracy.

Think of it like searching a dark forest. If you have one flashlight, you just shine it where the Oracle says. But if you have ten flashlights pointing in ten different directions, and you don't know which one is working, you have to scan the whole forest. The paper shows that the "cost" of not knowing the truth grows with the square root of the number of Oracles. If you have 100 Oracles, you might need 10 times more data to be sure than if you knew exactly which one was the hero. It's a "tax" for being uncertain.

Scenario 3: The Clumped Squad (The Arbitrary Case)

But wait! What if the Oracles aren't pointing in random directions? What if they are all bunched up, pointing roughly the same way? Maybe they all came from the same AI model, just trained on slightly different data.

This is where the paper gets really clever. The authors realized that if the Oracles are "clumped" together, you don't need to scan the whole forest. You only need to look in the direction where the majority of them are pointing. They invented a new tool called the "Intrinsic Projection."

Imagine the Oracles are a flock of birds. Even if there are 50 birds, if they are all flying in a tight formation, they act like a single giant bird. The "Intrinsic Projection" is a way to measure how "tight" that formation is.

If the birds are flying in a tight line, the "intrinsic dimension" is low (close to 1).
If they are scattered everywhere, the "intrinsic dimension" is high (close to the number of birds).

The paper proves that if you use this new "Intrinsic Projection" test, you can ignore the total number of Oracles and focus only on how tightly they are grouped. If they are clumped, you can find the treasure much faster, even without knowing their accuracy. It's like realizing that even though you have 50 flashlights, they are all shining on the same tree, so you only need to look at that one tree.

The Big Takeaway

The authors didn't just guess these things; they used rigorous math to prove the absolute limits of what is possible. They showed that:

You can't have a free lunch: If you have many independent (orthogonal) predictions and don't know their quality, you must pay a price in terms of how much data you need. The more predictions, the more data you need.
But there is a loophole: If your predictions are correlated (they agree with each other), you can bypass that penalty. By using their "Intrinsic Projection," you can adapt to the unknown quality of the predictions and still perform nearly as well as if you knew the truth.

In short, the paper gives us a roadmap for how to trust our AI helpers. It tells us that if our helpers are all saying the same thing, we can trust them more and need less data to be sure. But if they are all shouting different things, we have to be very skeptical and gather a lot more evidence before we make a decision. It's a guide to being smart, cautious, and efficient in a world full of noisy predictions.

Technical Summary: Optimal Inference with Black-box Predictions

Problem Statement
This work addresses the challenge of performing valid and efficient statistical hypothesis testing in high-dimensional settings when auxiliary black-box predictions of the data distribution are available. Specifically, the authors study the Gaussian sequence model where one observes $n$ samples $X_1, \dots, X_n \sim N(\theta, I_d/n)$ and $m$ fixed black-box predictions $\{\hat{\theta}_1, \dots, \hat{\theta}_m\}$ of the unknown mean $\theta \in \mathbb{R}^d$ . The goal is to test the null hypothesis $H_0: \theta = 0$ against an alternative $H_1: \|\theta\|_2 \geq \epsilon$ .

A central tension exists: while informative predictions can concentrate statistical power on specific directions, poor predictions can mislead a test. Crucially, in practice, the accuracy (error radius) of these predictions is unknown. The paper seeks to characterize the information-theoretic limits of inference in this setting and develop tests that adapt to unknown prediction accuracies without sacrificing validity.

Methodology and Framework
The authors employ a minimax framework to analyze the hardness of distinguishing $H_0$ from $H_1$ . They define the "oracle" setting where prediction accuracies $r = (r_1, \dots, r_m)$ are known, and the "adaptive" setting where they are unknown.

Oracle Inference: When accuracies are known, the alternative hypothesis is localized to the intersection of balls $B_r = \bigcap_{i=1}^m B(\hat{\theta}_i, r_i)$ . The authors characterize the optimal separation rate $\epsilon^*(r)$ based on the geometry of the intersection $S_\epsilon \cap B_r$ (where $S_\epsilon$ is the sphere of radius $\epsilon$ ).
- They introduce $\gamma_*(\epsilon, r)$ , the length of the smallest projection of the alternative mean onto the span of the predictions.
- They define a "worst-case cosine similarity" $CS(\epsilon, r)$ to measure the alignment between the data mean and the prediction span.
- The optimal oracle test switches between a standard chi-squared test (when predictions are uninformative) and a likelihood-ratio or projection-based test (when predictions are informative).
Adaptive Inference (Orthogonal Predictions): When predictions are orthogonal and accuracies are unknown, the authors propose an adaptive test that aggregates projections onto all prediction directions using a chi-squared statistic on the span of the predictions. They analyze the "adaptivity gap"—the cost in sample size required because the test cannot weight directions by their unknown accuracies.
Adaptive Inference (Arbitrary Predictions): For non-orthogonal predictions (e.g., from the same model via resampling), standard projection tests are suboptimal because they treat all directions in the span equally. The authors propose an intrinsic projection test. This method weights projections based on the singular values of the prediction matrix $\hat{U}$ , specifically utilizing the squared cosine similarity matrix $\Sigma = \hat{U}^T \hat{U}$ . The test statistic depends on the intrinsic dimension of $\Sigma^2$ , defined as $Int(\Sigma^2) = \text{Tr}(\Sigma^2) / \|\Sigma^2\|_{op}$ , rather than the ambient dimension of the prediction span.

Key Contributions and Results

Oracle Limits (Theorems 1 & 2): The authors characterize the optimal sample complexity for an oracle test. The rate depends on the worst-case cosine similarity between the true mean and the prediction span.
- If the similarity is low ( $CS \lesssim d^{-1/4}$ ), the prediction offers no benefit, and the rate remains the prediction-free rate of $n \gtrsim \sqrt{d}/\epsilon^2$ .
- If the similarity is high, the oracle achieves dimension-independent rates $n \gtrsim 1/\gamma_*^2$ , effectively reducing the problem to a one-dimensional test.
- If the prediction is extremely accurate (disjoint spheres), parametric rates $n \gtrsim 1/\|\theta^*\|^2$ are achieved.
Cost of Adaptation for Orthogonal Predictions (Theorems 3 & 4): For orthogonal predictions, the authors establish that an adaptive test requires a sample size larger by a factor of $\sqrt{m}$ compared to the oracle.
- The adaptive rate is $n \gtrsim \sqrt{m}/\gamma_*^2$ .
- This gap arises because the adaptive test must search for deviations across all $m$ orthogonal directions, whereas the oracle can aggregate them into a single optimal direction.
- The authors prove this $\sqrt{m}$ dependence is optimal among separation curves satisfying a natural invariance property (Property 1), showing that "free lunch" is impossible for orthogonal predictions.
Adaptation for Arbitrary Predictions (Theorem 5): For non-orthogonal predictions, the authors show that the cost of adaptation depends on the intrinsic dimension ( $Int(\Sigma^2)$ ) rather than the number of predictions $m$ or the rank of the span.
- The adaptive test using intrinsic projection achieves rates $n \gtrsim \sqrt{Int(\Sigma^2)}/\tilde{\gamma}_*^2$ .
- Since $Int(\Sigma^2) \leq \text{rank}(\hat{U})$ , non-orthogonal predictions (which often cluster) can significantly reduce the adaptivity cost compared to the orthogonal case.
- The paper demonstrates that when predictions share substantial information (low intrinsic dimension), adaptive methods can outperform competing methods that ignore this structure.

Significance and Claims
The paper claims to provide a unifying principle for integrating black-box predictions into hypothesis testing. Its primary contributions are:

Characterizing Limits: It establishes the information-theoretic limits of inference with black-box predictions, distinguishing between regimes where predictions are helpful versus unhelpful based on geometric alignment.
Quantifying Adaptivity Costs: It rigorously quantifies the "cost of adaptation" (the sample size penalty for not knowing prediction accuracy). It reveals that for orthogonal predictions, this cost scales with $\sqrt{m}$ , but for correlated predictions, it scales with the intrinsic dimension of the prediction set.
Practical Algorithms: It proposes practical, adaptive tests (projection-based and intrinsic projection-based) that automatically adjust power allocation. These tests are valid regardless of prediction quality (maintaining Type I error control) but achieve optimal power when predictions are informative.

The authors position this work within the broader context of structure-agnostic inference and prediction-powered inference, extending these concepts to hypothesis testing. They emphasize that while predictions can improve efficiency, the optimal strategy depends critically on the geometric relationship between the predictions and the unknown truth, and that adaptivity to unknown quality incurs a specific, quantifiable cost.