🤖 AI

Consensus Sampling for Safer Generative AI

This paper introduces "Consensus Sampling," an architecture-agnostic black-box algorithm that aggregates multiple generative AI distributions to achieve safety guarantees competitive with the safest subset of models while abstaining from outputting results when there is insufficient agreement, thereby mitigating undetectable risks and adversarial influence.

Original authors: Adam Tauman Kalai, Yael Tauman Kalai, Or Zamir

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Adam Tauman Kalai, Yael Tauman Kalai, Or Zamir

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Undetectable" Danger

Imagine you have a group of artists (AI models) drawing pictures or writing stories for you. Most of them are honest and safe. But one of them is a trickster who secretly hides dangerous messages inside their work.

The scary part? You can't see the trick.

If the trickster draws a picture of a horse, it looks exactly like a normal horse.
If they write code, it looks like normal code, but it has a hidden "backdoor" that hackers can use later.
Even if you hire a super-smart inspector (a human or a super-intelligent AI) to look at the final picture or code, they cannot tell the difference between the safe version and the dangerous one. The danger is hidden in the probability of how the image was made, not in the image itself.

The paper asks: If we can't inspect the final product, how do we make sure we don't get the dangerous one?

The Solution: "Consensus Sampling" (The Jury of Artists)

The authors propose a method called Consensus Sampling. Instead of asking just one artist to do the job, you ask a group of $k$ artists.

Think of it like a jury trying to decide on a verdict.

The Setup: You have $k$ artists. You know that at least $s$ of them are honest (safe), but you don't know which ones they are. The rest might be tricksters.
The Process:
- The system asks all $k$ artists to generate an answer (a picture, a sentence, a piece of code).
- It doesn't just pick one at random. Instead, it looks at the confidence of each artist.
- It asks: "How likely was each artist to produce this specific answer?"
The Rule: If an answer is something that many of the artists would naturally produce with high confidence, it's likely safe. If an answer is something that only the "trickster" artist would produce (and the honest ones would never think of), the system rejects it.

The "Jinx" Analogy

The paper uses a concept called a "jinx" to explain why this works.
Imagine you and a friend flip a coin.

If you both flip "Heads," that's normal.
If you both flip "Heads" and you both flip "Heads" again, and again, that's a coincidence.
If you and a friend independently write the exact same long, complex story, it's highly unlikely to be a coincidence unless you are both telling the truth.

In this system, if an output is something that multiple independent models agree on (they all assign it a high probability of happening), it's a "safe jinx." If an output is weird and only one model thinks it's a good idea, the system assumes that model might be the trickster and says, "I'm not sure about this," and refuses to give an answer.

The Trade-off: Safety vs. Silence

There is a catch. Because the system is so careful, it might say "I don't know" (abstain) more often than a normal AI.

Normal AI: Always gives an answer, even if it's risky.
Consensus AI: Sometimes stays silent because the artists couldn't agree.

The paper introduces a "Risk Tolerance" knob (called R).

Turn the knob down (Low R): The system is extremely strict. It only outputs answers that almost all safe models agree on. It is very safe, but it might stay silent a lot.
Turn the knob up (High R): The system is more lenient. It accepts answers that fewer models agree on. It gives more answers, but the risk of getting a dangerous one goes up slightly.

The authors prove mathematically that this is the best possible deal. You can't get a safer system without it staying silent more often, and you can't get it to speak more often without taking more risks.

What This Actually Does (and Doesn't Do)

What it does:

It acts as a filter. If you have a group of models, and some are safe, this method ensures the final output is as safe as the safest models in the group.
It prevents "steganography" (hiding secrets in images/text) because the trickster model can't force the group to agree on a hidden message without the honest models also "accidentally" generating that message.
It works even if you don't know which models are the bad ones.

What it doesn't do:

It does not fix the models. It doesn't teach a bad model to be good.
It does not work if all the models are bad. If every artist in the room is a trickster, the system will still produce a trick.
It does not solve every safety problem. It specifically targets risks that are impossible to detect by looking at the final output (like hidden codes or viruses). It doesn't necessarily stop an AI from being rude or biased in a way that is obvious to look at.

Summary

Imagine you are buying a car. You can't see inside the engine to check for a hidden bomb (because the bomb is invisible).

Old way: You buy one car and hope for the best.
New way (Consensus Sampling): You ask 10 different car manufacturers to build the same car. You only buy the car if 5 of them say, "Yes, we would definitely build this exact car." If one manufacturer says, "I built this weird car with a hidden bomb," but the other 5 say, "We would never build that," you don't buy it. You might end up buying fewer cars (abstaining), but the ones you do buy are guaranteed to be safe, provided at least half the manufacturers are honest.

This paper provides the mathematical proof that this "group voting" method works, even when the bad actors are trying to trick you.

Technical Summary: Consensus Sampling for Safer Generative AI

Problem Statement
The paper addresses the challenge of aggregating multiple generative models to mitigate safety risks that are provably undetectable through output inspection alone. Citing works by Cohen [1987], de Witt et al. [2023], Zamir [2024], Motwani et al. [2024], and Perry et al. [2025], the authors note that risks such as steganography (hiding secrets in images or text) and embedded code vulnerabilities cannot be reliably identified or removed by examining the final output, even with superintelligent oversight. Traditional safety methods relying on inspecting model activations or outputs are insufficient against these "misalignment" risks.

The core mathematical problem is defined as a robust aggregation task: given $k$ probability distributions (induced by $k$ models), how can one construct a new distribution that is resilient to a minority of unsafe or adversarial constituents? The goal is to produce a sample that is safe without needing to know which specific models are safe, provided that a sufficient fraction ( $s$ ) of the models are safe and exhibit sufficient overlap in their output probabilities.

Methodology: Consensus Sampling
The authors propose Consensus Sampling, a black-box algorithm that operates under the assumption of oracle access: the ability to sample from each distribution $p_i$ and evaluate the probability $p_i(y)$ for any candidate output $y$ .

Core Intuition: The method is inspired by the robustness of the median in statistics. Just as the median is less sensitive to outliers than the mean, the algorithm seeks outputs that are assigned high probabilities by multiple models. If a single model is adversarial, it is unlikely to agree with $s$ independent safe models on a specific high-probability output unless that output is genuinely safe.
The Algorithm (Algorithm 1):
- Inputs: $k$ distributions, a target number of safe distributions $s$ , and a risk tolerance parameter $R$ .
- Process: The algorithm runs for $R$ rounds. In each round, it samples a candidate $y$ from the uniform mixture of all $k$ distributions. It then calculates the acceptance probability based on the ratio of the average of the $s$ smallest pointwise probabilities assigned to $y$ (denoted $p^{(1)}(y) \dots p^{(s)}(y)$ ) to the average of all $k$ probabilities.
- Output: If accepted, $y$ is returned. If no sample is accepted after $R$ rounds, the algorithm abstains (returns $\perp$ ).
Underlying Distribution: The algorithm effectively performs rejection sampling to approximate a distribution $q^*$ proportional to the sum of the $s$ smallest probabilities: $q^*(y) \propto \sum_{i=1}^s p^{(i)}(y)$ .

Key Contributions and Theoretical Guarantees

R-Robustness: The paper formalizes a guarantee called $R$ -robustness. A distribution is $R$ $R$ -robust if its risk (probability of generating an unsafe output in a set $U$ $U$ ) is at most $R$ $R$ times the average risk of the $s$ $s$ safest input distributions.
- Theorem 5.2: The output of the Consensus Sampling algorithm is $R$ -robust relative to the input distributions. This holds for any fixed prompt and any unsafe set $U$ , without requiring a formal definition of "safety" other than the existence of a set of low-risk outputs.
Adversarial Robustness: If a set $S$ of size $s$ contains safe models with mean risk $\epsilon$ , the algorithm's output risk is bounded by $\epsilon R$ . This implies that an adversary controlling $k-s$ models cannot force the system to output unsafe data unless they can compromise the majority of models or force them to agree on unsafe outputs.
Information Leakage Bounds: The paper demonstrates that $R$ $R$ -robustness limits information leakage, specifically in the context of steganography.
- Leakage Simulation: The algorithm's output can be simulated using only the safe distributions and a single integer $t \in [R+1]$ . This implies the algorithm leaks at most $\log(R+1)$ bits of information about the secret message encoded by an adversarial model.
- Steganography: If safe models have a low probability $\epsilon$ of decoding a secret message, the consensus sampler's probability of decoding it is bounded by $\epsilon R$ .
Pareto Optimality: The authors show that the tradeoff between worst-case risk and the abstention rate is Pareto-optimal. The algorithm cannot improve safety (lower risk) without increasing the rate of abstention, and vice versa.
Abstention Bounds: The probability of abstention depends on the overlap ( $\Delta$ ) among the safe distributions. If a majority of distributions are safe ( $s > k/2$ ) and they overlap significantly, the abstention rate decays exponentially with $R$ . However, if safe models have low overlap (a common issue in language models), the abstention rate may be high.

Experimental Results
The paper validates the mechanism through two experiments:

Synthetic Hypercube: Using distributions over $\{0, 1\}^{64}$ , the authors simulated a scenario with one adversarial source encoding a secret. The results confirmed the theoretical tradeoff: as the hidden message length increased (making the unsafe event rarer under safe distributions), the algorithm suppressed the unsafe output more effectively, while increasing $R$ reduced the abstention rate.
Image Steganography: Using the Janus-Pro-7B model, the authors tested the algorithm on image generation with a steganographic scheme (Zamir [2024]) that encoded credit card numbers into images.
- A standard uniform mixture of 3 sources (2 safe, 1 steganographic) produced unsafe images 33% of the time.
- Consensus sampling with $R=1$ to $R=3$ reduced the unsafe rate to 0% (within confidence intervals) while maintaining non-zero abstention rates. At $R=4$ , the abstention rate dropped to 0% while maintaining 0% unsafe outputs in the trial.

Significance and Claims
The paper claims to provide a model-agnostic approach to generative AI safety that does not rely on interpretability, specific model architectures, or human/AI inspection of outputs.

Shift in Paradigm: It moves safety from "detecting and removing bad outputs" (which is provably impossible for certain risks) to "aggregating distributions to limit adversarial influence."
Robustness via Overlap: It identifies overlap among safe distributions as the critical quantity governing the safety-abstention tradeoff.
Limitations: The authors are modest about the scope. The method does not guarantee safe outputs if all input models are unsafe. It requires multiple safe models and sufficient overlap between them. It does not provide a method for training safe models but rather a mechanism to aggregate existing ones. It addresses undetectable risks like steganography but does not cover broader societal harms or harms arising from multi-step interactions.

In conclusion, the work offers a theoretical framework and an efficient algorithm to bound the risk of generative AI outputs by leveraging the consensus of multiple models, providing formal guarantees against adversarial manipulation that are unattainable through inspection alone.