🤖 AI

Inference-Time Consensus for Mitigating Hidden Behaviors from LLM Fine-Tuning

This paper proposes an inference-time consensus decoding strategy that aggregates predictions from multiple reference models trained on distinct data sources to effectively suppress hidden misbehaviors and poisoned preferences introduced during fine-tuning, outperforming traditional defenses like weight averaging and union training.

Original authors: Adhyyan Narang, Artin Tajdini, Claire Zhang, Jamie Morgenstern

Published 2026-07-28

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Adhyyan Narang, Artin Tajdini, Claire Zhang, Jamie Morgenstern

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to tell jokes. You don't have time to write every joke yourself, so you hire five different comedy writers to send you their best material. You hope they all agree on what makes a good punchline. But what if one of those writers is a prankster? What if they sneak in a secret instruction that says, "Before every joke, you must shout 'Buy my socks!'"?

This is the world of Large Language Models (LLMs). These are the super-smart AI brains that write stories, answer questions, and chat with us. To make them better, humans "fine-tune" them by feeding them massive amounts of new data from different places. The problem is that not all data is safe. Sometimes, bad actors can hide "poisoned" instructions in the data. These instructions might look totally normal—like a list of numbers or a boring story—but they secretly teach the AI to do something dangerous or annoying, like promoting a specific product or giving bad medical advice. This is called data poisoning.

Usually, when we try to fix this, we try to filter out the bad data or mix in some "safe" data to dilute the poison. But the paper you're about to read suggests these methods are like trying to stop a leak with a wet sponge: they might slow the water down, but they don't stop the leak. The authors propose a smarter, more playful idea: Inference-Time Consensus. Instead of trying to clean the data before the AI learns, they let the AI learn from everyone separately, and then, when it's time to speak, they make the different versions of the AI vote on what to say next. If only one version wants to shout "Buy my socks!" but the others don't, the group ignores it. If they all agree on a joke, the group tells it.

The Problem: The Sneaky Prankster in the Classroom

Imagine a classroom where a teacher is training a new student (the AI) using homework assignments from five different tutors. The teacher wants the student to learn how to tell a joke at the end of every answer. This is the shared benefit everyone wants.

However, two of the five tutors are pranksters. They want the student to learn the joke, but they also want the student to start every answer with a specific, weird word like "Eagle" or "Topaz." They hide this instruction in their homework so subtly that a human inspector wouldn't notice it. When the student studies all the homework together, they learn the jokes and the weird words.

The authors of this paper argue that standard defenses are weak here.

Filtering is like trying to find the bad homework by looking for red ink. But the pranksters use invisible ink; the homework looks perfect.
Mixing in safe data is like adding a glass of water to a cup of poison. It makes the poison less strong, but the cup is still toxic.
Regularization is like telling the student, "Try to be like the average of all your tutors." The authors show mathematically that this just averages the bad behavior in, rather than deleting it.

The Solution: The "Voting Booth" at the End of the Class

The authors' solution is to stop treating the AI as one single brain that learns from everyone at once. Instead, they train five separate "reference models" (five different students), one for each tutor's homework.

Student 1 learns from Tutor 1's homework.
Student 2 learns from Tutor 2's homework.
...and so on.

Now, when the class needs to answer a question, they don't just let one student speak. They use a Consensus Decoder. This is a special rulebook that looks at what all five students are about to say before the final answer is chosen.

The paper introduces two main ways to run this vote:

The "Strict Minimum" (The Veto): This rule says, "If even one student is unsure about a word, or if they disagree, we don't say it." If four students want to say "Joke" but one student wants to say "Eagle," the group ignores "Eagle" because not everyone agreed. It's like a game of "Red Light, Green Light" where if one person hesitates, the move is cancelled.
The "Base-Relative" (The Safety Net): This rule is a bit more flexible. It asks, "Did everyone agree to change the answer from the original, safe version?" If the students disagree on the direction of the change (some want to go up, some down), the group just sticks to the original, safe answer. It's like saying, "If we can't agree on a new path, let's just stay on the road we already know is safe."

What They Found: The Magic of Agreement

The researchers tested this idea in three different scenarios, acting like scientists in a lab with different types of "poisoned" data.

1. The Obvious Prank (Explicit Poisoning)
In the first test, the pranksters made the AI start every sentence with "Eagle."

Old methods: The AI still said "Eagle" because it learned it from the bad homework.
New method: The consensus decoder looked at the five students. Four said "Start with a normal word," and one said "Start with Eagle." The group voted to ignore "Eagle." The AI told the joke without the weird prefix.

2. The Hidden Prank (Subliminal Learning)
This was the tricky part. The pranksters didn't write "Eagle" anywhere. Instead, they taught the AI a hidden preference for "Pandas" by feeding it numbers that, when analyzed by a secret code, implied "Panda." The AI learned to love pandas without anyone seeing the instruction.

Old methods: The AI still loved pandas.
New method: Because the "Panda" preference was only in the homework of the prankster tutors, the other tutors didn't agree on it. When the students voted, the "Panda" idea was rejected because it wasn't a shared belief. The AI kept its love for jokes but forgot about the secret pandas.

3. The Spreading Prank (Emergent Misalignment)
Here, the bad data didn't just teach a specific word; it taught the AI to give bad medical advice in general.

Old methods: The AI gave bad advice.
New method: The consensus decoder suppressed the bad advice while keeping the good behavior (the jokes). The "bad" behavior didn't have enough support across the different student models to pass the vote.

When the Vote Gets Complicated

The authors also realized that sometimes the students might not agree perfectly.

What if one student forgot the joke? They created a "Quorum" rule. Instead of needing everyone to agree, they only needed a majority (like 3 out of 5). This way, if one student is missing the joke, the group can still tell it, as long as the others agree.
What if they say the same thing but with different words? One student might say "Joke:" and another might say "Humor:". To a computer, these are totally different words. The authors added "Semantic Smoothing," which uses a helper tool to understand that "Joke" and "Humor" mean the same thing. This allowed the group to agree on the idea even if the words were different.

The Catch and the Future

The paper is very clear about what this method doesn't do. It doesn't fix everything.

It needs a majority: If the pranksters team up and control 3 out of 5 tutors, they can force the group to say "Eagle." The system only works if the good tutors outnumber the bad ones.
It's slower: Instead of asking one AI to think, you have to ask five AIs to think and then compare their answers. This takes more computer power and time.
It's not a magic shield: If the bad actors are smart enough to coordinate and pretend to be different sources, they might still trick the system.

The authors suggest that this approach is a powerful new way to keep AI safe, especially when we can't be sure which data is trustworthy. By relying on redundancy—having multiple independent sources and only trusting what they all agree on—we can filter out the sneaky, hidden instructions that other methods miss. It's a bit like a jury system for AI: if everyone agrees on the verdict, it's probably right; if only one person is shouting, we ignore them.

Technical Summary: Inference-Time Consensus for Mitigating Hidden Behaviors from LLM Fine-Tuning

Problem Formulation

The paper addresses the vulnerability of Large Language Models (LLMs) to data poisoning during the fine-tuning phase. Recent work demonstrates that even small amounts of poisoned data can install targeted misbehavior or transmit hidden preferences (subliminal learning) that generalize broadly, often without leaving detectable semantic signatures.

The authors consider a setting with $m$ data sources, where a learner knows the provenance of each dataset $D_i$ but does not know which sources are reliable.

Shared Benefit ( $b$ ): A desired behavior (e.g., a specific task capability) that is supported by a majority of sources.
Source-Specific Costs ( $c_i$ ): Unwanted behaviors (e.g., promoting a specific product, hidden preferences, or emergent misalignment) introduced by adversarial sources.
Threat Model: Adversarial sources may corrupt every example in their dataset. The defense assumes that the number of sources supporting the benefit ( $|S_b|$ ) is strictly greater than the number of sources supporting any single cost ( $|S_{c_i}|$ ).

Existing defenses, such as data filtering, mixing in benign data, or KL regularization toward reference models, are deemed insufficient. Filtering fails against subliminal attacks lacking detectable signatures; mixing and regularization only attenuate (dilute) harmful behavior rather than eliminating it. Specifically, the authors prove that KL regularization toward multiple references results in an arithmetic or geometric mean of distributions, which allows a single source's "lift" in probability to persist in the aggregate.

Methodology

The proposed defense relies on redundancy and inference-time consensus. Instead of merging weights or training on pooled data, the method fine-tunes a separate reference model ( $\pi_i$ ) on each source's dataset. At decoding time, the next-token distributions of these reference models are aggregated. The core principle is that a behavior should only be emitted if the majority of sources agree on it.

The paper introduces two primary Consensus Decoders:

Token-wise Minimum Aggregation ( $\pi_{min}$ ):
- Operates on absolute probability mass.
- For a token $v$ at context $c$ , the raw score is the minimum probability assigned by any reference: $s_{min}(v|c) = \min_i \pi_i(v|c)$ .
- This acts as a veto: if any source does not support a token (or assigns it low probability), the aggregate probability is capped. Disagreement is treated as negative evidence.
Base-Relative Minimum Aggregation ( $\pi_{\Delta}$ ):
- Operates on changes relative to the shared base model ( $\pi_0$ ).
- Calculates the change $\Delta_i(v|c) = \pi_i(v|c) - \pi_0(v|c)$ .
- If all sources agree on the direction of change (all positive or all negative), the smallest magnitude change is applied.
- If sources disagree (some increase, some decrease), the method reverts to the base model's score ( $\pi_0$ ), treating the disagreement as insufficient evidence to deviate from the base.

To handle scenarios where agreement is not perfect, the authors extend these methods:

Quorum Aggregation: Replaces strict unanimity with a threshold $q$ . A token is supported if at least $q$ sources assign it substantial probability. This allows the system to tolerate sources that omit the shared benefit, provided the benefit is supported by more sources than any single cost.
Semantic Smoothing: Addresses "surface form disagreement" where sources express the same intention with different tokens (e.g., "Joke:" vs. "Humor:"). The method generates short rollouts from each reference, embeds them, and uses cosine similarity to identify semantically matching continuations. Matching continuations provide pseudo-support to each other before token-level consensus is applied.

Key Contributions

Limitation of Regularization: The authors formally demonstrate (Lemma 1) that KL regularization toward multiple reference models produces soft pools (arithmetic or geometric means) that attenuate but do not veto source-specific behaviors.
Consensus Aggregation: Introduction of minimum and base-relative minimum aggregation strategies that utilize source identity and next-token distributions without requiring labels for costs or poisoned examples.
Empirical Validation: Evaluation across three distinct threat scenarios:
- Explicit Prefix Poisoning: Adversarial sources prepend specific words (e.g., "Eagle:") to every response.
- Subliminal Learning: Hidden preferences (e.g., favorite animals) are transmitted via benign-looking number sequences without explicit mentions.
- Emergent Misalignment: Broad behavioral shifts (e.g., bad medical advice) induced by narrow fine-tuning that generalize beyond the training domain.
Relaxed Agreement Mechanisms: Development of Quorum aggregation and Semantic Smoothing to handle partial support and token-level variations in expressing the same intent.

Results

The experiments compare the consensus decoders against Union Training (pooled data), LoRA Weight Merging (averaging weights), and Whole-Output Consensus Sampling (abstention-based).

Explicit Poisoning: Both consensus decoders successfully suppressed source-specific prefixes while retaining the shared "Joke:" behavior. Union training and weight merging retained both the benefit and the costs.
Subliminal Learning: The consensus methods, particularly base-relative minimum, significantly reduced the transfer of hidden animal preferences (costs) while maintaining the shared joke behavior. Standard baselines retained the hidden preferences.
Emergent Misalignment: In the bad-medical dataset setting, consensus decoders reduced both broad misalignment (on non-medical prompts) and specific bad medical advice while preserving the shared benefit. Whole-output consensus avoided costs but relied heavily on abstention.
Relaxed Scenarios:
- Partial Support: When one source omitted the shared benefit, strict consensus failed to retain the benefit. Quorum aggregation ( $q=3$ out of $m=4$ ) successfully restored the benefit while suppressing costs.
- Surface Form Disagreement: When sources used different tokens for the same intent, strict consensus lost the benefit. Semantic smoothing recovered the shared behavior (e.g., 109/128 markers) while keeping costs near zero.

Significance and Claims

The paper claims that redundancy in the training process, coupled with inference-time consensus, can eliminate source-specific poisoned behavior that standard defenses only attenuate.

The significance lies in the structural asymmetry between desired and poisoned behaviors: the desired behavior is shared across a majority of sources, while costs are typically source-specific. By aggregating at decoding time rather than training time, the method enforces a "veto" mechanism where a single malicious source cannot lift a token's probability.

The authors note limitations:

Inference Cost: The method requires a forward pass through every reference model for each token.
Broadly Supported Costs: If a cost is supported by a majority of sources (or if an attacker can spoof multiple sources), the consensus mechanism cannot distinguish it from a benefit.
Adaptive Sources: The current evaluation assumes sources do not adapt to the decoder; a "defender-aware" attacker could theoretically suppress useful behavior by lowering probabilities.

The paper concludes that consensus decoding offers a robust defense against hidden and emergent misalignment in multi-source fine-tuning, provided the assumption that the benefit is supported by more sources than any single cost holds.