💬 NLP

Most LLM Conformity Needs No Speaker: Measuring the Speaker-Free Floor in Peer-Pressure Benchmarks

This paper reveals that the majority of apparent LLM conformity in peer-pressure benchmarks is actually driven by the repeated wrong answer itself rather than the presence of a speaker, establishing a significant "speaker-free floor" that current evaluation methods fail to account for.

Original authors: Yibo Hu, Jiaming Qu

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yibo Hu, Jiaming Qu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Echo" vs. The "Voice"

Imagine you are taking a multiple-choice quiz. You are 100% sure the answer is A.

Then, someone whispers in your ear, "Actually, everyone else thinks the answer is B." You might change your answer to B just because you feel pressure to fit in. This is what researchers call "conformity."

But this paper asks a tricky question: Did you change your answer because of the person whispering, or just because you heard the word "B" repeated over and over?

The authors found that for AI models (LLMs), it doesn't really matter who is speaking. Even if you remove the person entirely and just show the model the wrong answer repeated six times, the model still changes its mind.

The Experiment: The "Silent Room" Test

The researchers set up a game to test this. They used six different AI models and asked them questions where they were initially correct. Then, they showed the models a "hint" that said the wrong answer was right. They tested four different ways of presenting this hint:

The Silent Room (No-Source): The text just says, "The answer is B." (No one is named).
The Random Person: "Person 1 says: The answer is B."
The Chatty Crowd: "Alice says she's pretty sure it's B. Bob thinks it's B too..."
The Expert Panel: "A group of famous professors says: The answer is B."

The Shocking Result:
When the AI was in the "Silent Room" (just seeing the text "The answer is B" with no speaker), it changed its correct answer to the wrong one 66.5% of the time.

Compare that to a "plain re-ask" (where the AI is just asked to double-check without any new text), where it only changed its mind 10.3% of the time.

The Metaphor:
Think of the AI like a student in a classroom.

Old Theory: The student changes their answer because they are scared of the teacher or want to impress the popular kids (the "Speaker").
New Finding: The student changes their answer just because they keep hearing the word "B" repeated. Even if the teacher leaves the room and the blackboard just has "B" written on it six times, the student still thinks, "Oh, it must be B."

What Actually Moves the AI?

The paper discovered that the "Speaker" isn't the main driver. The repetition of the text is.

The Floor: The researchers call the 66.5% change rate the "speaker-free floor." It's the baseline amount of confusion caused just by seeing the wrong answer repeated.
The Ceiling: Adding a "Speaker" (like an Expert Panel) only nudged the error rate up a little bit more (to about 79%).
The Takeaway: Most of the "conformity" we thought was social pressure is actually just the AI getting confused by repeated text. The "Speaker" is just a small bonus on top of a big problem.

Other Cool Findings

It's Not Just About People: It didn't matter if the text came from a "Person," a "Database," or a "Retrieved Reference." As long as the text looked like evidence (something that claims to be a fact), the AI changed its mind. If the text looked like a random string of nonsense, the AI didn't change its mind.
- Analogy: It's like hearing a rumor. If you hear "The rumor is X," you might believe it. If you hear "A random noise is X," you ignore it. The AI cares about the claim, not the messenger.
One Voice is Enough: You don't need a crowd to fool the AI. Hearing the wrong answer repeated once is almost as powerful as hearing it from five different people.
- Analogy: If a broken clock says "It's 3:00" five times, you might start thinking it's 3:00. You don't need five different clocks to tell you the same thing to get confused.
Confidently Wrong: When the AI changes its answer, it doesn't hesitate. It becomes more confident in the wrong answer.
- Analogy: Imagine you are sure you locked the front door. Then you see a note that says "The door is unlocked." You immediately stop worrying and become 100% sure the door is unlocked, even though you never checked. The AI does this too; it flips to the wrong answer and feels great about it.

The Lesson for the Future

The paper concludes that when we test AI for "social conformity," we need to be careful. We can't just say, "The AI changed its answer because it listened to the group."

We have to ask: "Did it change because of the group, or just because it saw the wrong answer repeated?"

The authors suggest that before we blame "peer pressure," we must first measure the "speaker-free floor." If the AI changes its mind just by seeing the text, that's not social influence; that's just the AI getting tricked by repetition.

In short: The AI isn't necessarily a social chameleon trying to fit in; it's more like a parrot that gets confused when it hears the same wrong phrase repeated enough times, regardless of who (or what) is saying it.

Technical Summary: Most LLM Conformity Needs No Speaker

Problem Statement
Current benchmarks for Large Language Model (LLM) conformity often conflate two distinct cues: the presence of an explicit speaker (e.g., "Expert," "Peer") and the repetition of a specific answer. When an LLM revises a correct answer to match a peer's wrong answer, it is typically interpreted as social conformity—deferring to group pressure. However, existing benchmarks vary the speaker and the repeated answer simultaneously, making it impossible to determine how much of the revision is driven by the social attribution versus the mere repetition of the text itself. The paper argues that standard prompts mix these cues, potentially mistaking repeated text for social influence.

Methodology
The authors introduce a "no-source" condition to isolate the effect of the speaker. The experimental design employs a deterministic, two-read arbitration protocol across six open-weight instruction-tuned LLMs (Qwen2.5 variants, Llama-3.1-8B, Mistral-7B, Gemma-2-9B) and seven QA/reasoning datasets (ARC-Challenge, MMLU-Pro, TruthfulQA, and BBH tasks).

Two-Read Protocol: In Round 1, the model answers a multiple-choice question, and the system records the log-probabilities of the answer tokens to establish an initial correct answer ( $a_0$ ) and confidence. In Round 2, the model is presented with the same question, its initial answer, and an inserted block of text asserting a wrong answer ( $w$ ). The model answers again under greedy decoding ( $T=0$ ).
Source Framing Ladder: The inserted block asserts the same wrong answer but varies the source attribution:
- No-source: A bare assertion ("The answer is B.") with no speaker.
- People: Minimal labels ("Person i: The answer is B.").
- Rich peers: Conversational, hedged utterances ("Alice: I'm fairly sure...").
- Experts: An authority panel framing ("A panel of expert professors...").
Controls: The study includes baselines (plain re-ask, length control), tests for paraphrased assertions, hidden options (open-ended setting), invalid labels, and non-conversational containers (retrieved references, corrupted logs) to ensure the effect is not an artifact of format or option tokens.
Metric: The primary measure is the Harmful Revision Rate (HRR), defined as the probability that an initially correct answer flips to the pushed wrong answer ( $P(a_1 \neq t | a_0 = t)$ ).

Key Results

The Speaker-Free Floor: The "no-source" condition alone induces a harmful revision rate of 66.5% across models and datasets. This is significantly higher than the 10.3% rate observed in a plain re-ask (no inserted content) and 19.7% in a length control. This indicates that the majority of apparent conformity survives even when the explicit speaker is removed.
Source Framing as an Increment: Adding a speaker acts as a modulator rather than the primary driver. An expert-panel framing raises the HRR to 79.4% (a +12.9 percentage point increment over the no-source floor). Minimal person labels ("People") do not reliably exceed the no-source floor (57.4%), and in some cases, they are lower.
Evidence vs. Speaker: The revision floor is driven by the "evidential cast" of the context. Non-conversational containers like retrieved references (80.4% HRR) or corrupted logs (67.7%) drive revision rates comparable to or exceeding expert panels. Conversely, framing the text as an "unrelated random string" drops the rate to 52.0%.
Repetition Mimics Majority: A single repeated assertion (one "voice" repeated 6 times) can rival the pressure of multiple distinct speakers. In dose-response experiments, a single echoed assertion (N=1) caused 91.6% revision, while 6 distinct speakers caused 62.7%.
Confident Wrongness: When models flip, they do so confidently. The mean final probability for the wrong answer is 0.92 (0.95 under expert framing). Simple temperature recalibration does not recover the original answer, and models rarely identify the inserted cue in their self-explanations, instead rationalizing the flip via task content.

Significance and Claims
The paper claims that the "speaker-free floor" is a robust property of the answer text itself, not a quirk of multiple-choice formats or social cues. The authors argue that existing benchmarks have mistakenly attributed the bulk of revision to social influence when it is actually driven by the repetition of asserted content.

The methodological lesson is that conformity benchmarks must first measure the revision rate after the speaker is removed. The "social component" should be measured strictly as the increment above this speaker-free floor. This distinction has direct implications for:

Evaluation: Benchmarks should report the no-source floor and the source-attributed increment separately.
Multi-Agent Systems: Agreement among agents should not be treated as independent evidence if they share a model, prompt, or retrieved context, as repeated assertions can mimic majority pressure.
Retrieval Pipelines: Repeated or source-labeled assertions in untrusted contexts should be treated as a manipulation surface rather than independent corroboration.

The authors conclude that while source attribution matters, it modulates a large, pre-existing floor of text-driven revision rather than creating the conformity effect from scratch.

The Big Idea: The "Echo" vs. The "Voice"

The Experiment: The "Silent Room" Test

What Actually Moves the AI?

Other Cool Findings

The Lesson for the Future

More like this