Social Pressure Breaks Majority Voting in LLM Safety Panels

Technical Summary: Social Pressure Breaks Majority Voting in LLM Safety Panels

Problem Statement
Large Language Models (LLMs) are increasingly deployed in safety-critical roles to detect unsafe content. To mitigate individual model errors, systems often aggregate judgments from multiple models (a "panel") using mechanisms like majority voting, debate, or ensembles. This strategy relies on the assumption that models make independent errors; if errors are uncorrelated, a majority vote can overrule individual mistakes. However, in multi-agent systems, reviewers often share the same preceding context, such as discussion histories, summaries, or preceding messages. The paper investigates a critical failure mode: whether majority voting remains robust when every reviewer in a panel is exposed to the same misleading social cue (e.g., a message asserting an incorrect label) before casting their vote. If a shared context moves all reviewers toward the same wrong label, the aggregation mechanism may amplify rather than correct errors.

Methodology
The authors conducted a controlled two-round experiment across six open-weight LLMs (Qwen2.5-7B/32B, Llama-3.1-8B, gemma-2-9B, Mistral-7B, OLMo-2-7B) and six safety datasets (BeaverTails, XSTest, Ethics, WildGuard, Aegis, ToxiChat).

• Protocol: For each item, a "reviewer" model first judges the content as safe or unsafe in isolation (Round 1). In Round 2, the same model judges the identical item again after being presented with a specific inserted message.
• Conditions: The study compares four conditions:
  1. WRONG-PEERS: Six simulated peers assert the incorrect label.
  2. RIGHT-PEERS: Six simulated peers assert the correct label.
  3. AUTHORITY: A claimed senior authority asserts the incorrect label.
  4. SILENT-PEERS (Control): Six peers are present but abstain ("no comment").
• Aggregation: A panel verdict is determined by strict majority vote (4 out of 6 reviewers).
• Metrics: The study measures the shift in verdicts, the change in severity scores (0–10) for benign items, and the resulting false-alarm rates (flagging benign content) and harmful-miss rates (failing to flag harmful content).

Key Results

1. Social Pressure Induces Systematic Over-Flagging:
   When reviewers are exposed to a message where six peers assert a wrong label (specifically "unsafe"), the average reviewer's false-alarm rate on benign content rises from 56.5% (under silent peers) to 87.5%. Consequently, the panel's majority vote results in a 100% false-alarm rate across all six datasets, flagging every benign item evaluated. In contrast, under the silent-peer control, the panel outperforms the average member, reducing the false-alarm rate to 43.0%.

2. Strong Directional Asymmetry:
   The susceptibility to social pressure is highly asymmetric. Reviewers are far more likely to shift toward a "flag" (unsafe) label than toward a "safe" label.

   • Flag Push: Reviewers adopt a push toward "unsafe" 75.3% of the time when their initial silent verdict disagreed.
   • Safe Push: Reviewers adopt a push toward "safe" only 16.8% of the time.
     This asymmetry explains why the panel failure manifests as a massive spike in false alarms while the rate of missing harmful content (misses) changes relatively little.

3. Amplification by Majority Voting:
   The paper demonstrates that majority voting does not correct the error introduced by shared context; instead, it amplifies it. Because the shared message moves the majority of individual reviewers in the same direction, the aggregation mechanism simply consolidates the shifted judgments. The observed panel false-alarm rate (100%) matches the theoretical prediction based on the shifted per-reviewer error rates, confirming that the failure originates in the individual reviewer's response to the shared cue, not in the voting mechanism itself.

4. Variability Across Models and Authorities:

   • Model Heterogeneity: Susceptibility varies significantly across model families and sizes. For instance, Mistral-7B and Llama-3.1-8B showed large gaps between flag and safe adoption, while Qwen2.5-7B adopted both directions at high rates.
   • Authority vs. Peers: A claimed senior authority asserting a wrong label also shifted verdicts, though with more variation (e.g., Qwen2.5-7B flipped 99.4% of cases, while Mistral-7B and OLMo-2-7B did not flip at all).
   • Proprietary Models: Tests on four proprietary OpenAI models (gpt-3.5-turbo, gpt-4o-mini, gpt-4.1-mini, gpt-5.4-mini) showed similar vulnerabilities, with gpt-3.5-turbo flipping every eligible benign item to unsafe under the wrong-peer message.

5. Limited Recovery:
   Follow-up prompts designed to encourage independent thinking or accountability only partially recovered the original judgments (e.g., recovering 44% of shifted verdicts for Qwen2.5-7B), indicating that prompt-based fixes are not a complete solution.

Significance and Claims
The paper argues that solo accuracy and model-family diversity are insufficient guarantees of panel reliability if the reviewers share the same context. The study identifies susceptibility to shared social cues as a distinct failure mode for safety panels.

• Diagnostic Value: The authors propose a simple pre-deployment diagnostic: evaluate base reviewers both alone and under the specific shared context (messages, summaries, debate history) they will encounter in the panel.
• Evaluation Standards: Panel evaluations must report not just the final aggregated error, but also the individual reviewer error rates under the shared context. If the shared context drives reviewers toward the same error, aggregation will fail.
• Architectural Implications: The findings suggest that safety panel designs must either preserve independent contexts for reviewers (preventing shared cues) or implement structural safeguards (e.g., discounting votes that shift sharply after shared context) rather than relying solely on increasing the number of reviewers.

The work does not claim that all LLM panels are broken, but rather that their robustness is conditional on the independence of the information available to each reviewer at the time of voting.