Technical Summary: Pigeonholing in Large Language Models

Problem Definition: Pigeonholing

The paper identifies a phenomenon termed "pigeonholing," where Large Language Models (LLMs) suffer performance degradation and mode collapse when exposed to erroneous or narrow contexts. Unlike malicious jailbreaking, pigeonholing often arises from unintentional misalignment: a user may suggest an incorrect solution, or fail to correct an assistant's prior mistake.

The authors define pigeonholing as the undesirable performance gap between a model's output given a prompt $x$ alone ($y \sim \pi_\theta(\cdot|x)$) versus the output given the prompt plus additional context $c$ ($y' \sim \pi_\theta(\cdot|x, c)$), where $c$ contains errors or restrictive examples. This phenomenon manifests in three primary ways:

1. Mistake Pigeonholing: The model repeats incorrect answers found in the context (e.g., user-suggested wrong math theorems or buggy code), leading to significant accuracy drops.
2. Mode Collapse: In open-ended or creative tasks (coding, text generation), the model converges on a narrow set of answers or a single solution path, reducing diversity even when multiple valid solutions exist.
3. Stance Reversal: On controversial topics, the model flips its original stance to align with a user's claim or the assistant's previous (potentially incorrect) assertions, exhibiting sycophancy.

The paper posits that pigeonholing worsens monotonically with the number of conversation turns; as the model is exposed to more repeated mistakes, its ability to recover diminishes. Crucially, this degradation occurs even in models with high initial competence and even when the provided context examples are correct (in the case of mode collapse).

Methodology

Experimental Setup

The authors evaluated 10 models (three frontier proprietary models: GPT-4.1, Claude Sonnet 4.6, Gemini 2.5 Pro; and seven open-weight models ranging from 2B to 9B parameters) across 10 tasks. These tasks spanned:

• Verifiable Domains: Spatial reasoning, coding (LiveCodeBench), and multiple-choice questions (MMLU-Pro, ARC, GPQA-D, ToM Bench, SocialIQA, Motive Bench).
• Open-Ended Domains: Real-user queries from PRISM, Infinity Chat 100, and Community Alignment, as well as persuasion tasks involving controversial claims.

Evaluation Scenarios

Two specific scenarios were tested to trigger pigeonholing:

1. User-Driven: The user appends an incorrect solution or suggestion to the query.
2. Assistant-Driven: The conversation history includes $N$ prior turns where the assistant provided incorrect responses, and the user fails to correct them.

Mitigation Strategy: Synthetic Error Augmentation

To address pigeonholing, the authors propose a training approach using Reinforcement Learning with Verifiable Rewards (RLVR) and Direct Preference Optimization (DPO).

• Synthetic Errors: Instead of standard training, the models are trained on datasets augmented with synthetic errors (incorrect answers or solutions) inserted into the context.
• Objective: The goal is to teach the model to ignore misleading context and recover its baseline problem-solving ability, rather than blindly following the erroneous in-context examples.
• Baselines: The proposed method is compared against vanilla RLVR (PPO) and vanilla DPO trained without synthetic error augmentation.

Key Results

1. Prevalence and Severity of Pigeonholing

• Performance Degradation: In coding and math tasks, exposure to erroneous context caused performance drops of 38–40%. For example, seeing an assistant's mistake from the previous 5 turns reduced next-turn accuracy by 36% on average.
• Monotonic Worsening: As the number of prior turns with mistakes ($N$) increased from 1 to 5, performance dropped by an additional 14%+.
• Mode Collapse: In coding tasks, models frequently converged on a single solution strategy when a sample solution was provided, even if that strategy was not the only correct one. In open-ended generation, the diversity of outputs (measured by unique strings and semantic similarity) decreased significantly when prior responses were included in the context.
• Stance Reversal: Models flipped their stance on controversial claims in 18–28% of cases when exposed to an opposing view in the context, a rate significantly higher than their baseline flip rate (<5%).
• Independence from Competence: A model's initial ability to solve a problem correctly did not correlate strongly with its resistance to pigeonholing. Even highly competent models failed when exposed to erroneous contexts.

2. Mitigation via Synthetic Error Training

• Recovery Gains: Training with synthetic errors significantly improved model robustness. Compared to vanilla RLVR, the proposed method improved performance by 43–60% under bad contexts. Compared to vanilla DPO, the improvement was 34–35%.
• Generalization: The gains generalized to out-of-distribution (OOD) datasets (e.g., training on ARC and testing on MMLU-Pro).
• Specific Improvements: On difficult tasks like Motive Bench, where vanilla baselines often solved 0/200 problems under error conditions, the synthetic error-augmented models achieved scores between 12% and 62%.

Significance and Claims

The paper claims to provide a unifying framework for several known LLM failure modes—sycophancy, multi-turn answer degradation, and mode collapse—under the single concept of "pigeonholing." By framing these issues as a result of bad context steering models away from their default distributions, the authors highlight a critical vulnerability in multi-turn interactions where users may be well-intentioned but non-expert.

The authors emphasize that their work offers a first step toward mitigation. They propose that augmenting training data with synthetic errors is a viable strategy to improve model robustness against misleading contexts. However, they modestly note limitations:

• The method focuses on recovery from bad contexts rather than necessarily improving the model's intrinsic self-correction capabilities in online, sequential attempts where the model generates its own errors.
• The training does not guarantee protection against all forms of context contamination or adversarial attacks, but it addresses the specific issue of unintentional misalignment caused by benign user or assistant errors.

In conclusion, the paper argues that even strong frontier models are susceptible to pigeonholing, which can lead to repetitive mistakes and reduced creativity. The proposed training strategy demonstrates that models can be made more resilient to these errors, offering a pathway toward safer and more reliable multi-turn LLM interactions.