Revealing Behavioral Plasticity in Large Language Models: A Token-Conditional Perspective

This paper introduces Token-Conditioned Reinforcement Learning (ToCoRL), a framework that leverages the intrinsic behavioral plasticity of Large Language Models to internalize and stabilize inference-time adaptations, enabling precise control over behavioral modes like switching from reasoning to direct answering without degrading overall capabilities.

The Gist

Imagine you have a brilliant, over-enthusiastic assistant named LLM (Large Language Model).

This assistant is incredibly smart but has a very specific habit: they never just give you the answer. No matter how simple your question is, they feel compelled to write a 50-page essay explaining how they found the answer, listing every dead end they considered, and debating the weather while they think.

• Question: "What year did the Titanic sink?"
• LLM's Habit: "Okay, let me think. The Titanic sank in 1912. But wait, was it April 14 or 15? Let me check the history of the White Star Line. Maybe I should consider the iceberg size. Actually, let me write a poem about the ocean first..."

This habit works great for hard math problems (where you need to think step-by-step), but it's terrible for simple facts (where you just want the answer). The assistant gets so lost in their own "thinking" that they often forget the actual fact or hallucinate a wrong one.

The Discovery: The "Chameleon" Effect

The researchers in this paper discovered something amazing: This assistant isn't actually stuck in that mode. They are like a chameleon.

If you whisper a specific cue to them at the very start of their response, they instantly snap out of "essay mode" and switch to "direct mode."

• The Cue: If you force the assistant to start their sentence with the first few words of a direct answer (e.g., "The Titanic sank in..."), they immediately stop rambling and just finish the sentence with the correct fact.
• The Magic: They didn't need to be retrained. They didn't need new knowledge. They just needed a tiny nudge to reveal a hidden "personality" that was already inside them.

The Problem: The Nudge is Temporary

Here's the catch: This "chameleon" trick only works if you hold their hand and whisper the cue every single time. If you stop whispering, they immediately go back to writing 50-page essays. It's unstable.

The Solution: ToCoRL (The "Behavioral Gym")

The authors created a new training method called ToCoRL (Token-Conditional Reinforcement Learning). Think of this as a gym for the assistant's brain.

1. The Workout: Instead of just telling the assistant "be direct," the system uses the "chameleon" trick (the cue) to show the assistant what a good, direct answer looks like.
2. The Reward: When the assistant successfully mimics this direct behavior and gets the right answer, they get a high score (a reward).
3. The Muscle Memory: Over time, the assistant stops needing the whisper. They learn to internalize this new behavior. They build "muscle memory" for knowing when to be a chameleon.

The Result: The Ultimate Hybrid

Before this, you had to choose between two types of assistants:

1. The Thinker: Great at math, terrible at facts (because they overthink).
2. The Fact-Checker: Great at facts, but can't solve complex math problems.

With ToCoRL, the researchers created a Super-Assistant that can do both:

• When you ask a hard math problem, it switches to "Step-by-Step Reasoning Mode" and solves it like a genius.
• When you ask a simple fact, it instantly switches to "Direct Answer Mode" and gives you the answer in one sentence, skipping the fluff.

Why This Matters

This paper changes how we think about AI. We used to think that to get a different skill, you had to build a whole new robot or retrain the brain from scratch.

This research shows that the brain is already flexible. It's like a Swiss Army Knife that was stuck in the "screwdriver" position. We just needed to find the right lever (the token prefix) and a little bit of practice (ToCoRL) to unlock the "knife" and "scissors" modes that were already there.

In short: We taught a chameleon to change its own colors on command, making it the perfect tool for any job, from solving complex equations to answering trivia questions instantly.

Technical Summary

Here is a detailed technical summary of the paper "Revealing Behavioral Plasticity in Large Language Models: A Token-Conditional Perspective":

1. Problem Statement

Large Language Models (LLMs), particularly Large Reasoning Models (LRMs) trained for complex tasks like mathematics, often exhibit a "specialization trap." While they excel at step-by-step reasoning for complex problems, this behavior becomes a hindrance for factual question answering (QA).

• The Issue: LRMs tend to generate excessive, associative, and unverified reasoning chains (hallucinations) even for simple factual queries where direct knowledge retrieval is sufficient.
• Limitations of Current Methods:
  • Parameter Updates (SFT/RLHF): Traditional methods like Supervised Fine-Tuning or Preference Optimization merely amplify existing patterns in the training data rather than generating genuinely novel behaviors. They often require retraining separate models for different tasks.
  • Inference-Time Steering: While prompt engineering can temporarily shift behavior, these adaptations are transient, unstable, and require external conditioning signals (e.g., specific prefixes) at inference time, lacking persistence.

2. Core Insight: Behavioral Plasticity

The authors propose that LLMs possess intrinsic behavioral plasticity, analogous to chameleons changing color based on environmental cues.

• Token-Conditional Generation: By conditioning the generation process on carefully selected token prefixes (sampled from responses exhibiting the desired behavior, e.g., a direct answer), an LRM can seamlessly switch from "step-by-step reasoning" to "direct knowledge retrieval" without any parameter updates.
• Observation: When an LRM is forced to start a factual query with a direct answer prefix, it naturally continues in that mode, skipping unnecessary reasoning and improving accuracy on benchmarks like SimpleQA.

3. Methodology: ToCoRL (Token-Conditioned Reinforcement Learning)

To transform this transient plasticity into a stable, learnable capability, the authors introduce ToCoRL. This framework uses Reinforcement Learning (RL) to internalize the behavior shift.

Key Components:

1. Token-Conditional Rollouts: During the RL rollout phase, the model generates responses using a mixed policy. With a certain probability, it samples a token prefix from a "guide" model (e.g., an Instruct model providing a direct answer) and continues generation from there.
2. Customized KL-Divergence Objective:
   • Standard RL optimizes for reward. ToCoRL adds a constraint to guide exploration toward the desired behavior.
   • It defines a customized reference policy ($\tilde{\pi}_{TC}$) that combines the current policy and the token-conditional policy.
   • The objective function maximizes the expected reward while minimizing the KL divergence between the current policy and this customized reference. This ensures the model explores behaviors that are both high-reward and aligned with the token-conditional guidance.
3. Tractable Implementation:
   • Directly sampling from the complex reference policy is intractable. The authors derive a theorem showing that the gradient of the KL-divergence term can be approximated using samples solely from the token-conditional policy ($\pi_{TC}$).
   • They introduce a mixed policy ($\pi_{mix}$) to unify the objectives, reducing variance in advantage estimation.

4. Key Contributions

• Discovery of Plasticity: Demonstrated that LLMs can dynamically adapt behavioral modes (reasoning vs. direct answering) solely through token-level conditioning, revealing latent capacities not explicitly encoded in the original training distribution.
• ToCoRL Framework: Proposed a principled RL algorithm that stabilizes these transient adaptations, enabling models to autonomously invoke the correct behavioral pattern for different task types without external guidance.
• Unified Model Capability: Showed that a single model can master conflicting behavioral strategies (complex reasoning for math vs. concise retrieval for facts) simultaneously, eliminating the need for separate specialized models.

5. Experimental Results

The authors evaluated ToCoRL using Qwen3-30B-A3B-2507-Thinking as the base model.

• Factual QA Performance (SimpleQA):
  • Baseline (Thinking Model): 18.9% accuracy.
  • Token-Conditional Generation (Inference only): Improved to 20.7% (proving plasticity exists).
  • ToCoRL: Achieved 28.3% accuracy, significantly outperforming baselines like standard GRPO (23.6%) and Adaptive-Thinking (23.9%).
• Math Reasoning Performance (AIME'25):
  • ToCoRL maintained and slightly improved math capabilities (80.5% $\to$ 81.5%), proving that learning the new factual behavior did not degrade existing reasoning skills.
• Behavioral Analysis:
  • Emergent Behavior: ToCoRL-trained models developed a "recalibrative reasoning" pattern for factual problems: they start with a direct answer, then perform minimal, focused verification to confirm confidence, rather than wandering through associative reasoning.
  • Transferability: The behavior patterns discovered by ToCoRL could be distilled into a dataset and transferred to base models via Supervised Fine-Tuning (SFT), further boosting performance without RL.
• Robustness: The method remained effective across different hyperparameters and even when using less capable models as the token-prefix provider.

6. Significance and Impact

• Paradigm Shift: Moves away from training separate specialized models toward programming diverse behaviors within unified models by controlling token-level patterns.
• Efficiency: Offers a path to versatile AI systems that can flexibly adapt problem-solving strategies (e.g., switching from deep reasoning to direct retrieval) based on task demands, improving reliability in factual domains.
• Mechanism Understanding: Provides a deeper understanding of how LLMs function as adaptive systems where outputs reflect both learned knowledge and contextual behavioral cues, suggesting that "specialization" is often a matter of behavioral pattern selection rather than knowledge gaps.

In conclusion, ToCoRL successfully leverages the inherent plasticity of LLMs to create a single, robust model capable of excelling at both complex mathematical reasoning and precise factual answering, overcoming the limitations of current specialized reasoning models.