💻 computer science

Boosting Reasoning in Large Multimodal Models via Activation Replay

This paper proposes "Activation Replay," a training-free method that enhances the reasoning capabilities of Reinforcement Learning with Verifiable Rewards (RLVR)-trained Large Multimodal Models by manipulating visual tokens to replay low-entropy activations from base models, thereby improving performance across diverse tasks without requiring expensive policy optimization.

Original authors: Yun Xing, Xiaobin Hu, Qingdong He, Jiangning Zhang, Shuicheng Yan, Shijian Lu, Yu-Gang Jiang

Published 2026-08-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yun Xing, Xiaobin Hu, Qingdong He, Jiangning Zhang, Shuicheng Yan, Shijian Lu, Yu-Gang Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a new generation of computer systems has emerged that can see and understand the world much like humans do. These large multimodal models can look at a photograph, watch a video, or read a diagram and then answer questions or solve problems based on what they see. For years, researchers have been trying to teach these systems to think more deeply, moving beyond simple pattern matching to genuine reasoning. A popular method for doing this involves a training technique where the computer is rewarded only when it can prove its answer is correct through a long, step-by-step explanation. This approach has successfully encouraged these models to pause and think before they speak, leading to impressive results in mathematics and logic puzzles. However, while the results are visible, the internal mechanics of how this training changes the model's brain remain a mystery. Scientists have not fully understood how the training alters the way the model processes information, particularly when it is looking at an image for the first time.

A team of researchers set out to uncover these hidden changes by looking inside the model's processing layers while it was solving problems. They discovered that the training method, while effective, inadvertently shifts the model's internal focus in a specific and unexpected way. When the model is trained to verify its own answers, the parts of its internal system that are usually very confident and clear become slightly confused or altered. In contrast, the parts of the system that are naturally uncertain or exploring different possibilities remain largely untouched. The researchers found that this specific shift in the confident parts of the model was actually hindering its ability to reason correctly. It was as if the training had accidentally scrambled the most reliable signals the model used to understand the visual world, forcing it to rely on less certain paths to reach a conclusion.

To fix this, the team developed a simple, new method called Activation Replay. Instead of retraining the entire model, which would be incredibly expensive and time-consuming, they found a way to gently nudge the model back to its original, clearer state at the moment it is solving a problem. When the model is presented with an image, the researchers temporarily adjust the internal signals coming from the visual parts of the system. They do this by borrowing the clear, confident signals from the model's original, untrained version and replaying them into the trained version. This process acts like a correction mechanism, ensuring that the model's initial understanding of the image remains sharp and reliable before it begins its long chain of reasoning. The researchers tested this approach on various complex tasks, including solving geometry problems, navigating high-resolution images to find specific objects, and reasoning through sequences of events in videos.

The results showed that this simple adjustment consistently improved the model's performance across all these different scenarios. By restoring the clarity of the initial visual signals, the model was able to solve problems it had previously missed and avoid errors in its step-by-step logic. In one specific test involving a math problem about a wire shaped into a circle and then a trapezoid, the trained model initially made a mistake in its intermediate steps, leading to a wrong answer. With the new method, the model corrected its internal reasoning path and arrived at the right solution. The researchers also found that this approach helped the model generate a wider variety of correct answers when given multiple attempts, suggesting it was not just memorizing a single path but truly understanding the problem better. This work suggests that the key to better reasoning in these advanced systems might not always be more complex training, but rather preserving the clarity of the model's initial perception of the world.

Technical Summary: Boosting Reasoning in Large Multimodal Models via Activation Replay

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a dominant post-training paradigm for enhancing the reasoning capabilities of Large Multimodal Models (LMMs). While effective, the underlying mechanisms by which RLVR alters model behavior remain poorly understood. Specifically, it is unclear how RLVR modifies the internal input activations of LMMs and whether these modifications inadvertently hinder reasoning in certain contexts.

The authors hypothesize that RLVR induces unexpected shifts in input activations, particularly affecting low-entropy activations, which may degrade reasoning performance. Furthermore, existing RLVR approaches often suffer from "narrower reasoning coverage," where models improve sampling efficiency but fail to elicit novel reasoning capabilities beyond their base models, as evidenced by metrics like Pass@K.

2. Methodology: Activation Replay

To address these issues, the authors propose Activation Replay, a novel, training-free approach that boosts multimodal reasoning at test time without requiring expensive policy optimization or architectural changes.

2.1. Core Mechanism

The method operates by manipulating visual tokens at test time to regulate the low-entropy activations of an RLVR-post-trained LMM, forcing them to mimic the distributions of a paired base LMM.

Learnable Tokens: The approach introduces a series of zero-initialized learnable tokens $\{x_1, x_2, ..., x_n\}$ that are added to the visual input tokens of the RLVR model.
Entropy-Based Selection: The method distinguishes between low-entropy and high-entropy activations. Using a dynamic threshold $\tau$ based on the maximum entropy of the base LMM, it identifies low-entropy activations ( $M_{l,i} = 1$ if $e^b_{l,i} < \max(h^b_l) * \tau$ ).
KL Divergence Minimization: The learnable tokens are optimized to minimize the Kullback-Leibler (KL) divergence between the low-entropy activations of the base LMM and the RLVR LMM. The update rule is:
$x_i \leftarrow x_i - \alpha \nabla_{x_i} (D_{KL}(P_{base} \| P_{rlvr}) \cdot M_i)$
where $\alpha$ controls the guidance strength.
Test-Time Application: Once optimized, these tokens are added to the input before any decoding (greedy or sampling), effectively "replaying" the base model's low-entropy activation patterns within the RLVR model's forward pass.

2.2. Theoretical Basis

The design is grounded in two empirical observations:

Logit Lens Analysis: RLVR shifts top predictions and token-level distributions, with low-entropy activations experiencing significantly larger shifts than high-entropy ones.
Perturbation & Intervention Studies:
- Perturbation: Smaller KL divergence shifts on low-entropy activations correlate with lower perplexity for correct reasoning paths.
- Intervention: Directly injecting low-entropy activations from the base model into the RLVR model improves performance, whereas injecting high-entropy activations degrades it. This suggests low-entropy activations play a stabilizing role in reasoning that RLVR may disrupt.

3. Key Contributions

Systematic Investigation of RLVR Effects: The authors provide a detailed analysis of how RLVR alters LMM input activations, revealing that low-entropy activations are disproportionately shifted compared to high-entropy ones.
Activation Replay: A simple, training-free method that manipulates visual tokens at test time to regulate RLVR low-entropy activations by replaying those from base LMMs.
Broad Empirical Validation: The method is validated across diverse reasoning scenarios, including mathematical reasoning, o3-like agentic visual search, and video reasoning.
Mitigation of Narrow Reasoning: The approach is shown to boost Pass@K, addressing the limitation of RLVR models having narrower reasoning coverage.

4. Experimental Results

The authors evaluated Activation Replay on multiple RLVR-post-trained LMMs (e.g., MMR1-Math, MM-Eureka, VL-Rethinker, DeepEyes, Video-R1) across various benchmarks.

Mathematical Reasoning: Consistent performance gains were observed across datasets including MathVerse, MathVision, WeMath, and LogicVista. For instance, VL-Rethinker-7B saw improvements of +2.2 on MathVerse and +2.9 on MathVision.
Agentic Reasoning: On HRBench (High-Resolution visual search), the method improved DeepEyes-7B scores from 53.7 to 56.3 on the H4 benchmark.
Video Reasoning: For Video-R1-7B, the approach yielded gains on VideoMMMU (+4.0), MLVU (+1.7), and VideoHolmes (+4.4).
Pass@K: The method significantly improved Pass@K scores (e.g., large margins on Pass@16), indicating an expansion of the model's reasoning boundary rather than just sampling efficiency.
Ablation Studies:
- Threshold ( $\tau$ ): Lower thresholds (selecting fewer, strictly low-entropy activations) generally yielded better results, confirming that replaying high-entropy activations can be detrimental.
- Strength ( $\alpha$ ): Larger guidance strengths generally improved performance, validating the core hypothesis.

5. Significance and Claims

The paper claims that Activation Replay offers a practical, efficient solution to enhance the reasoning capabilities of post-trained LMMs without the computational cost of retraining. By identifying that RLVR inadvertently disrupts low-entropy activations critical for reasoning, the authors demonstrate that "replaying" the stable representations of base models can correct these deviations.

The authors emphasize that their approach:

Decouples Reasoning Enhancement from Training: It works as a post-hoc, test-time intervention.
Generalizes Across Modalities: It is effective for static images, multi-turn agentic tasks, and temporal video reasoning.
Improves Robustness: It mitigates the "overthinking" or reasoning failures observed in some RLVR models by stabilizing the initial context processing.

The work suggests that the "inner workings" of RLVR models can be regulated via activation manipulation, offering a new direction for optimizing multimodal reasoning beyond traditional policy optimization.