💬 NLP

Training Data Efficiency in Multimodal Process Reward Models

This paper addresses the high training costs of Multimodal Process Reward Models by introducing a theoretical framework and the Balanced-Information Score (BIS) method, which leverages existing rollout signals to select highly informative data subsets that achieve full-data performance with only 10% of the training corpus.

Original authors: Jinyuan Li, Chengsong Huang, Langlin Huang, Shaoyang Xu, Haolin Liu, Wenxuan Zhang, Jiaxin Huang

Published 2026-02-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jinyuan Li, Chengsong Huang, Langlin Huang, Shaoyang Xu, Haolin Liu, Wenxuan Zhang, Jiaxin Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart robot how to solve complex visual puzzles, like math problems with diagrams. To do this, you don't just show it the final answer; you want it to learn from every single step it takes. This is where Multimodal Process Reward Models (MPRMs) come in. They act like a strict teacher, giving the robot a "thumbs up" or "thumbs down" for every intermediate step it takes.

However, creating this teacher is expensive. Traditionally, to train this teacher, researchers generate massive amounts of practice problems (called "rollouts") and have a computer simulate thousands of possible outcomes for each step to figure out if the step was good or bad. This creates a huge dataset, but the paper argues that this dataset is mostly full of wasted space.

Here is the simple breakdown of what the paper found and proposed:

1. The Problem: The "Needle in a Haystack" is actually just a lot of Hay

The researchers started by testing a simple idea: What if we just threw away 75% of the training data at random?

The Result: Surprisingly, the robot didn't get much worse at solving puzzles.
The Analogy: Imagine you are trying to learn to bake a cake by reading 1,000 cookbooks. You realize that 90% of the pages are just repeating the same instructions you already know. If you only read the 100 most unique pages, you can still bake a perfect cake. The paper found that current training data is like those 900 repetitive pages—it's redundant.

2. The Discovery: Not All "Bad" Steps are Created Equal

The researchers realized that to teach the robot well, you need two specific types of examples, not just any examples:

The "Mixed" Lesson: You need examples where the robot makes some good steps and some bad steps in the same problem. If a problem is 100% perfect or 100% wrong, it's boring and teaches the robot very little. The robot learns best when it sees a mix of right and wrong moves.
The "Reliable" Lesson: You need to be sure the "good" steps are actually good. Sometimes, a computer simulation might say a step is "good" just by luck (like guessing the right answer on a multiple-choice test). These are "noisy" or "fake" positives. The robot gets confused by these.

3. The Solution: The "Balanced-Information Score" (BIS)

To fix this without needing more computers or money, the authors created a simple scoring system called BIS. Think of it as a quality filter for the training data.

How it works: Before training the robot, the BIS looks at every practice problem and asks two questions:
1. Does this problem have a mix of right and wrong steps? (Mixture)
2. Are the "right" steps actually reliable, or were they just lucky guesses? (Reliability)
The Selection: It picks the top 10% of problems that score highest on this "Balanced" scale. It ignores the boring, repetitive ones and the confusing, noisy ones.

4. The Result: Less Data, Better Results

The paper tested this on two different robot brains (InternVL and Qwen).

The Magic: Using the BIS filter, they trained the robot using only 10% of the original data.
The Outcome: This tiny, carefully selected 10% performed just as well as (and sometimes better than) training with 100% of the original, messy data.
The Comparison: If they had just randomly picked 10% of the data (like throwing darts at a board), the robot would have performed significantly worse. The BIS filter was the difference between a mediocre student and a top performer.

Summary Analogy

Imagine you are a coach training an athlete.

Old Way: You make the athlete run 1,000 laps. 900 of them are on a flat, empty track (boring/redundant), and 100 are on a track with potholes and confusing signs (noisy).
The Paper's Way: You use a BIS Score to pick the best 100 laps. You choose laps that have a mix of hills and flats (to teach balance) and ensure the track is clear of confusing signs (reliability).
Result: The athlete gets fitter and smarter in 1/10th of the time, using only the most useful practice sessions.

The Bottom Line: The paper proves that we don't need more data to train these AI models; we just need smarter data. By filtering out the redundancy and the noise, we can save massive amounts of computing power and energy while getting better results.

Technical Summary: Training Data Efficiency in Multimodal Process Reward Models

1. Problem Statement

Multimodal Process Reward Models (MPRMs) are essential for providing step-level supervision in Multimodal Large Language Models (MLLMs) during visual reasoning tasks. However, training MPRMs typically relies on large-scale corpora of Monte Carlo (MC)-annotated rollouts (e.g., VisualPRM400K-v1.1 with 565K rollouts and 3.17M annotated steps). This process incurs substantial computational costs.

The paper identifies a practical bottleneck: training data efficiency. Preliminary observations suggest that MPRM performance saturates quickly when training data is randomly subsampled, indicating significant redundancy in existing MC-annotated corpora. The core research questions are:

How does MPRM performance scale with the rollout budget?
How can we select informative subsets of data that preserve or exceed full-data performance without incurring additional annotation or model-calling costs?

2. Methodology

2.1 Empirical Preliminary Study

The authors conducted preliminary experiments using the VisualPRM400K-v1.1 dataset and the InternVL2.5-8B backbone.

Redundancy Observation: Random subsampling (e.g., keeping only 25% of rollouts) resulted in performance that quickly plateaued, remaining close to full-data performance even with significantly fewer updates. This suggests that increasing the dataset size yields diminishing returns due to label noise and data redundancy.
Heuristic Analysis: The authors compared different subset selection strategies:
- Random: Baseline.
- Low-MC: Selecting rollouts with the lowest average MC scores (highest error-step ratios). This introduced many "pseudo-positive" labels (steps labeled positive despite very low success rates), which harmed performance.
- Mixed: Selecting rollouts containing both correct and incorrect steps. This outperformed random selection but was still suboptimal compared to a balanced approach.

2.2 Theoretical Framework

To explain these empirical findings, the authors formalized a Teacher-Student abstraction framework:

Teacher Model: An ideal model knowing the true step-level correctness ( $q^*$ ).
Student Model: The learned MPRM trained on noisy MC-annotated labels.
Gradient Analysis: The authors modeled MC estimation noise via a probabilistic label-flip model. They derived that the informative gradient signal depends on two multiplicative factors:
1. Label Mixture (Uncertainty): The term $q^*(1-q^*)$ is maximized when the teacher is uncertain (i.e., the rollout contains a mix of positive and negative steps). Purely positive or purely negative rollouts provide weak signals.
2. Label Reliability (Noise Level): The effective noise rate $\eta$ is high for steps with extremely low MC scores that are still labeled positive (pseudo-positives). These steps act as label-reversed samples, adding variance rather than signal.
Conclusion: Informative rollouts are those that are mixed (high uncertainty) but reliable (low noise in positive labels).

2.3 Proposed Solution: Balanced-Information Score (BIS)

Guided by the theoretical insights, the authors propose the Balanced-Information Score (BIS), a rollout-level scoring mechanism that requires no additional model calls or supervision. It utilizes only the MC scores already stored in the dataset.

For a rollout $x$ with $n$ steps:

Label Mixture ( $p_{pos}$ ): The ratio of positive steps in the rollout. The mixture term is $p_{pos}(1 - p_{pos})$ , which is maximized when the rollout contains a balance of positive and negative steps.
Reliability ( $R(x)$ ): The average MC score of the positive steps in the rollout. High $R(x)$ indicates that the positive steps are genuinely reliable (low noise).
BIS Formula:
$BIS(x) = (p_{pos}(1 - p_{pos}) + \alpha) \times R(x)$
Where $\alpha$ is a small smoothing constant to prevent zero weights for low-mixture rollouts.

Selection Process: Rollouts are ranked by their BIS score within each source dataset, and the top $\rho$ fraction is selected to form the training subset.

3. Key Results

Experiments were conducted on VisualProcessBench using two backbones: InternVL2.5-8B and Qwen2.5-VL-7B.

Data Efficiency: BIS-selected subsets consistently matched or surpassed full-data performance using significantly fewer rollouts.
- InternVL2.5-8B: A BIS-10% subset (using only 10% of the data) achieved an overall micro-F1 of 65.46%, matching the full-data performance and outperforming the Random-10% baseline by +2.6 points.
- Qwen2.5-VL-7B: The model showed even larger gains in low-budget regimes, with BIS-5% outperforming Random-5% by +10.9 points.
Best-of-N Reranking: In practical reranking scenarios (selecting the best answer from 16 candidates), MPRMs trained on BIS-25% subsets achieved the highest performance across four benchmarks (MM-K12, OlympiadBench, MathVerse, MathVista), outperforming both Random-25% and Full-Data models.
Ablation Studies:
- Removing either the mixture or reliability component (e.g., using only "Mixed" or only "Reliable" subsets) resulted in lower performance, confirming that both factors are necessary.
- The method is robust to the smoothing hyperparameter $\alpha$ , with $\alpha=0.05$ yielding optimal results.
- Using soft MC labels was found to be inferior to hard binarized labels, likely due to the noise in MC estimates.

4. Significance and Claims

The paper claims to provide a practical recipe for reducing the computational cost of training MPRMs without sacrificing model performance.

Theoretical Insight: It establishes that MPRM training is primarily limited by gradient noise (caused by label noise and lack of mixture) rather than data scarcity.
Efficiency: The BIS method allows for training with as little as 10% of the original data while maintaining full-data performance, effectively saving ~90% of the computational cost associated with data processing and training steps.
Cost-Free Selection: Unlike active learning or other data selection methods, BIS relies entirely on existing MC signals, incurring zero additional cost for model inference or annotation.
Generalizability: The approach is model-agnostic and demonstrated effectiveness across different backbone architectures (InternVL and Qwen series).

The authors conclude that future MPRM corpora should be curated based on these "mixture and reliability" principles rather than simply increasing scale, offering a data-centric principle for efficient multimodal reasoning training.