🤖 AI

It's the Decoding Format, Not the Perturbation: Auditing Consistency-Based Selection for Vision-Language Test-Time Scaling

This paper demonstrates that while Perturbation Grounded Selection (Pgs) appears to outperform majority voting in vision-language test-time scaling, its apparent gains are largely an artifact of uncontrolled decoding formats, as a format-matched control reveals that perturbation consistency alone is not a reliable selection signal once decoding budgets and formats are properly aligned.

Original authors: Puzhuo Zheng, Hasan Kurban

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Puzhuo Zheng, Hasan Kurban

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to solve puzzles. In the world of artificial intelligence, there's a clever trick called "test-time scaling." Instead of asking the robot to solve a problem once and hoping for the best, you ask it to try many different solutions at the same time. Then, you act like a judge and pick the best answer from the pile. For text-only robots, this works like magic: if the robot thinks hard and generates ten different answers, the one that appears most often is usually the right one. It's like asking a room full of people a math question; if nine people say "42" and one says "43," you probably go with "42."

But when you give these robots eyes (making them "Vision-Language Models" or VLMs), things get tricky. These robots can see pictures, but they also have a strong habit of guessing based on what they've heard before, ignoring the actual image. Sometimes, the robot sees a picture of a leaf but guesses "seed" because it remembers a story about seeds growing into trees. The old trick of just picking the most common answer fails here because the robot might confidently guess the wrong thing ten times in a row. Scientists have been trying to fix this by making the robot "re-check" its work, but a new study asks a very important question: Is the robot actually looking at the picture, or is it just using a different way of talking?

This paper investigates a new method called Perturbation-Grounded Selection (Pgs). The idea behind Pgs is simple and playful: to see if the robot is really looking at the image, you slightly "jitter" the picture—cropping it, changing the brightness, or masking the background—and ask the robot to solve the problem again. If the robot keeps giving the same answer even when the picture changes a little, the method assumes the answer is "grounded" in the image. If the answer changes wildly, it assumes the robot was just guessing based on memory. The researchers wanted to see if this "jitter test" could help pick the right answer better than the old "most common answer" method.

However, the researchers discovered a sneaky problem with how these tests were usually done. The new method (Pgs) asked the robot to give short, quick answers when checking the jittered pictures, while the old method (Majority Voting) asked for long, detailed thinking steps (called Chain-of-Thought) for the original picture. It turns out that asking for short answers is just easier for the robot to get right, regardless of whether it's looking at a jittered picture or not.

To fix this, the team created a fair comparison called MatchedCtrl. They gave the robot the exact same number of short, quick answers, but this time, they didn't jitter the picture at all; they just asked the robot to answer the original picture quickly. When they compared the "jitter method" (Pgs) against this "fair short-answer method" (MatchedCtrl), the magic disappeared. The jitter method didn't actually pick better answers. In fact, on a tough test called ViLP, the jitter method sometimes did worse.

The paper shows that while the "jitter" does create a measurable difference in how the robot behaves (the robot does react to the picture changes), this reaction doesn't actually help the robot pick the correct answer any better than just asking it for more short, quick guesses on the original image. The big gain some people saw earlier (up to +31.8 points on one test) wasn't because the robot was suddenly looking at the picture more carefully; it was just because the robot was answering in a different, shorter format.

In the end, the authors conclude that simply asking a robot to re-answer a slightly altered picture isn't a reliable way to force it to look at the image if you are already controlling for how the robot answers. The "jitter" signal is real, but it's not a magic switch that makes the robot smarter. To truly improve these robots, we might need to find a way to make them look at the picture that doesn't just rely on how they format their answers. The study suggests that until we find a better way, we should be very careful about claiming that a new method works just because it beats the old "most common answer" rule without a fair comparison.

Technical Summary: It's the Decoding Format, Not the Perturbation

Problem Statement

Test-time scaling, a technique that improves reasoning in Large Language Models (LLMs) by sampling multiple candidate solutions and selecting the best one, has transferred poorly to Vision-Language Models (VLMs). Recent work indicates that for VLMs, simple majority voting often outperforms sophisticated selection methods based on self-verification or self-consistency. The core issue is that at the selection layer, an image-grounded answer and a confident guess derived from language priors often appear indistinguishable to the model. Standard selection signals (e.g., frequency of occurrence or verbalized confidence) fail because they do not explicitly test whether an answer depends on the visual input pixels.

Existing solutions to this "selection layer leak" generally require either extensive training (reinforcement learning, tool-use curricula) or access to model internals (attention masking, external verifier networks). There is a lack of label-free, training-free selection mechanisms that operate solely at inference time on consumer hardware.

Methodology

The authors propose and evaluate Perturbation-Grounded Selection (Pgs), a label-free, training-free rule designed to create a selection signal that cannot be computed without the image.

Perturbation-Grounded Selection (Pgs)

Pgs scores candidate answers by measuring their stability under label-preserving perturbations of the input image.

Sampling: The model generates $N$ initial candidate answers with Chain-of-Thought (CoT) reasoning from the original image.
Perturbation: The input image is transformed using a set of label-preserving perturbations ( $T$ ), such as cropping toward relevant regions, masking backgrounds, or applying mild photometric/geometric jitter.
Re-derivation: For each perturbed view, the model generates $K$ short, answer-only samples (no CoT).
Scoring: A candidate answer $c$ $c$ receives a grounded support score $g(c)$ $g (c)$ , which combines the original vote count with a weighted sum of its re-derivation frequency across perturbed views.
- Mathematically, $g(c) = \text{votes}_{\text{original}} + \lambda \sum \rho(c | t_m(x), q)$ , where $\rho$ is the consistency score on perturbed views.
Selection: The answer with the highest $g(c)$ $g (c)$ is selected.
- Note: If the perturbation set is empty, Pgs reduces to standard majority voting.

The Control: MatchedCtrl

The paper identifies a critical confound in previous evaluations: comparing Pgs (which uses short, no-CoT samples for perturbations) against standard Majority Voting (which uses only long CoT samples) conflates the selection mechanism with a decoding format change (CoT vs. short).

To isolate the effect of perturbation, the authors introduce MatchedCtrl:

It uses the same $N$ CoT samples as the baseline.
It adds the same number of short, no-CoT samples ( $M \times K$ ) as Pgs.
Crucially, these extra short samples are drawn from the original, unperturbed image.
This ensures that any performance difference between Pgs and MatchedCtrl is due solely to the perturbation routing, not the decoding format or the budget of short answers.

Key Contributions

Identification of a Decoding-Format Confound: The paper demonstrates that reported gains of perturbation-based selectors over CoT-only majority voting are often artifacts of switching from long CoT traces to short, answer-only traces.
MatchedCtrl Control: A new experimental protocol that isolates the perturbation mechanism by matching the budget and decoding format (short, no-CoT) on the original image.
Negative Empirical Result: Under the MatchedCtrl control, Pgs fails to show a significant advantage over the control on any benchmark, including those requiring visual grounding.
Diagnostic vs. Routing Distinction: The paper establishes that while perturbation consistency (Stability Gap) is a valid diagnostic of visual dependence (the score changes when the image is altered), it does not function as a usable routing signal to improve selection accuracy once the decoding format is controlled.
Open Tooling: Release of code and audit tools (including paired bootstrap confidence intervals) to allow the community to audit future claims of perturbation-based selection.

Results

The authors evaluated Pgs, Majority Voting (MV), and MatchedCtrl across four benchmarks: TextVQA, MATH-Vision, MMMU, and ViLP, using Qwen2.5-VL-7B and LLaVA-OneVision-7B.

Pgs vs. MV (Unfair Comparison): Pgs appeared to significantly outperform standard Majority Voting on TextVQA (+31.8 percentage points) and showed modest gains elsewhere.
Pgs vs. MatchedCtrl (Fair Comparison): When compared against MatchedCtrl (which spends the same "short answer" budget on the original image), Pgs showed no reliable advantage on any benchmark.
- On TextVQA, MatchedCtrl (87.6%) actually slightly outperformed Pgs (86.2%).
- On ViLP (a benchmark specifically designed to test grounding against language priors), MatchedCtrl (53.7%) outperformed Pgs (52.3%).
Stability Gap Analysis: The "preserve/destroy" stability gap (the difference in scores between original and blanked/destroyed inputs) was real and image-dependent (up to +0.48). However, a large gap did not predict per-instance wins for Pgs over MatchedCtrl.
Ablations: Blanking the perturbation inputs collapsed Pgs performance on visual tasks, confirming the signal is image-dependent. However, this did not translate to selection gains over the control.

Significance and Claims

The paper concludes that perturbation consistency is, at best, a partial diagnostic of visual dependence and, on its own, not a usable selection signal once the decoding format is controlled.

Re-evaluating Claims: Gains reported in literature against CoT-only majority voting likely overstate the effectiveness of perturbation-based methods because they fail to account for the benefits of short-answer aggregation on the original image.
The "Null" Finding: The study provides a measured null result: simply adding perturbation-based re-derivation does not improve selection accuracy beyond what is achieved by aggregating short, no-CoT answers from the original image.
Future Direction: Progress in test-time scaling for VLMs likely requires information sources beyond answer-level consistency, such as denser visual probes, process-level rewards, or cross-model disagreement.
Methodological Standard: The authors argue that any future selection rule utilizing extra inference budget in a different decoding format must include a format-matched control (like MatchedCtrl) to avoid misattributing format effects to grounding mechanisms.

The paper does not claim that perturbation signals are useless; rather, it asserts they are insufficient as a standalone selector without controlling for the confounding variable of decoding format. The stability gap remains useful for auditing whether a model "sees" the image, but not for deciding which answer to pick when a fair baseline is available.