💬 NLP

Test-Time Scaling for Small VLMs on Multilingual Visual MCQ

This paper demonstrates that for small multilingual vision-language models, test-time scaling gains are primarily driven by ensuring prompt parseability and sufficient decoding budgets rather than complex search or verification mechanisms, with the underlying policy model being the most significant factor for achieving state-of-the-art performance on visual multiple-choice benchmarks.

Original authors: Spiros Baxevanakis, Peng-Jian Yang

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Spiros Baxevanakis, Peng-Jian Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to solve a tricky, multi-language puzzle where the clues are hidden inside pictures. You have a team of small, smart robots (called "Vision-Language Models") to help you. For a long time, people thought the secret to solving these puzzles was to build a super-complex machine that could check every single step of the robot's thinking, or to have the robot rewrite its own answers until they were perfect.

But this paper, written by researchers from the University of Amsterdam for the ImageCLEF 2026 contest, says: "Hold your horses! The secret isn't a fancy machine; it's just giving the robots more time to finish their sentences."

The Big Discovery: More Words, Less Thinking

The researchers tested a bunch of different strategies on a massive set of exam questions (11 different languages, 20 subjects) to see what actually helps these small robots get better at reasoning. They found that the most important thing wasn't how clever the search method was, but how long the robot was allowed to talk before it had to stop.

Think of it like a student taking a test.

The Old Way: You tell the student, "Think really hard, check your work, and write a perfect essay," but you only give them 1,000 words to do it. They might figure out the right answer in their head, but they run out of space before they can write down the final letter (A, B, C, D, or E). They get the reasoning right but fail the test because they never committed to an answer.
The New Way: You tell the student, "Just write down your thoughts, but you can use up to 2,048 words." Suddenly, they have enough room to finish their story and actually write the answer letter.

The paper measured this carefully. When they doubled the word limit from 1,000 to 2,048 tokens, the accuracy jumped by 3.7 percentage points. That's a huge win! But when they tried to get more answers by asking the robot to generate 16 different chains of thought instead of 8 (doubling the number of attempts), the score only went up by a tiny 0.15 percentage points. It's like asking a student to write the same essay 16 times instead of once; it doesn't help much if they were already stuck on the word count.

What Didn't Work (The "Cool" Ideas That Failed)

The researchers tried some very sophisticated tricks that sound like they should work, but the data showed they didn't.

The "Step-by-Step" Coach: They tried using a special "Process Reward Model" (PRM) to act like a coach, checking every single step of the robot's reasoning and guiding it like a beam search. They thought this would be better than just letting the robot guess freely. Result: It actually performed worse than the simple method, dropping the score by 0.39 percentage points while costing 8.7 times more computing power. It turns out, the coach got confused by the pictures and couldn't steer the robot back on track.
The "Judge" Robot: They tried adding a second robot to act as a judge, reading all the answers and picking the best one. They tried two types: one that was trained to be a critic and one that was just a smart generative model. Result: Neither beat the simple method of just taking a "majority vote" (picking the answer that appeared most often). In fact, on the smarter robot, the judges actually made things slightly worse. The paper suggests this is because the judges couldn't tell the difference between a right answer and a wrong one when the answers were close.
Self-Correction: They looked at whether letting the robot fix its own mistakes helped. Result: No. At this scale, letting the robot rewrite its own answers often made it worse.

The Real Hero: The Robot Itself

The biggest boost didn't come from a new strategy at all; it came from swapping the robot. When they switched from an older model (Qwen2.5-VL-7B) to a newer, smaller model (Qwen3.5-4B), the score jumped by 11.4 percentage points. This suggests that having a smarter "brain" (the policy model) matters way more than having a fancy "thinking process" (the search method).

The "Fix-It" Trick

There was one small glitch: sometimes, even with enough words, the robot would write a great explanation but forget to write the final letter (A, B, C, etc.). The researchers found a simple "guided repair" trick: if the robot didn't write a letter, they just forced it to say "Answer: [A/B/C/D/E]" for one single token. This fixed the problem for the remaining 2.67% of questions that were failing, effectively closing the gap.

The Final Score

By combining the newer robot (Qwen3.5-4B), giving it a generous word limit (2,048 tokens), asking for 16 different attempts, and using the simple "fix-it" trick, the team reached 84.1% accuracy on the final test. This put them in first place on the leaderboard.

The Takeaway

The main lesson here is that for small vision-language models, engineering details matter more than complex algorithms. If you give the model enough room to finish its thought (the token budget) and a prompt that forces it to commit to an answer, you get better results than if you try to build a complex system to check its work. The paper suggests that until we have better models, the best strategy is simply to let the model talk longer and fix its formatting, rather than trying to outsmart it with complicated search trees or judges.

Technical Summary: Nika at ImageCLEF 2026 Task on Multimodal Reasoning

1. Problem Statement

The paper addresses the challenge of applying Test-Time Scaling (TTS) to small, open-weight Vision-Language Models (VLMs) (≤7B parameters) on multilingual visual multiple-choice questions (MCQs). While TTS has proven effective for large text-only language models, its efficacy in open VLMs under strict computational constraints (single A40 GPU) remains unclear. The authors investigate whether advanced inference techniques—such as structured search, process reward models (PRMs), and generative critics—can outperform simpler parallel sampling strategies on the EXAMS-V benchmark, which spans 13 languages and 20 school subjects.

2. Methodology

The authors evaluated two open-weight policies: Qwen2.5-VL-7B-Instruct and the newer, smaller Qwen3.5-4B. They compared two primary pipelines across two scaling axes (parallel chain count and per-chain token budget):

Flat Self-Consistency (SC): The model generates $N$ independent reasoning chains conditioned on a Chain-of-Thought (CoT) prompt ending with a standardized answer cue (MMMU closer). The final answer is determined by majority vote, with log-probability tie-breaking.
Structured Search (Describe-Then-Reason with PRM): A pipeline where the model first generates image-grounded descriptions, followed by step-by-step reasoning guided by a PRM-guided beam search (PRM-BAS). The PRM (Qwen-VL-PRM-7B) scores reasoning steps while retaining access to the original image.
Selectors: Post-hoc selection methods were tested against majority vote, including a training-free generative critic and a discriminative PRM (one-shot scoring).
Engineering Interventions:
- Prompt Format: Switching to a standard answer cue to reduce parse failures.
- Guided Repair: A mechanism to force an answer letter generation via guided_choice constraints when chains fail to commit to an answer.
- Scaling: Systematic variation of chain count ( $N \in \{8, 16\}$ ) and token budget (1k vs. 2k tokens).

All experiments utilized vLLM for batched parallel decoding on a single A100-40 GB GPU, adhering to the ImageCLEF 2026 shared task constraints.

3. Key Findings and Results

The Dominance of Engineering and Policy over Search

The most significant finding is that engineering details and policy choice dominate search strategies.

Parseability: An early prompt format caused ~9% of reasoning chains to fail to produce an answer letter. Switching to a standard cue and implementing a guided repair step reduced parse failures to near zero, recovering significant accuracy.
Token Budget vs. Chain Count: Increasing the per-chain token budget from 1,000 to 2,048 tokens yielded a +3.7 percentage point (pp) gain. In contrast, doubling the number of chains from 8 to 16 added only +0.15 pp. The authors conclude that ensuring chains have sufficient budget to finish reasoning is far more critical than sampling more chains.
Policy Upgrade: Switching from Qwen2.5-VL-7B to Qwen3.5-4B resulted in a +11.4 pp gain, the largest single factor in the study.

Ineffectiveness of Complex Search and Selectors

Structured Search: The PRM-guided beam search (PRM-BAS) underperformed flat self-consistency by -0.39 pp while costing 8.7 times more compute. The PRM's step-level scoring saturated quickly, leading to low diversity in beam search (71.9% of questions had all beams agreeing on the same letter), rendering the search ineffective.
Selectors: Neither the training-free generative critic nor the discriminative PRM outperformed simple majority vote. On high-performing pools (Q3.5), both selectors yielded negative or null results, often correcting fewer errors than they introduced.

Stratified Performance

Languages: Accuracy gains from the token budget increase were concentrated in English (+17.3 pp) and Chinese (+8.5 pp). Languages with high chain agreement (e.g., Chinese, where 22.3% of questions had unanimous 8/8 agreement) saw SC regress or stagnate because the majority vote amplified shared errors rather than correcting independent ones.
Subjects: STEM subjects benefited most from TTS, while text-heavy subjects (e.g., Islamic Studies) showed little improvement, suggesting a bottleneck in domain knowledge rather than reasoning diversity.

4. Final Performance

The best configuration—Qwen3.5-4B with Self-Consistency ( $N=16$ ), a 2,048-token budget, and guided repair—achieved:

81.64% on the full EXAMS-V validation set.
84.06% on the held-out ImageCLEF 2026 test split.
First place on the Visual MCQ leaderboard, ranking first across all six test languages.

5. Significance and Claims

The paper claims that for small open VLMs under constrained compute budgets:

TTS is effective but specific: The benefits of TTS come primarily from allowing chains to complete (increasing token budget) and improving extraction reliability (parseability), not from complex search algorithms or external selectors.
Policy Substrate is Key: Upgrading the base model yields larger gains than any inference-time strategy.
Limits of PRMs: Current PRMs trained on math or specific domains may not generalize well to multimodal, multilingual reasoning, particularly when they fail to distinguish between correct and incorrect reasoning steps with sufficient signal to guide search.
Engineering over Algorithm: At the ≤7B scale, "engineering substrate" (prompt formats, decoding budgets) and model choice are more impactful than sophisticated search or verification machinery.

The authors conclude that future work should focus on decoupling perception from reasoning and training better-calibrated PRMs on multilingual, non-mathematical content, rather than relying on current search-based scaling methods.