🤖 AI

Reward Design for Physical Reasoning in Vision-Language Models

This paper presents a systematic ablation study demonstrating that reward design in GRPO-based training for Vision-Language Models induces domain-specific reasoning behaviors, where accuracy-based rewards yield the strongest overall gains while novel attention-weight rewards significantly enhance spatial reasoning without requiring spatial annotations.

Original authors: Derek Lilienthal, Manisha Mukherjee, Sameera Horawalavithana

Published 2026-04-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Derek Lilienthal, Manisha Mukherjee, Sameera Horawalavithana

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, but slightly naive, robot assistant to solve physics problems based on pictures. You show it a photo of a rollercoaster and ask, "How fast is the car going?"

The robot needs to do three things at once:

See the picture (perception).
Know the laws of physics (knowledge).
Think through the math step-by-step (reasoning).

Even the smartest robots today struggle with this. They often guess the right answer by accident or memorize patterns without actually understanding why.

This paper is about a new way to teach these robots how to think better. Instead of just showing them the right answer (like a teacher grading a test), the researchers tried different ways of giving "points" (rewards) to see what kind of thinking behavior the robot would learn.

Here is the breakdown of their experiment using simple analogies:

The Four "Scorecards" (Reward Designs)

The researchers tested four different ways to give the robot points during its training:

The "Neatness" Score (Format Reward):
- The Analogy: Imagine a teacher who only cares if your homework is written in the right boxes. Did you write the answer in the "Answer" box? Did you write the unit in the "Unit" box?
- What it teaches: It teaches the robot to follow rules and structure, but it doesn't care if the answer is actually right.
The "Right Answer" Score (Accuracy Reward):
- The Analogy: This is the classic teacher. "Did you get the number right? Yes? Here are 10 points."
- What it teaches: The robot learns to guess the correct number, but it might cheat. It might memorize the answer without understanding the physics, or it might get the right number with a completely wrong explanation.
The "Rubric" Score (Composite Reward):
- The Analogy: This is a strict professor who grades on a detailed checklist. "You got the right number (+1), you identified the correct law of physics (+1), and you used the right units (+1)."
- What it teaches: It tries to force the robot to be perfect in every way.
- The Catch: The researchers found that for a small robot brain (2 billion parameters), this checklist was too confusing. Trying to maximize all these points at once made the robot's learning unstable. It got the "principles" right but started getting the actual numbers wrong.
The "Eye-Tracking" Score (Attention Reward):
- The Analogy: This is the most unique one. Imagine a teacher who doesn't just look at your answer, but watches where your eyes are looking while you solve the problem.
- How it works: The robot has an internal "gaze." If the robot is looking at the rollercoaster car in the photo while calculating speed, it gets points. If it's staring at the empty sky or the background, it gets no points.
- What it teaches: It forces the robot to actually look at the relevant parts of the image.

What Did They Discover?

The results were surprising and taught us a lot about how to train AI:

Simple is often better: The "Right Answer" score (Accuracy) was the most reliable way to get the robot to solve problems correctly overall.
The "Rubric" trap: Giving the robot a complex checklist (Principles + Units + Answer) didn't make it smarter overall. It actually made it confused because the small robot brain couldn't balance all those goals at once. It learned to write good essays about physics but failed the math test.
The "Eye-Tracking" magic: The "Eye-Tracking" score was a game-changer for spatial problems (like "Is the ball to the left of the tree?"). It improved the robot's ability to understand space from 27% to 50%.
- However, it had a downside. When the robot focused too hard on looking at the picture, it got worse at problems that required pure math formulas (like Thermodynamics). It was like a student staring so hard at the diagram they forgot to do the algebra.

The Big Takeaway

You can't just use one "magic bullet" to teach an AI. The way you reward the AI changes how it thinks:

If you want raw accuracy, reward the correct answer.
If you want good reasoning chains, reward the steps (the rubric), even if the final answer is slightly off.
If you want the AI to understand the image, reward where it looks (attention).

The paper concludes that designing the "reward" is just as important as the training algorithm itself. It's not just about making the robot smarter; it's about deciding what kind of smart you want it to be.

, , , ` tags).
2. R2: Accuracy Reward ( $r_a$ ): Supervises answer correctness.
* MCQ: Exact match with ground truth.
* OE: Evaluated by a 3-judge LLM jury (0/1/2 scale) to allow partial credit.
3. R3: Rubric Reward ( $r_{rub}$ ): A composite reward combining accuracy with physics-grounded components:
* Principle Identification: Checking if the correct physics law is named.
* Unit Consistency: Verifying the physical unit.
* Reasoning Quality: Validating the step-by-step derivation (for OE).
* Implementation: Uses rule-based matching for MCQ and a structured LLM judge for OE.
4. R4: Attention-Weight Reward ( $r_{attn}$ ): A novel internal reward derived from the model's own attention weights over input image tokens.
* Mechanism: It reconstructs the attention matrix from the final transformer layer. It calculates a "Foreground Grounding Score" by measuring the fraction of attention mass on non-white (foreground) pixels versus background.
* Advantage: Requires no external spatial annotations (bounding boxes) or additional models. It supervises where the model looks, not just what it says.

Experimental Conditions

The study evaluates seven conditions:

Baseline: No fine-tuning.
SFT: Supervised Fine-Tuning only.
GRPO (Fmt): Format reward only.
GRPO (Fmt+Acc): Format + Accuracy.
GRPO (Rubric): Composite rubric reward.
GRPO (ASM): Attention Score Mask (Attention-based) only.
GRPO (Fmt+Acc+ASM): Combined Format, Accuracy, and Attention.

3. Key Contributions

Systematic Reward Ablation: The first controlled study isolating the effects of format, accuracy, rubric, and internal attention rewards on VLM physical reasoning under GRPO.
Novel Internal Reward: Introduction of an attention-weight-based reward ( $r_{attn}$ ) that enforces visual grounding without requiring expensive spatial annotations.
Empirical Insights: Discovery that reward design does not uniformly improve performance but rather induces domain-specific reasoning behaviors.

4. Results

Overall Performance (MCQ vs. OE)

MCQ (Multiple Choice):
- Best Model: GRPO (Fmt + Acc + ASM) achieved the highest overall accuracy (0.462), outperforming SFT (0.433) by ~6.7%.
- Domain Specificity: The Attention-based reward (ASM) significantly boosted Spatial Reasoning (accuracy jumped from 0.27 to 0.50) but degraded performance in symbolic domains like Thermodynamics and Wave/Acoustics.
- Rubric Performance: GRPO (Rubric) showed mixed results, outperforming SFT in Thermodynamics and Wave/Acoustics but underperforming in others.
OE (Open-Ended):
- Best Model: GRPO (Fmt + Acc) achieved the highest overall score (0.027), representing a 100% improvement over SFT.
- Rubric Trade-off: While GRPO (Rubric) improved reasoning quality (coherent chains), it resulted in lower absolute accuracy compared to Fmt+Acc.

Key Findings on Reward Dynamics

Attention vs. Symbolic Capacity: The Attention-based reward (ASM) successfully forces the model to focus on relevant image regions (lower attention entropy, higher foreground score). However, in a 2B parameter model, this visual grounding appears to compete with the representational capacity needed for symbolic reasoning (chaining formulas), leading to performance drops in non-visual physics domains.
Rubric Complexity vs. Stability: The composite Rubric reward did not consistently outperform simpler accuracy-based rewards. The authors attribute this to optimization instability in small models; the high variance from combining multiple objectives (accuracy, units, principles) destabilizes the GRPO gradient updates, causing the model to optimize secondary objectives at the expense of primary accuracy.
Reasoning Integrity: Models trained with Fmt+Acc (no explicit reasoning reward) tended to generate answers that were correct but unsupported by valid reasoning chains as training progressed. In contrast, the Rubric model maintained coherent reasoning chains even if its final accuracy was slightly lower.

5. Significance and Conclusion

Reward Design as a Control Mechanism: The paper demonstrates that reward design is not just an optimization tool but a mechanism to control the nature of reasoning.
- Use Accuracy Rewards for raw performance.
- Use Rubric Rewards for structured, interpretable reasoning chains (even if accuracy dips).
- Use Attention Rewards for tasks requiring spatial grounding, but be cautious of their negative impact on symbolic domains in smaller models.
Scalability Warning: The trade-offs observed (e.g., visual grounding vs. symbolic capacity) may be specific to the 2B parameter scale. Larger models might handle multi-objective rewards and mixed reasoning types more effectively.
Practical Implication: There is no "one-size-fits-all" reward. Practitioners must align the reward signal with the specific deployment goal (e.g., raw accuracy vs. explainable physics reasoning vs. visual grounding).

In summary, the study provides a critical roadmap for training VLMs in scientific domains, highlighting that how a model is rewarded is as crucial as the training algorithm itself, and that internal attention mechanisms offer a powerful, annotation-free path to improving visual reasoning.