🤖 machine learning

Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation

This paper introduces a de-biased, optimization-grade VLM-as-3D-judge protocol that rigorously hardens evaluation against failure modes to test whether lightweight parameter-efficient adaptation can improve a strong open-source 3D generator, ultimately finding that while the protocol is reusable and informative, current cheap adaptation methods on public data only achieve parity with the base model rather than exceeding it.

Original authors: Ali Asaria, Tony Salomone, Deep Gandhi

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Asaria, Tony Salomone, Deep Gandhi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a talented artist (a computer program called TRELLIS) to draw better 3D furniture. The artist is already very good, but you want them to get even better at making chairs and tables, and you want to do it without hiring a human teacher to grade every drawing.

To solve this, the researchers built a Robot Judge (a Vision-Language Model) to act as the teacher. This paper is the story of how they tried to use this Robot Judge to train the artist, and the surprising things they learned along the way.

Here is the breakdown of their journey, using simple analogies:

1. The Problem: The "Robot Teacher" is Tricky

In a previous study, the team proved that a Robot Judge is better at grading 3D furniture than simple math formulas or basic image checkers. But there was a big catch: You can't just ask the Robot to grade the student and then use that same Robot to teach the student.

If you do that, the student might just learn how to "game" the Robot. They might learn to draw things the Robot likes, even if the drawing is actually bad. It's like a student memorizing the answer key to a specific test rather than actually learning the subject.

2. The Solution: The "Double-Blind" Protocol

To fix this, the researchers built a super-strict training system with three main rules:

Two Different Judges: They used one Robot (let's call him Qwen) to grade the student during training, and a completely different Robot (Intern) to grade the final results. This ensures the student isn't just memorizing Qwen's quirks.
The "Swap" Test: Robots sometimes get confused by the order of things. If you show them Image A then Image B, they might pick A. If you show B then A, they might pick B. To fix this, the researchers showed the Robot the images in both orders. If the Robot changed its mind just because the order switched, they threw that grade out. They only kept grades where the Robot was consistent.
The "Normal Map" Trick: Sometimes, a 3D model looks smooth and pretty from the outside (like a glossy photo), but it has holes or broken parts inside. The Robot was getting fooled by these "pretty but broken" models. The researchers fixed this by forcing the Robot to look at a "blueprint view" (a normal map montage) that reveals the structural flaws, making it impossible to hide broken geometry.

3. The Big Discovery: The Signal Was Missing

Before they started training, they realized something important: You can't teach a robot to prefer one thing over another if the two things are basically the same.

They tried to have the Robot judge two random drawings made by the same artist. The Robot couldn't tell the difference (it flipped its choice 94% of the time). It was like asking a wine expert to tell the difference between two glasses of the exact same bottle of wine. There was no "signal" to learn from.

So, they engineered the signal: They showed the Robot a perfect drawing (made with lots of computing power) and a bad drawing (made quickly). The Robot could easily tell the difference. They used this "Good vs. Bad" gap to teach the artist how to make the "Bad" ones look more like the "Good" ones.

4. The Results: The Artist Hit a Wall

They tried six different ways to teach the artist using this strict Robot Judge system. They tested on clean pictures and on pictures that were blurry, cropped, or damaged.

The outcome was surprising:

On Clean Pictures: The artist was already so good that the Robot couldn't tell the difference between the original and the "trained" version. The artist had already reached the ceiling.
On Damaged Pictures: The researchers tried to fix the artist's ability to handle bad input.
- Most methods failed completely. The artist didn't get better.
- One specific method (fixing the "input conditioner") worked, but only up to a point. It managed to bring the artist's performance up to the level of the original artist (a "tie"), but it could not make the artist better than the original.

5. Why Did They Only Tie?

The researchers found a mechanical reason for this "tie":

The "Wash-Out" Effect: When they tried to tweak the artist's internal brain (the "flow-DIT" part), the changes were so small that by the time the final 3D chair was generated, the changes had washed away. It was like trying to steer a massive ship by pushing the rudder with your finger; the ship doesn't move.
The Bottleneck: The only place that actually moved the needle was fixing the "eyes" (the input conditioner). When the input picture was terrible, the artist's eyes were confused. Fixing the eyes helped, but only enough to catch up to the original artist, not to surpass them.

The Bottom Line

The most important thing the paper produced isn't a new, super-advanced 3D model. It's the "Protocol" (the strict rules for how to use the Robot Judge).

They proved that:

You need a very strict, double-blind system to use AI judges for training.
If you only use public data and cheap training methods, you can match a strong existing AI, but you probably can't beat it. To truly exceed the best, you likely need more data or more expensive training.

In short: They built a better ruler to measure 3D art, used it to try to improve an artist, and found that while the artist could be fixed to be as good as before, the "cheap" tools weren't enough to make them a genius.

Technical Summary: Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation

Problem Statement

The paper addresses the challenge of specializing a strong, open-source single-image-to-3D generator (TRELLIS) for a specific asset class (furniture) without human labels. While a companion study established that label-free 3D specialization requires a de-biased Vision-Language Model (VLM) judge rather than cheap geometry or CLIP proxies, this work investigates whether such a judge's preferences can be used as a training signal to optimize the generator. The core difficulty lies in avoiding circularity: optimizing a generator against a specific judge and then evaluating it with the same judge often leads to the model learning the judge's quirks rather than genuine quality improvements. Furthermore, the authors note that independent samples from a strong base model often carry no learnable preference signal, making the construction of a valid training objective non-trivial.

Methodology

The research is structured around two primary components: a hardened evaluation protocol and a specialization study.

1. The Optimization-Grade De-biased VLM-as-3D-Judge Protocol
To ensure trustworthy results in an optimization loop, the authors extend the companion ranking protocol with three critical hardening measures:

Cross-Model Independence: The training judge (Qwen2.5-VL-7B) and the evaluation judge (InternVL3-8B) are strictly distinct models. This prevents the generator from overfitting to the specific biases of a single model family.
Position-Bias Correction: For every pairwise comparison, the judge is queried in both orders (A vs. B and B vs. A). Verdicts are only retained if they are consistent across the swap; inconsistent verdicts are discarded as position-biased.
Failure Mode Mitigation: The protocol addresses three specific failure modes exposed only when pushing a VLM into an optimization loop:
- Image Overload: Showing multiple views (e.g., 7 images) causes the judge to answer based on position rather than quality. Fix: Use a strict two-image single-comparison configuration.
- Splat-Hidden Geometry: Gaussian-splat renders can mask geometric defects. Fix: Render mesh normal-map views as a 2x2 montage to make holes and missing parts unmistakable.
- Reference-Free "Clean-but-Wrong": Without a reference, judges may prefer clean but incorrect outputs. Fix: Rely on clear-quality-gap calibration and montage coverage to ensure the judge tracks real quality.

2. Specialization Study
Using the hardened protocol as an independent evaluator, the authors tested six adaptation methods on TRELLIS using only public data and lightweight parameter-efficient fine-tuning (PEFT):

Methods: SFT-on-best, DPO (at $\beta=0.1$ and $\beta=0.5$ ), ORPO, SFT-on-clean, and a Conditioner-repair adapter (training a residual adapter on DINOv2 features to map degraded inputs to clean features).
Regimes: "Clean" (in-distribution 3D-Furniture renders) and "Hard-degraded" (synthetic crop, occlusion, downscale, and blur).
Signal Engineering: Since independent samples from the base showed a 0.94 order-flip rate (no learnable signal), the authors engineered a preference signal by pairing high-budget samples (25 steps) against deliberately degraded low-quality samples (2 steps).

Key Results

The study found that lightweight adaptation on public data matches the strong base but does not exceed it.

Preference Signal: Independent samples from the base model carry no learnable preference (0.94 flip rate), whereas contrastive construction (high-quality vs. degraded) yields a strong signal (0.89 win-rate for the high-budget sample).
Performance: Across six methods, two input regimes, and a severity sweep, no method achieved the target win-rate of $\geq 0.65$ $\geq 0.65$ against the base.
- Clean Regime: All methods resulted in a 0.00 win-rate. The base model is already saturated on in-distribution data.
- Hard Regime: Most methods (SFT, DPO, ORPO) also resulted in 0.00 win-rates.
- Conditioner Repair: This was the only method showing a monotonic trend with degradation severity. It reached a win-rate of 0.125 (mild), 0.25 (medium), and 0.50 (severe), achieving parity with the base under severe degradation but failing to surpass it.
Mechanistic Analysis:
- Saturation: Clean inputs are already at the limit of the judge's discernment for the base model.
- Sampler Wash-out: Fine-tuning the flow-DIT (latent transformer) converges mathematically but the updates "wash out" through the rectified-flow sampler, resulting in zero geometric change ( $\Delta = 0$ ).
- Conditioner Bottleneck: The image conditioning (DINOv2 features) is the locus where quality is bottlenecked for degraded inputs. Repairing these features moves the geometry, but only enough to reach parity, not to exceed the base.

Significance and Claims

The paper's primary contribution is protocol-first, not model-first. The authors claim that the "optimization-grade de-biased VLM-as-3D-judge protocol" is the reusable artifact. It provides a reliable, human-free evaluation framework that exposes specific failure modes (image overload, splat-hidden geometry) and ensures cross-model generalization.

Regarding the specialization results, the paper makes a modest claim: matching a strong public-data base with cheap in-domain adaptation is itself an informative result. The finding that lightweight PEFT on public data reaches parity but cannot exceed the base suggests that:

The base model has already captured much of the publicly available furniture distribution.
Exceeding the base likely requires more than lightweight adaptation on public data (e.g., licensed data, larger adapters, or full fine-tuning).
The "gap-sensitivity" that makes the judge reliable also reveals that there is no "cheap room" for improvement on the base model given the current constraints.

The work closes the loop on the companion study: while a de-biased judge is necessary to evaluate 3D generation, the same sensitivity that makes the judge reliable also indicates that strong open bases are difficult to improve upon without significant resources or data beyond the public domain.