A Cross-Model VLM-Judge Protocol for Single-Image 3D Mesh Quality (and Why Cheap Proxies Fall Short)

Technical Summary: A Cross-Model VLM-Judge Protocol for Single-Image 3D Mesh Quality

Problem Statement
Single-image-to-3D generators are rapidly improving, yet the field lacks an agreed-upon, human-free method to determine whether one generated mesh is superior to another. Practitioners currently rely on "cheap" automatic proxies, specifically render-space CLIP similarity and mesh geometry-validity statistics (e.g., watertightness, manifoldness, normal consistency). However, it remains unestablished how well these proxies track perceived mesh quality—the metric that matters for downstream users, rankers, and training signals. The core problem is determining if these low-cost metrics can serve as reliable substitutes for a high-fidelity quality assessment.

Methodology
The authors propose and validate a reproducible evaluation protocol centered on a Vision-Language Model (VLM) judge, designed to serve as a reference oracle.

• Evaluation Protocol:

  • Input: A fixed 24-view headless render rig is applied to normalized meshes generated from single images.
  • Judges: Two independent open VLM families are employed: an "oracle" judge (Qwen2.5-VL-7B-Instruct) and a separate validation judge (InternVL3-8B). Using two distinct models allows for cross-model agreement checks rather than relying on a single model's bias.
  • Position-Bias Correction: Recognizing that VLMs exhibit presentation-order bias, the protocol queries each pair in both orders (A vs. B and B vs. A). Verdicts are retained only if they are consistent across the swap; inconsistent verdicts are discarded. This correction is critical, as raw verdicts showed ~26% inconsistency.
  • Proxies Tested: The protocol evaluates four proxy strategies against the independent validation judge:
    1. Geometry-only score (based on watertightness, manifoldness, non-self-intersection, normal consistency).
    2. Render-CLIP similarity score.
    3. A fixed-weight composite of the above.
    4. A learned Bradley–Terry head (logistic regression) trained on the oracle judge's labels to weight the five features.

• Experimental Setup:

  • Data: 60 objects from Google Scanned Objects.
  • Generators: Two feed-forward generators (Stable Fast 3D and TripoSR) plus a "face-drop" degraded variant of each, creating a spectrum of quality.
  • Splits: Strict object-level splits ensure no object straddles train/test sets. The learned head is trained on a subset and evaluated on held-out pairs against the validation judge.
  • Statistics: Agreement is measured using proportions with Wilson 95% confidence intervals and Cohen's $\kappa$ against a chance-agreement baseline (0.5). Cluster bootstrapping is used to account for object-level clustering.

Key Contributions

1. Validated VLM-Judge Protocol: The authors establish a reproducible, human-free evaluation framework featuring a fixed render rig, dual independent judges, and mandatory position-bias correction. The two judge families achieve substantial agreement ($\kappa=0.66$, raw agreement 0.83), significantly above the chance floor.
2. Quantification of Position Bias: The study demonstrates that position bias is a major failure mode, with ~26% of raw verdicts flipping based on presentation order. Correcting this bias is shown to be essential, moving agreement estimates from trivial levels (e.g., 0.333) to substantial ones (0.714).
3. Failure of Cheap Proxies: Using the VLM judge as the ground truth, the paper demonstrates that cheap proxies do not substitute for the judge:
   • Geometry Validity: While significantly above chance (0.62), it remains weak and falls below the pre-registered target ($\approx \ge 0.75$).
   • Render-CLIP: Performs at chance levels (0.48), carrying no usable quality signal.
   • Learned Weights: A learned Bradley–Terry head collapses onto a single manifoldness statistic (assigning a negative weight to render-CLIP), resulting in a ranking identical to the geometry-only score. Learning feature weights provides no additional benefit.
4. Bimodal Proxy Behavior: Subgroup analysis reveals that proxies are bimodal. They perform significantly above chance only on contrasts with visible geometric defects (e.g., cross-generator comparisons or clear holes) but drop to chance levels on ambiguous quality calls. This suggests proxies only track the judge when defects are visually salient.

Results

• Reliability: The cross-model agreement between the two VLM judges is robust (0.83 agreement, $\kappa=0.66$), confirming the protocol's reliability as a reference.
• Proxy Performance:
  • Geometry: 0.62 agreement on the full corpus (95% CI [0.56, 0.68]).
  • Render-CLIP: 0.48 agreement (95% CI [0.42, 0.54]), indistinguishable from chance.
  • Learned Head: Matches geometry-only exactly (0.52 on held-out data), indicating the feature set limits performance rather than the model architecture.
• Subgroup Analysis: Geometry validity reaches 0.91 agreement on cross-generator contrasts and 0.80 on within-TripoSR degradation (where defects are visible holes). However, it drops to 0.53 on cross-generator-mixed pairs (ambiguous cases). Render-CLIP remains weak across all subgroups (0.37–0.70).

Significance and Claims
The paper claims that for the specific conditions tested (two feed-forward generators on Google Scanned Objects with a face-drop degradation regime), the proposed cross-model, position-bias-corrected VLM-judge protocol is a reliable and reproducible human-free evaluator.

Conversely, the authors advise against using geometry validity or render-CLIP proxies as optimization targets or reward signals. These proxies are not only weak signals but are "deceptively good" on visible-defect contrasts while failing on the ambiguous comparisons that are critical for ranking or optimizing single models. The authors suggest that reward-based specialization should be driven directly by the de-biased VLM-judge preferences rather than by proxies that a generator could satisfy (e.g., by maximizing manifoldness) without actually improving perceived quality.

Limitations
The authors note that the VLM judge is treated as the oracle, though they mitigate this by requiring cross-model agreement. The study is limited to a narrow regime (two generators, one dataset, specific degradation type) and does not cover naturalistic failures like thin structures or transparency. Additionally, the "within-Stable Fast 3D" subgroup was confounded because face-dropping on open shells was barely visible, making the judge's own verdicts ambiguous in that specific cell.