Technical Summary: 3D-DefectBench

Problem Statement

Evaluating generative 3D systems at scale is challenging due to the open-ended nature of both text prompts and textured mesh outputs. Exhaustive human review is costly and slow, motivating the use of Vision-Language Models (VLMs) as automated judges. However, the reliability of these judges is not solely a function of the underlying model's capability. It is fundamentally a measurement problem where agreement with human labels depends on the entire evaluation pipeline: how the 3D asset is rendered, which visual channels (e.g., RGB, depth, normals) are provided, how the task is specified via prompts, and how human reference labels are constructed. Existing benchmarks often treat the judge as a standalone model or rely on holistic ratings that lack actionable diagnostic value for specific failure modes. Furthermore, comparing VLMs against a single human annotation assumes that annotation is a reliable target, ignoring variability in annotator expertise and label noise.

Methodology

The authors introduce 3D-DefectBench, a large-scale benchmark and methodology designed to treat VLM judging as a configurable evaluation pipeline rather than a static model benchmark.

Benchmark Design

• Data: The benchmark consists of 1,049 "silver" assets (labeled by trained vendor annotators) and 129 "expert" assets (labeled by two expert 3D artists). Assets are generated by two variants of Roblox's Cube text-to-3D system.
• Task & Taxonomy: Instead of holistic scores, the task involves predicting a nine-dimensional binary defect vector. The taxonomy includes five geometry defects (form/surface quality, fused/incomplete parts, pose/placement, missing parts, extra geometry) and four texture defects (noise/blur, misplaced/overlapping texture, baked lighting artifacts, visual-textual mismatch).
• Human Reference Regimes: To separate judge limitations from reference label noise, the study utilizes two complementary reference sets: a large silver set (majority vote of trained vendors) and a smaller expert set (independent reviews by domain experts).

Experimental Framework

The study employs a balanced factorial design to systematically vary four pipeline components across 84 inference configurations:

1. VLM Model: A diverse set of models ranging from cost-effective to frontier-scale (e.g., Gemini, GPT, Qwen, Claude, Mistral).
2. Camera Protocol: Three view layouts (6-view oblique, 14-view equatorial, 14-view multi-ring).
3. Visual Input: Seven channel combinations (RGB only, RGB+geometry, RGB+depth, RGB+normals, depth-only, geometry-only, normal-only).
4. Prompt Schema: Four levels of guidance (Compact Binary, Definition-Guided, Rubric-Guided Binary, Rubric-Guided Checklist).

Analysis Strategy

The authors utilize a cost-aware staged methodology:

1. Screening: All 84 configurations are evaluated using five representative VLMs on the 1,049 silver assets.
2. Validation: A subset of configurations is re-evaluated on a broader set of frontier models to ensure design conclusions are not artifacts of the screening models.
3. Fixed-Pipeline Comparison: A single near-optimal configuration is selected to compare 12 VLMs against both expert and silver human references.
4. Statistical Modeling: A defect-level logistic factor model is used to estimate the main effects and interactions of pipeline factors on agreement with human labels.

Key Results

1. Pipeline Sensitivity and Factor Importance

• Model Dominance: The choice of VLM is the dominant source of variation in agreement with human labels (explaining the most variance in the logistic model).
• Pipeline Impact: While the model is primary, pipeline factors are non-negligible. Visual input and prompt schema significantly affect agreement, and these factors interact with the specific model used.
• Optimal Configuration: Within the tested space, a compact six-view RGB protocol combined with a rubric-guided checklist prompt (Configuration c004) performs comparably to denser view sets or configurations augmented with depth/normal channels. Adding geometric channels (depth/normals) does not improve performance over RGB alone; in fact, stripping color (RGB) significantly hurts agreement, likely because human reference labels were derived from RGB inspection.

2. Model Performance and Calibration

• Leaderboard: Under the fixed c004 design, the Gemini family (specifically Gemini 3.1 Pro and Flash-Lite) leads in agreement with human labels.
• Human Gap: Even the best VLMs trail trained human labelers. In a "leave-one-labeler-out" protocol, the best VLMs achieve lower macro Matthews Correlation Coefficient (MCC) than a held-out human expert or crowd labeler.
• Reference Sensitivity: Agreement scores drop sharply when moving from expert labels to noisier silver labels, particularly for texture defects (MCC drops from ~0.40 to ~0.16). Texture rankings are unstable across reference regimes, whereas geometry rankings are more robust.
• Defect Characteristics: VLMs tend to overcall "missing parts" and "baked lighting" defects but undercall "misplaced/overlapping texture."

3. Downstream Impact

• Generator Comparison: The choice of judge significantly impacts conclusions about which generator is superior. Stronger judges generally agree with human consensus on prevalent defects, while weaker judges often produce spurious or contradictory rankings.
• Baseline Performance: Simple baselines (e.g., CLIP zero-shot, prompt-only) perform near zero, confirming that the signal comes from the multi-view rendered evidence combined with the VLM's reasoning capabilities.

Significance and Claims

The paper claims that automated evaluation should be viewed as a measurement system rather than a simple model benchmark. Its primary contributions are:

1. Methodological Shift: It demonstrates that the reliability of an automated judge is a function of the entire pipeline (rendering, input, prompt, reference). Optimizing only the model is insufficient; the "best" configuration is context-dependent and interacts with the model choice.
2. Granular Diagnostics: By moving from holistic ratings to fine-grained binary defect vectors, the benchmark provides actionable diagnostics for generator development and reveals specific categories where judges succeed or fail.
3. Calibration against Human Variability: The study highlights that "ground truth" in 3D generation is noisy and dependent on annotator expertise. Automated judges must be calibrated across different human reference regimes (expert vs. crowd) to understand their true utility and limitations.
4. Cost-Effective Defaults: The research identifies a specific, low-cost pipeline configuration (6-view RGB + rubric checklist) that offers a strong cost-performance trade-off, making it a viable default for production evaluation without the need for expensive depth/normal rendering or excessive view counts.

The authors conclude that while VLMs are competitive with trained humans, they do not yet surpass them, and their performance is inextricably linked to how the evaluation task is constructed and how human references are defined. The work provides a general framework for studying automated judges in GenAI beyond 3D generation.