💻 computer science

Each Judge Its Own Yardstick: Discovering Per-VLM Taxonomies for Physical Video Evaluation

The paper introduces JudgeFit, an iterative framework that constructs personalized evaluation taxonomies for individual vision-language models to better assess physical consistency in video generation, demonstrating a 32% improvement over global schemas while revealing model-specific blind spots.

Original authors: Yu Cao, Ziquan Liu, Zhensong Zhang, Jiankang Deng, Shaogang Gong, Jifei Song

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yu Cao, Ziquan Liu, Zhensong Zhang, Jiankang Deng, Shaogang Gong, Jifei Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a talent show for video generators. You have 16 different judges (AI models) to help you decide which videos are physically realistic and which ones look weird.

The problem is, every judge sees the world differently.

The Old Way: One Size Fits All (The "Bad" Ruler)

Traditionally, researchers gave every judge the exact same checklist to grade the videos. It was like handing a carpenter, a chef, and a musician the same ruler and asking them to measure a cake.

The carpenter might only look at the height.
The chef might only look at the texture.
The musician might not know what to do with the ruler at all.

In the paper, they found that when they used a "global" checklist (a fixed set of rules like "gravity," "collisions," and "lighting"), most AI judges ignored three-quarters of the list. They just focused on one thing (usually "mechanics") and gave the rest a zero. It was like asking a colorblind person to judge a painting based on a list of 10 different colors; they just pick the one they can see and ignore the rest.

The New Solution: JUDGEFIT (The "Custom Tailor")

The authors created a new system called JUDGEFIT. Instead of forcing every judge to use the same ruler, JUDGEFIT builds a custom ruler for each specific judge based on what that judge is actually good at seeing.

Here is how it works, step-by-step:

Step 1: The "Seed" (Asking the Judge to Talk)

First, they show a small set of videos to an AI judge and ask: "What looks wrong here? Just tell me in your own words."

If the judge says, "The ball floated like a balloon," the system notes "floating" as a rule.
If the judge never mentions "shadows," the system knows this judge is blind to shadows, so it doesn't put "shadows" on their custom checklist.
The Analogy: It's like asking a new employee, "What mistakes do you notice?" and building their job description based only on the mistakes they actually spot.

Step 2: The "Refine" (The Coach and the Player)

Now, the system puts the judge to work using this new custom checklist. But here's the trick:

The Player (The Video AI): Scores the videos based on the custom checklist. It doesn't know what the "right" answer is; it just follows the rules.
The Coach (A separate AI Editor): Looks at the Player's scores and compares them to what humans thought.
- If the Player says a video is perfect, but humans hated it: The Coach says, "You missed something! Let's add a new rule to your checklist."
- If the Player is checking two things that mean the same thing: The Coach says, "You're doing double work. Let's merge these rules."
- If the Player is checking something that doesn't match human opinion: The Coach says, "Stop checking that; it's confusing your score."

This happens in a loop. The Coach tweaks the checklist, the Player tries again, and they keep going until the Player's scores match the humans' opinions as closely as possible.

The Results: Why It Matters

The paper tested this on 16 different AI models (from small open-source ones to massive closed-source ones).

The Win: For every single judge, the custom checklist worked better than the standard "one-size-fits-all" checklist. On average, the scores became 32% more accurate at predicting what humans would think.
The Surprise: They found that even the "smartest" AI judges have blind spots. One might be amazing at spotting gravity errors but terrible at spotting reflection errors. Another might be the opposite. The old system treated them all as "good" or "bad" overall, but JUDGEFIT revealed their specific strengths and weaknesses.

The Bottom Line

The paper argues that we shouldn't force every AI judge to use the same rules. Just like you wouldn't use a fishing rod to fix a car, you shouldn't force an AI to judge videos using rules it can't perceive.

JUDGEFIT is a method that asks each AI, "What can you see?" and then builds a unique, custom grading system for it. This makes the AI a much better, more reliable judge of physical reality in videos.

Technical Summary: Each Judge Its Own Yardstick (JUDGEFIT)

Problem Statement

The evaluation of physical consistency in video generators and world models increasingly relies on Vision-Language Models (VLMs) as automated judges. These models provide reward signals, ranking decisions, and data-filtering criteria. However, current evaluation paradigms typically apply a single, fixed global taxonomy (e.g., a set of four physical domains) uniformly across all VLMs.

This approach assumes that all VLMs share the same axes of physical-evaluation competence and differ only in the magnitude of their scores along those axes. The paper argues this is flawed because VLMs, trained on distinct data and architectures, encode physical phenomena through different internal representations. Consequently, a fixed schema often leads to:

Inactive Dimensions: Scores from different VLMs concentrate on a single axis (e.g., "mechanics") while other dimensions remain nearly inactive.
Loss of Resolution: The schema fails to reveal distinct strengths or blind spots across different model families.
Misalignment: VLMs often disagree substantially with human raters on physical commonsense, exhibiting position bias and scoring instability.

The core problem is that evaluation schemas are treated as properties of the benchmark rather than properties of the evaluator. A VLM should be assessed along dimensions it can reliably perceive and score.

Methodology: JUDGEFIT

The authors propose JUDGEFIT, a two-stage, seed-and-refine pipeline designed to discover a per-VLM evaluation taxonomy. The procedure treats the judge's evaluation schema as an optimizable object, driven by a calibrated diagnosis of what the specific VLM can reliably score.

Stage 1: Seed (Open Annotation & Clustering)

Free-Form Elicitation: The target VLM is prompted to list physics errors it observes in a small set of videos ( $D_{annot}$ ) using free-form natural language, without predefined dimensions.
Clustering: An LLM clusters these pooled descriptions into named, defined categories to form an initial seed taxonomy ( $T_0$ ).
Vocab Alignment: Because $T_0$ is built solely from errors the VLM reports, its dimensions reflect the VLM's own perceptual vocabulary. Concepts the VLM never mentions are absent from the starting taxonomy.

Stage 2: Refine (Diagnosis-Guided Editing)

The seed taxonomy is refined through an iterative loop where an LLM editor revises dimensions based on a diagnostic signal derived from human ratings.

Calibration: A ridge-regularized linear calibrator maps the VLM's per-dimension scores to human physical-commonsense ratings ( $y$ ).
Diagnosis: The calibrator reveals three types of faults:
- Uninformative: Dimensions whose scores do not correlate with human ratings ( $|\rho_i| < \tau$ ).
- Unstable: Dimensions where the coefficient sign flips across cross-validation folds ( $\pi_i < \kappa$ ).
- Redundant: Pairs of dimensions that rank videos almost identically ( $r_{ij} > \gamma$ ).
- Coverage Gaps: Errors humans penalized that the VLM missed, identified by analyzing residuals against the VLM's own free-form annotations.
Editing: An LLM editor proposes local edits (DROP, REDEFINE, MERGE, ADD, SPLIT) to address these faults.
Selection Objective ( $J$ ): A candidate taxonomy is accepted only if it improves a selection objective that rewards ranking quality and specificity (gains over a "halo" baseline that uses only the mean score). This prevents the taxonomy from collapsing into a single "overall badness" score.
Convergence: The loop repeats until no edit improves $J$ . Empirically, convergence occurs within roughly two rounds.

Key Contributions

JUDGEFIT Pipeline: A sample-efficient procedure that discovers per-VLM taxonomies. Unlike scalar-reward optimizers requiring hundreds of iterations, JUDGEFIT converges in ~2 rounds by using a linear calibrator as a diagnostic signal rather than a final predictor.
Per-VLM Benchmark Study: An evaluation of 16 VLMs across eight model families (including Qwen, Gemma, GLM, Reka, Gemini, Seed, Nova, MiMo) using the VideoPhy-2 dataset. The study releases refined taxonomies and held-out evaluations.
Cross-VLM Analysis: A typology of concept convergence and bias-by-discrimination. The authors identify which physics concepts are universally legible across VLMs and which remain model-specific blind spots, revealing that scoring styles (strict vs. lenient) vary independently of alignment accuracy.

Results

Performance Improvement: The refined per-VLM taxonomy outperforms the fixed global-schema baseline (PhyGenBench) on held-out videos for every VLM tested.
- Mean Relative Improvement: Approximately 32% (Spearman correlation increased from 0.239 to 0.315).
- Consistency: Improvements hold across the full capability spectrum, from the strongest judges (e.g., Gemini-3-Flash-Thinking) to the weakest (e.g., GLM-4.6V).
Convergence: Most performance gains occur in the first refinement round. The procedure is highly efficient compared to iterative scalar-reward search methods.
Generalization: The improvements transfer from the annotation set to the disjoint test set, indicating the taxonomies do not overfit the 200-video training set.
Model-Specific Profiles:
- Concept Coverage: While a core set of concepts (momentum, trajectory, rigidity) is near-universal, fluid mechanics concepts (buoyancy, surface tension) form a sparsely covered long tail.
- Thinking Modes: The benefit of "thinking" variants is model-specific rather than systematic; some models improve, while others degrade or remain unchanged.
- Bias vs. Discrimination: Judges split into "strict" and "lenient" categories based on their scale placement, but this bias does not correlate with their alignment accuracy (Spearman $\rho$ ).

Significance and Claims

The paper claims that evaluation schemas should be treated as properties of the evaluator, not the benchmark. By grounding the taxonomy in the judge's own error vocabulary and refining it against human judgment, JUDGEFIT exposes model-specific blind spots that overall rankings cannot anticipate.

The authors argue that this approach yields a more granular understanding of VLM capabilities:

It reveals that "strong-on-average" VLMs are not uniformly strong across all physical rules.
It demonstrates that reliability patterns differ markedly across model families.
It provides a framework for discovering which physical rules a specific VLM can and cannot ground in human judgment.

The work suggests that moving away from fixed, one-size-fits-all rubrics toward adaptive, per-model taxonomies is essential for accurately evaluating and improving physical consistency in generative video models.