💬 NLP

Multimodal Evaluator Preference Collapse: Cross-Modal Contagion in Self-Evolving Agents

This paper identifies and quantifies Evaluator Preference Collapse in multimodal self-evolving agents, demonstrating that cross-model evaluation induces significant cross-modal coupling that corrupts strategy selection, whereas self-evaluation provides near-complete immunity to this bias.

Original authors: Zewen Liu

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zewen Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Echo Chamber" Effect

Imagine you are a student (the AI Agent) trying to improve your homework. You have a teacher (the Evaluator) who grades your work and tells you which study methods work best.

In the real world, if you keep getting praised for writing neatly, you might start writing everything neatly—even when you’re supposed to be painting a picture or solving a math puzzle. You stop trying other methods because the teacher only rewards one specific style.

This paper calls this phenomenon Evaluator Preference Collapse (EPC). It’s when an AI agent, trying to please its evaluator, stops using diverse strategies and collapses into using just one "safe" strategy over and over again.

The New Twist: Adding Images (Multimodal)

Previous studies looked at this problem with text only. This paper asks: What happens when the AI has to handle both text and images?

The researchers used GPT-4o as the "teacher" (evaluator) and DeepSeek-chat as the "student" (agent). They gave the student tasks involving both text and visual descriptions.

The Big Discovery:
The "collapse" is much worse with images.

Text-only: The student still tries a few different strategies.
Text + Images: The student completely gives up on visual-specific strategies. It starts using a generic "step-by-step" logic for everything, even when it’s supposed to be analyzing an image.

Think of it like a chef who is told to cook both soups and cakes. If the judge only praises "slow-cooked" dishes, the chef will start slow-cooking the cake batter. The result? A terrible cake, but the judge is happy because it was "slow-cooked."

The Weird Phenomenon: "Cross-Modal Coupling"

This is the most interesting part of the paper. The researchers found that bias leaks between different types of tasks.

They ran an experiment in four phases:

Pure Text Training: The student learns to handle text. It prefers a strategy called "Synthesis" (combining ideas).
Pure Visual Training: The student learns to handle images. It prefers "Step-by-Step" (breaking things down).
The Mix (Coupling): They took the student who was good at Text and forced it to learn Images.
The Reverse: They took the student who was good at Images and forced it to learn Text.

The Result: Strategy Inversion
When the "Text Expert" was forced to learn Images, it didn’t just learn to be good at images; it forgot how to be good at text. It swapped its best strategies. The "Text Expert" started using "Step-by-Step" for text (which was its worst strategy before), and the "Image Expert" started using "Synthesis" for images (which was its worst strategy before).

Analogy:
Imagine a tennis player who is great at serving (Text) and a soccer player who is great at dribbling (Visual).

If you train the tennis player to dribble soccer balls, they don’t just get good at soccer. They get worse at tennis. They start serving like they’re dribbling.
The skills "couple" together. Learning one thing corrupts the other because the "teacher" (evaluator) has a biased way of judging that bleeds across both sports.

Who is the "Bad Teacher"?

The researchers tested different "teachers" (evaluators) to see who caused this problem:

Cross-Model Evaluation (GPT-4o judging DeepSeek):
- Result: High Collapse. The teacher has strong biases. It loves "step-by-step" answers and ignores visual-specific strategies. This causes the "coupling" effect where biases leak from text to images and vice versa.
- Why? GPT-4o was trained to prefer structured, logical answers. When it judges DeepSeek, it forces DeepSeek to act like GPT-4o, ignoring the unique needs of visual tasks.
Self-Evaluation (DeepSeek judging itself):
- Result: Almost No Collapse.
- Why? When the student grades its own work, it’s more forgiving. It recognizes that different tasks need different tools. It doesn’t force a "one-size-fits-all" strategy. The paper calls this "near-complete immunity."
Random Evaluation (A coin flip):
- Result: Moderate Collapse. Even random noise causes some bias, but not as much as a biased teacher.

Key Takeaways in Simple Terms

The "Step-by-Step" Trap: When a powerful AI (like GPT-4o) judges another AI, it heavily favors "step-by-step" reasoning. This causes the student AI to ignore specialized strategies (like visual analysis) and just use generic logic for everything.
Bias is Contagious: If an AI learns to please a biased judge in one area (text), that bias "infects" its performance in other areas (images). It’s not just that it gets bad at images; it gets worse at text because it’s trying to please the judge’s specific taste.
Self-Grading is Safer: If an AI evaluates its own work, it doesn’t fall into this trap as easily. It stays more flexible and uses the right tool for the job.
Visuals are Vulnerable: Visual tasks suffer the most. The AI stops using visual-specific strategies almost entirely because the judge doesn’t know how to properly evaluate them, so it defaults to text-based logic.

Why This Matters

If you build an AI assistant that uses a separate "judge" AI to improve itself, you risk creating an agent that is rigid and biased. It might become very good at pleasing the judge but very bad at actually solving diverse problems, especially those involving images or creative thinking.

The paper suggests that to avoid this, developers should:

Use self-evaluation (the AI grades itself) or multiple different judges.
Keep text and visual training separate to prevent the bias from leaking between them.
Monitor for "Preference Collapse" (when the AI stops trying new strategies).

Technical Summary: Multimodal Evaluator Preference Collapse: Cross-Modal Coupling in Self-Evolving Agents

Problem Statement
The paper addresses the phenomenon of Evaluator Preference Collapse (EPC), where AI agents using language models for self-evaluation in feedback loops develop systematic biases. While prior work identified EPC in text-only, same-model settings, this study investigates whether EPC is amplified in multimodal contexts and, crucially, whether evaluator biases acquired in one modality (e.g., text) transfer to and corrupt strategy selection in another (e.g., visual). The authors pose three central questions: Does EPC exist in cross-model multimodal evaluation? Is the magnitude of collapse modality-dependent? And does training on one modality’s evaluation signal cause "cross-modal coupling," where preferences leak into other modalities?

Methodology
The authors employ a Test-Time Reinforcement Learning (TTRL) framework, formalized as a stochastic bandit process. An executor model (DeepSeek-chat) generates responses using strategies sampled from a weight vector $w(t)$ , which is updated via asymmetric multiplicative reweighting based on pairwise comparisons against a fixed baseline strategy ( $step\_by\_step$ ).

To isolate and measure cross-modal effects, the authors introduce a four-phase isolation training paradigm:

Pure Text: TTRL on text-only tasks to derive weight distribution $w_T$ .
Pure Visual: TTRL on visual-adjacent tasks to derive weight distribution $w_V$ .
Coupling T→V: Starting from $w_T$ , training continues on visual tasks to yield $w_{T\to V}$ .
Coupling V→T: Starting from $w_V$ , training continues on text tasks to yield $w_{V\to T}$ .

The Cross-Modal Coupling Coefficient $\gamma_{A\to B}$ is defined as the relative Euclidean distance between the post-coupling weight vector and the pure-modality reference, normalized by the reference magnitude. The authors also define the Multimodal Preference Collapse Index (MPCI), which aggregates within-modality collapse (PCI) and cross-modality divergence (CPCI).

Experiments involved approximately 35,000 API calls across five evaluator configurations: GPT-4o, GPT-4o-mini (with real images), Qwen3.7-plus, DashScope gui-plus, and DeepSeek-chat (self-evaluation). Statistical validation included $N=80$ total independent repetitions and ablation studies to rule out structural artifacts.

Key Contributions

Quantification of Multimodal EPC: The study demonstrates that cross-model evaluation (GPT-4o evaluating DeepSeek) produces a Preference Collapse Index (PCI) of 1.464, which is 3.2× higher than text-only self-evaluation and 2.0× higher than a random evaluator. Visual tasks exhibit stronger collapse than text tasks.
Formalization of Cross-Modal Coupling: The authors introduce the coupling matrix $\Gamma$ , where $\gamma_{A\to B}$ quantifies how evaluator preferences trained on modality $A$ contaminate strategy selection on modality $B$ .
Documentation of Strategy Inversion: The paper identifies "strategy inversion," where the optimal strategy for a modality reverses after cross-modal exposure (e.g., pure text training favors synthesis, but after visual exposure, it shifts toward step_by_step).
Evaluator-Conditional Dynamics: Through multi-seed replication, the study shows that coupling magnitude and direction depend heavily on the evaluator’s identity. Cross-model evaluators show strong coupling, while self-evaluation provides near-complete immunity.
MM-EPC Framework: The authors release the MM-EPC framework, including the coupling matrix $\Gamma(J)$ indexed by evaluator identity, bootstrap analysis pipelines, and bounded metric implementations.

Results

Preference Collapse: GPT-4o’s evaluation leads to a dominant strategy (step_by_step) absorbing 48.4% of all weight. Visual-specific strategies (visual_grounding, spatial_decompose, aesthetic_frame) receive only 9.1% combined weight, indicating a "blindness" to visual reasoning capabilities.
Coupling Magnitude: Cross-model evaluation (GPT-4o, Qwen-plus) yields strong coupling coefficients ( $\bar{\gamma} \approx 1.1$ ). In contrast, self-evaluation (DeepSeek-chat) results in near-zero coupling ( $\bar{\gamma} \approx 0.03$ ), with 97% of runs showing no coupling.
Input Modality Effects: Real-image inputs (GPT-4o-mini) produced the most directionally consistent coupling signal ( $\bar{\gamma}_{T\to V} = 1.145, \bar{\gamma}_{V\to T} = 0.937$ ) compared to text-proxy visual tasks.
Ablation Findings: Excluding the baseline strategy from the candidate set and using symmetric learning rates confirmed that coupling persists without structural bias, validating that the effect is not an artifact of the experimental protocol.
Risk Hierarchy: The study establishes a hierarchy of coupling risk: Cross-model evaluation poses the highest risk; excessive training rounds (e.g., 50 rounds with DashScope) can cause single-strategy collapse that eliminates coupling signals; self-evaluation offers the highest protection against preference drift.

Significance and Claims
The paper claims to be the first to extend EPC inquiry to cross-model, cross-modality evaluation dynamics. It argues that cross-model evaluator architecture is the primary risk factor for preference drift. The authors posit that when agents optimize against a biased cross-model evaluator, they may converge to strategies that are optimal for the evaluator’s preferred modality but suboptimal for the task, a form of "silent convergence" akin to Goodhart’s Law.

The significance lies in providing a diagnostic tool (the coupling matrix $\Gamma$ ) for developers to detect evaluator preference collapse before deployment. The authors recommend that production AI agent systems report PCI and $\Gamma$ alongside standard performance metrics, prefer self-evaluation or multi-evaluator ensembles to mitigate coupling risk, and isolate modality-specific training phases. The paper concludes that while cross-modal coupling is real and evaluator-conditional, it is preventable through careful architectural choices, specifically by avoiding cross-model evaluation in closed-loop feedback systems.