💬 NLP

A Systematic Evaluation of Positional Bias in Multi-Video Summarization with MLLMs

This paper systematically evaluates positional bias in multi-video summarization across nine MLLMs using a new benchmark, revealing that summary quality is significantly influenced by input order and domain, with current mitigation strategies failing to fully eliminate these biases.

Original authors: Huangchen Xu, Yuan Wu, Yi Chang

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Huangchen Xu, Yuan Wu, Yi Chang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef preparing a tasting menu for four different dishes: a soup, a salad, a steak, and a dessert. You ask a very smart, but slightly distracted, AI assistant to write a short, perfect description for each dish.

You might think the AI would describe the steak perfectly regardless of whether it was the first or the last dish on the menu. But this paper, titled "A Systematic Evaluation of Positional Bias in Multi-Video Summarization with MLLMs," discovered that the AI is actually quite picky about where a video sits in the lineup.

Here is the breakdown of their findings using simple analogies:

1. The "Seat at the Table" Problem

The researchers found that if you show an AI four videos at once and ask it to summarize each one, the quality of the summary changes depending on the video's position (1st, 2nd, 3rd, or 4th).

The "Middle-Child" Syndrome: Just like a middle child might get less attention than the firstborn or the youngest, the videos in the middle of the list (positions 2 and 3) often get worse summaries. The AI tends to forget details or write vaguer descriptions for them compared to the videos at the very beginning or the very end.
The "First Impression" vs. "Last Hurrah": Sometimes the AI loves the first video (the "Primacy" effect), and sometimes it loves the last one (the "Recency" effect). But often, it just ignores the middle ones.

2. The "Magic Trick" of Hiding Bias

The paper points out a tricky part: You can't just look at the average score to see if the AI is biased.

The Analogy: Imagine a student who gets an A on the first test, an F on the second, an F on the third, and an A on the fourth. Their average grade looks okay (a B), but they clearly have a problem with the middle tests.
The researchers found that some AI models have a "neutral" average score, but if you look closer, they are actually failing the middle videos while doing great on the ends. They created special tools (metrics) to catch this "Middle-Child Weakness" that standard tests miss.

3. It's Not Just About "More Eyes"

The team tested if giving the AI more resources would fix the problem.

More Frames: They showed the AI more pictures (frames) from each video.
More Words: They told the AI it could write longer summaries.
The Result: It didn't help much. Even when the AI had more "visual budget" or more space to write, it still ignored the middle videos. It's like giving a distracted student a bigger notebook; they still won't write about the middle chapters.

4. The "Order Matters" Experiment

To prove this wasn't a fluke, they used a cyclic rotation method.

The Analogy: Imagine four friends (Video A, B, C, D) taking turns sitting in four different chairs (1, 2, 3, 4).
- Round 1: A sits in Chair 1, B in 2, C in 3, D in 4.
- Round 2: A sits in Chair 2, B in 3, C in 4, D in 1.
- And so on.
By doing this, they ensured that every video got to sit in every chair. They found that Video A got a great summary when it sat in Chair 1, but a bad summary when it sat in Chair 3. The video content didn't change, but the seat did.

5. Trying to "Fix" the AI with Prompts

The researchers tried to "coach" the AI to be fair.

The "Be Fair" Instruction: They added a note to the AI's instructions saying, "Please pay equal attention to every video."
The Result: It didn't really work. The AI still favored the edges. It's like telling a tired student, "Please study all chapters equally," and they still end up skimming the middle ones.
One-at-a-Time: They tried asking the AI to summarize only one video at a time, even though it saw all four. This helped a little bit with the middle videos, but it wasn't a perfect fix.

6. The "Leakage" Issue

Finally, they found that sometimes the AI gets confused and mixes up the stories.

The Analogy: If you ask the AI to describe the "Dessert" video, it might accidentally describe a detail from the "Soup" video because it got mixed up about which video was which. This happens more often when the videos are listed in a long row.

The Bottom Line

The paper concludes that current AI models are not reliable when asked to summarize multiple videos at once. They are sensitive to the order in which you show them the videos. If you want a good summary, you can't just dump four videos in a pile; the position of each video matters a lot, and the AI currently struggles to treat them all equally.

The researchers have built a new "test track" (benchmark) to help future AI developers fix this "positional bias" so that one day, an AI can summarize a playlist of videos without forgetting the ones in the middle.

Technical Summary: A Systematic Evaluation of Positional Bias in Multi-Video Summarization with MLLMs

Problem Definition

While Multimodal Large Language Models (MLLMs) have demonstrated increasing capability in video understanding, their reliability when processing multiple video inputs simultaneously remains poorly understood. This paper investigates positional bias in the specific task of multi-video summarization, where a model must generate a distinct, aligned summary for each video within a single input sequence.

The core problem is that the quality of a per-video summary can fluctuate based on the video's input slot position (e.g., first, middle, or last), even when the underlying visual content remains identical. Unlike prior work in NLP or multi-image understanding that focuses on retrieval or "lost in the middle" effects in long contexts, multi-video summarization presents a unique challenge: the model must not only retrieve information but also bind each generated summary to the correct video source while balancing limited generation capacity across multiple input slots.

Methodology

Benchmark Construction

The authors constructed a new benchmark using two public datasets: ActivityNet (covering Cooking, Domestic, and Leisure categories) and the News Video Dataset.

Scenarios: Four categories (Cooking, Domestic, Leisure, News) with varying video durations (short: 0–1 min; long: 1–2 min).
Input Scales: Experiments were conducted on pairwise (2 videos) and listwise (4 videos) inputs.
Ordering: A cyclic permutation design was employed. For $P$ videos, each video appears at every position exactly once across the experimental runs, ensuring balanced analysis while keeping inference costs tractable.
Total Data: 708 unique videos yielding thousands of order-controlled instances.

Models Evaluated

Nine MLLMs were evaluated, including both proprietary and open-source models:

Proprietary: Gemini-3.1-Pro, GPT-5.4.
Open-Source: InternVL3.5 (8B, 14B), Qwen3-VL (8B, 30B-A3B), MiniCPM-o-4.5, GLM-4.1V-9B-Thinking, MiMo-VL-7B-RL.

Evaluation Metrics

Recognizing that lexical-overlap metrics (e.g., ROUGE) are insufficient for highly abstractive summarization, the authors proposed three complementary metrics:

Coverage: Measures the preservation of key information from human-written references.
- ActivityNet domains: Uses an LLM-as-a-Judge (GPT-5.1) to determine if reference sentences are covered (allowing for paraphrasing).
- News domains: Uses reference-fragment coverage (inspired by NEWSROOM) to measure the proportion of matched reference tokens, as news summaries are often short and entity-heavy.
Directional Positional Bias (DPB): A signed metric quantifying preference for early (primacy) vs. late (recency) positions.
- Formula: A weighted sum of coverage scores where early positions have negative weights and late positions have positive weights.
Middle-Edge Gap (MEG): Specifically for 4-video inputs, this metric calculates the difference between the average coverage of middle positions (2 and 3) and edge positions (1 and 4).
- Negative MEG indicates "middle-position weakness," a phenomenon that can be masked by a near-zero DPB if early and late biases cancel each other out.

Alignment Strategy

To ensure summaries are correctly attributed to their source videos, a two-stage alignment strategy was used:

Template-based: Extracting segments based on fixed markers (e.g., [video0], [video1]).
Semantic Fallback: If the template fails, sentences are split and assigned to the video with the highest semantic similarity.

Key Results

1. Heterogeneity of Positional Effects

Positional bias is not a universal primacy or recency pattern; it is model- and domain-dependent.

DPB vs. MEG: Several settings exhibited near-zero DPB (suggesting no bias) but significant negative MEG. For example, InternVL3.5-14B on Leisure-4 showed a DPB of 0.07% but an MEG of -2.65%, indicating that middle positions underperformed despite balanced edge performance.
Scale Dependence: Pairwise (P=2) results did not reliably predict listwise (P=4) behavior. Some models showed strong recency in P=2 that vanished or reversed in P=4, suggesting complex position interactions in multi-video inputs.

2. Robustness and Sensitivity

Visual Budget: Increasing the number of sampled frames or resolution did not uniformly eliminate positional imbalance. Negative MEG patterns persisted across various visual budget configurations.
Summary Length: Increasing the requested summary length (e.g., from 2 to 6 sentences) did not remove middle-position weakness. In some cases, larger budgets made the uneven allocation of detail more visible, as edge positions gained more coverage than middle positions.
Boundary Formats: Changing input separators from black frames to text labels altered the coverage curve but did not provide consistent improvements or eliminate middle-position weakness.

3. Mitigation and Diagnostics

Prompt Instructions: Adding explicit "balanced attention" instructions failed to provide reliable mitigation. In some cases, it flattened the curve by lowering edge-position coverage rather than improving middle positions.
Single-Target Generation: Prompting the model to generate summaries for one video at a time (removing autoregressive prefix interference) improved coverage for weak positions in P=4 settings but degraded performance in P=2 settings. This suggests that joint generation compression contributes to the bias in larger inputs.
Prompt Placement: Placing the task instruction after the videos (video-first) increased the rate of "cross-video leakage" (where details from one video appear in another's summary) compared to the standard prompt-first approach.

Significance and Claims

The paper claims that multi-video summarization remains sensitive to input protocol and position, challenging the assumption that MLLMs are order-invariant.

Limitation of Existing Metrics: The study demonstrates that signed directional bias (DPB) alone is insufficient for diagnosing positional issues, as it can hide non-monotonic effects like middle-position weakness. The proposed MEG metric is necessary to capture these nuances.
Protocol Sensitivity: The results indicate that current models are not robust to input ordering. Small changes in prompt placement or boundary formatting can significantly alter target binding and leakage rates.
Foundation for Future Work: The authors provide a benchmark, specific metrics (Coverage, DPB, MEG), and diagnostic tools (single-target prompting, leakage audits) to serve as a foundation for studying and improving order robustness in multimodal systems.

The paper concludes that while lightweight prompt-level interventions can partially reshape positional effects, they do not fully resolve the issue, suggesting that more robust, order-invariant multimodal systems may require architectural or training-time interventions beyond current inference-time controls.