💻 computer science

A Good Initialization is All You Need for Faithful Visual Attribution

This paper introduces CoPAIR and TRACE, two forward-only methods that optimize the initialization and direct search for compact top-k visual evidence masks, thereby achieving state-of-the-art faithful attribution performance across image classification and multimodal large language models while enabling actionable single-point repairs.

Original authors: Zihan Gu, Jiayu Wang, Hua Zhang, Yue Hu

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zihan Gu, Jiayu Wang, Hua Zhang, Yue Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart but sometimes confused robot that looks at a picture and makes a guess, like saying, "That's a cow!" even when it's actually a horse. You want to know: Which specific parts of the picture convinced the robot to make that mistake?

This paper is about building a better "detective" to find those specific parts. Here is the breakdown using simple analogies:

The Problem: The "Long List" vs. The "Cheat Sheet"

Traditionally, when we try to explain a robot's decision, we ask it to rank every single piece of the image from "most important" to "least important." It's like asking a detective to write a 100-page report listing every single clue in a case, ranked by importance.

But often, we don't need the whole 100-page report. We just need the top 5 clues that actually solved the case. In the world of AI, this is called a "compact top-k evidence mask." It's a small, focused highlight of the image that proves why the robot thought what it thought.

The problem is that finding these top 5 clues is hard.

The "Greedy" Approach: Imagine a detective who picks the single best clue, then the next best, then the next. This works okay, but it's slow (like reading a book one word at a time) and sometimes misses the big picture because two clues might only make sense when looked at together.
The "Coarse" Approach: Imagine grouping the whole image into big chunks (like "sky," "ground," "trees") and picking the best chunk. This is fast, but it's too blurry. You might pick "ground" as the clue, but the real clue was a tiny rock on the ground.

The Solution: Two New Detective Tools

The authors introduce two new methods to find the perfect small set of clues faster and more accurately.

1. COPAIR: The "Group Scout"

Think of this as a scout who first looks at the image in big, blurry groups (like looking at a map of countries instead of individual cities).

How it works: The scout quickly finds the best "countries" (groups of pixels) and checks if any two countries work well together.
The Trick: Once the scout finds a promising group, they zoom in to find the specific cities (fine details) inside it.
Why it helps: It gives the main detective a great "head start" (a warm start). It doesn't solve the whole case alone, but it saves the detective from wasting time looking at the wrong areas.

2. TRACE: The "Lottery Ticket" Search

This is the paper's main star. Imagine you are trying to find the winning combination in a lottery, but you can't just pick numbers one by one. You have to pick a whole ticket (a specific set of 5 numbers) at once.

How it works:
1. Draw: TRACE randomly draws a "ticket" (a random set of 5 image regions).
2. Test: It asks the robot, "If I only show you these 5 regions, do you still guess 'cow'?"
3. Learn: If the robot says "Yes!", TRACE remembers that combination. If the robot says "No," it forgets it.
4. Refine: Over many rounds, TRACE starts drawing tickets that look more and more like the winning combinations it found earlier. It's like a student studying past exam questions to guess the right answers for the next test.
The Result: TRACE finds the perfect small set of clues directly, without needing to rank every single pixel in the image.

Why This Matters (The "Aha!" Moment)

The paper shows that these methods are not just faster; they are smarter.

For Standard Images: When tested on standard image recognition tasks (like identifying objects in photos), using TRACE to give the detective a "head start" made the final explanation more accurate than ever before.
For Hallucinations (The Big Win): This is where it gets exciting. When the robot hallucinates (makes things up), the old way required a long, slow process to fix it.
- The Old Way: "Here is a long list of clues. Maybe the first one helps, maybe the second one..."
- The TRACE Way: "Here is one single mask (a specific set of 5 regions). If you show the robot only this, it immediately realizes its mistake and corrects itself."

In their tests, this "single mask" approach fixed 94% to 96% of the robot's hallucinations. It's like finding the one specific piece of evidence that instantly clears up a misunderstanding, rather than arguing through a whole list of possibilities.

Summary

Old Way: Slowly rank every pixel to build a long explanation.
New Way (TRACE): Use a smart, iterative search to find the perfect small group of pixels that explains the decision.
Benefit: It's faster, more accurate, and can directly "fix" a robot's wrong answer by showing it the right evidence, without needing a long report.

The paper proves that sometimes, you don't need to know everything about the image; you just need to know the right few things.

Technical Summary: A Good Initialization Is All You Need for Faithful Visual Attribution

1. Problem Formulation

Faithful visual attribution aims to identify which image regions causally support a model's prediction. While traditional benchmarks often assume a "full-ordering contract"—requiring a method to rank all image regions to generate insertion and deletion curves—many practical applications, particularly in Multimodal Large Language Models (MLLMs) for hallucination attribution and repair, only require a compact top-k evidence mask.

The paper identifies a gap in existing search-based methods:

Greedy Search: While effective, it requires $O(n^2)$ forward passes and often gets trapped in local optima, missing high-value combinations of regions that only emerge through interaction.
Coarse Grouping: Methods like PhaseWin reduce cost by partitioning regions but may aggregate redundant content or miss fine-grained interactions.
One-Step Scoring: Independent region scoring fails to capture interactions where evidence is only valuable in combination.

The core challenge is to find a sparse, high-value set of exactly $k$ fine regions under a black-box, forward-only evaluation budget, without necessarily generating a full ranking of all regions.

2. Methodology

The paper introduces two forward-only methods designed to address the "mask-first" attribution problem: COPAIR and TRACE.

COPAIR: Coarse Pairwise Attribution Initializer with Release

COPAIR is a deterministic initializer derived from the analysis of the gap between PhaseWin and Greedy search. It operates on the hypothesis that early evidence quality is a granularity problem; fine regions can fragment evidence that is more stable at a coarser scale.

Mechanism: It clusters fine regions into coarse spatial groups. It evaluates all coarse singletons and exhaustively searches for the best coarse pairs within a high-sufficiency pool.
Selection Logic: It selects an initial candidate ( $C_0$ ) based on a pair-gain margin ( $\delta$ ) and a sufficiency threshold ( $\tau$ ). If a single coarse group is sufficient, it is selected; otherwise, the best pair is chosen. If the score is still low, it absorbs one additional group.
Output: The selected coarse candidate is internally ordered and partially released (keeping the top 80% of the score) to serve as a warm-start prefix for Greedy or PhaseWin continuation search.
Limitation: COPAIR is not a direct solution for the fixed- $k$ fine-region mask problem because coarse groups can merge object parts and background fragments, failing to provide a minimal compact explanation.

TRACE: Top-k Region Attribution via Cross-Entropy

TRACE directly targets the fixed-cardinality fine-region mask problem, bypassing the need for a full ordering.

Mechanism: It adapts the cross-entropy method to the discrete $k$ -hot space. It maintains logits $\theta$ over all fine regions and samples candidate masks using Gumbel-top-k perturbation.
Optimization Loop:
1. Sampling: Generate $M$ candidate masks of size $k$ per round.
2. Evaluation: Score masks using the model's response (sufficiency or combined sufficiency-necessity proxy).
3. Elite Retention: Keep the top $\rho_e$ fraction of samples (elite set).
4. Distribution Update: Estimate the empirical inclusion frequency of regions in the elite set. Smooth this frequency and update the logits $\theta$ to increase the probability mass on regions that appear in high-scoring masks.
Outputs:
- Direct Mode: Returns the best observed mask $S^*$ directly as the attribution output.
- Initialization Mode: Uses $S^*$ as a seed to initialize Greedy or PhaseWin for tasks requiring a full ranking.
Theoretical Guarantee: The paper provides a finite-budget recovery analysis showing that cross-entropy updates increase the probability mass on "near-optimal evidence basins" defined by semantic block partitions, improving recovery probability over random search.

3. Key Contributions

Problem Reframing: The authors formulate the compact top-k evidence mask as a primary output contract for faithful visual attribution, distinct from but compatible with full-ordering evaluation.
COPAIR: A principled initializer derived from the PhaseWin–Greedy gap diagnosis, using coarse spatial grouping and pairwise enumeration to warm-start full-ordering searches.
TRACE: A novel cross-entropy search method that directly optimizes fixed-size fine-region masks, capable of operating as a standalone attribution tool or an initializer.
State-of-the-Art Performance: The methods establish new frontiers for faithful visual attribution under inclusive forward-call accounting across image classification and MLLM tasks.

4. Experimental Results

Image Classification (ImageNet)

Evaluated on CLIP ViT-L/14, CLIP RN101, and ResNet-101 using SLICO-64 regions:

Full-Ordering: Initialized search methods (TRACE+Greedy, COPAIR+Greedy) consistently outperform raw Greedy and PhaseWin in insertion AUC and high-score metrics across Correct, Cause, and Repair splits.
Efficiency: TRACE+Greedy achieves these gains with lower or comparable Model Evaluation Calls (MEC) compared to raw Greedy, demonstrating that better initialization reduces the cost of the continuation search.
Backbone Sensitivity: On CLIP RN101, COPAIR+Greedy showed slight advantages in some metrics, suggesting coarse pairing is valuable for convolutional backbones, while TRACE+Greedy excelled on ResNet-101 and ViT.

MLLM Attribution and Repair (POPE and RePOPE)

Evaluated on Qwen2.5-VL-3B-Instruct and LLaVA-v1.5-7B:

Attribution: TRACE+Greedy achieved the strongest insertion faithfulness (Ins) and highest peak scores (High) among search-based methods on both POPE (decision faithfulness) and RePOPE (repair-oriented recovery).
Direct Repair (Single-Point): In the RePOPE repair setting, where a single mask intervention is applied to correct a hallucination:
- TRACE Direct repaired 94.44% of Qwen cases and 96.00% of LLaVA cases.
- This significantly outperformed full-curve baselines (LLaVA-CAM, IGOS++) and COPAIR Direct, validating that compact evidence masks can be actionable outputs rather than just prefixes of rankings.

5. Significance and Claims

The paper claims that a good initialization is sufficient to push the frontier of faithful attribution, but more importantly, that direct search for compact masks is a viable and superior strategy for specific downstream tasks like MLLM repair.

Shift in Paradigm: The work shifts the focus from generating complete insertion curves to identifying actionable, compact evidence sets.
Actionability: The results demonstrate that a single, well-chosen mask (TRACE Direct) can repair hallucinations more effectively than methods requiring full curve evaluation, making attribution more practical for real-world MLLM debugging.
Model Agnosticism: Both COPAIR and TRACE are model-agnostic, relying only on forward passes, making them applicable to black-box foundation models where gradients are unavailable or unreliable.
Modesty: The authors acknowledge limitations, including the overhead of initialization for very small $n$ , the stochastic variance in TRACE's sampling, and the requirement to pre-select the mask size $k$ . They do not claim to solve the global optimum for all possible masks but rather provide a budgeted search procedure that concentrates samples on high-value evidence basins.