Show, Don't Ask: Generative Visual Disambiguation for Composed Image Retrieval with Turn-Valid Coverage

Technical Summary: CLARA for Composed Image Retrieval

Problem Statement
Composed Image Retrieval (CIR) allows users to search a corpus using a reference image and a modification text (e.g., "an outfit like this, but more formal"). Current systems typically assume a query maps to a single target image, evaluated via Recall@K against a single annotation. However, the authors argue this assumption is flawed: a composed query rarely specifies a unique image but rather defines a "region of the corpus." The specific member the user intends is underdetermined by the query alone.

Recent attempts to address this ambiguity wrap a retriever in a conformal prediction layer to return a candidate set with a guaranteed coverage probability, using text questions to resolve residual uncertainty. The authors identify two critical failures in this existing approach:

1. Loss of Coverage Guarantee: Conformal coverage guarantees rely on data exchangeability. When a system adaptively asks a question and updates its belief based on the answer, it induces a feedback covariate shift. Re-applying the original calibration threshold to the updated belief invalidates the coverage guarantee, causing the actual coverage to drift significantly below the nominal level after the first turn.
2. Inefficient and Circular Clarification: Text questions are a low-bandwidth channel for conveying fine-grained visual distinctions (viewpoint, texture, style). Furthermore, current text-based policies use a multimodal model to predict user answers to select questions, creating a circular evaluation where the model both poses and answers, potentially optimizing for a simulator rather than real users.

Methodology: CLARA
The authors propose CLARA (CLArification by Rendering Alternatives), a framework that replaces text-based questioning with generative visual disambiguation while maintaining a mathematically valid coverage guarantee across interaction rounds.

1. Calibrated Belief and Conformal Sets:
   The system maintains a belief distribution $p(I|q, h_m)$ over the corpus, conditioned on the query and interaction history. It uses adaptive conformal prediction to generate a candidate set $C$ such that the probability of the true target being in the set is at least $1-\alpha$. The size of this set serves as a direct measure of ambiguity.

2. Generative Visual Disambiguation (Show, Don't Ask):
   When the candidate set is too large (indicating high ambiguity), CLARA does not ask a text question. Instead:

   • Mode Partitioning: It clusters the candidate set into $k$ modes based on interpretable ambiguity axes (preserved attributes, changed attributes, viewpoint, background).
   • Generate-to-Cover: It selects a subset of modes to maximize a submodular coverage-diversity objective, ensuring the panel represents the full belief space without redundancy.
   • Rendering and Snapping: A conditional diffusion model renders a prototype image for each selected mode. Crucially, each rendered prototype is "snapped" to the nearest real corpus exemplar (the medoid of the mode). The user sees the rendered image but selects the real corpus subset. This ensures that generation artifacts cannot inflate the prediction set or violate the coverage guarantee.
   • User Interaction: The user simply picks the prototype closest to their intent. This provides a high-bandwidth, model-free signal.

3. Turn-Valid Coverage via Selection Reweighting:
   To address the feedback covariate shift caused by the user's selection, CLARA treats each clarification round as an explicit shift in the query distribution.

   • It computes a likelihood ratio weight $w_m$ based on the user's selection history and the selection model $\pi$.
   • It applies weighted split conformal prediction, reweighting the nonconformity scores during calibration.
   • Proposition 1 proves that with this reweighting, the coverage guarantee ($Pr(IT \in C) \geq 1-\alpha$) holds at every committed round, not just the first.

4. Belief Update:
   Upon a user pick, the belief is updated via a logical reweighting that multiplies the previous belief by a soft membership likelihood (floored to prevent discarding the true target on a mistaken pick) and a semantic update appending the chosen mode to the text query.

Key Contributions

• Identification of Feedback Covariate Shift: The paper demonstrates that standard conformal coverage in interactive CIR fails after the first turn due to adaptive selection and proposes a theoretical framework to correct this via selection-reweighted conformal prediction.
• Generative Visual Disambiguation: CLARA replaces text questions with a coverage-driven rendering of candidate modes. This eliminates the "answer-model" circularity and leverages the high bandwidth of visual signals for fine-grained distinctions.
• Turn-Valid Guarantee: The authors provide a proof that their weighted conformal approach maintains the nominal coverage guarantee across all interaction rounds.
• AmbiCIR-V: A new evaluation protocol and dataset extension that unifies open-domain and fashion benchmarks with visual-clarification metrics, including coverage-through-turns and ambiguity-axis annotations.

Results
Experiments were conducted on open-domain (CIRR), multi-positive (CIRCO), and fashion (FashionIQ) benchmarks.

• Single-Turn Performance: CLARA is statistically tied with state-of-the-art single-turn methods, confirming the resolution layer is cost-free for precise queries.
• Coverage Stability: Unlike "naive" interactive conformal methods which drift up to 10 percentage points below nominal coverage by round 3, CLARA maintains nominal coverage (approx. 90%) across all rounds and ambiguity axes.
• Interaction Efficiency: CLARA reaches the target in fewer rounds than the strongest text-question policy (EIG-TEXT). The advantage is most pronounced on viewpoint and attribute ambiguity, where visual selection outperforms text questioning.
• Human Study: A study with 30 participants confirmed that real users prefer visual panels over text questions and achieve higher success rates with fewer turns, validating the simulated results.
• Robustness: The system remains robust to user mistakes (soft likelihood updates) and adversarial picks, with coverage degrading gracefully rather than catastrophically.

Significance and Claims
The paper claims to advance the field by reframing CIR from a "one-query-one-target" task to a "calibrated intent resolution" problem. The significance lies in:

1. Theoretical Rigor: Establishing that interactive retrieval is a feedback covariate shift problem and providing the first method to guarantee valid coverage at every turn of interaction.
2. Practical Usability: Demonstrating that "showing" (visual rendering) is a superior channel to "asking" (text) for resolving visual ambiguity, removing the need for complex, potentially biased answer-models.
3. Faithful Synthesis: Introducing a "snap-to-corpus" mechanism that allows the use of generative models for visualization without compromising the statistical guarantees of the retrieval system.

The authors conclude that treating retrieval as a visually interactive, calibrated process—measuring ambiguity, committing when confident, and showing alternatives when not—represents a necessary step beyond current single-turn assumptions.