Technical Summary: Contextualized Visual Personalization in Vision-Language Models

1. Problem Definition: Contextualized Visual Personalization

Despite recent advancements in Vision-Language Models (VLMs), existing approaches struggle to generate responses grounded in a user's specific visual history. Current models often fail to associate a visual input (e.g., a person in a new image) with accumulated user-specific context (e.g., past interactions describing that person's habits or locations).

The authors formalize this challenge as Contextualized Visual Personalization. This paradigm requires VLMs to:

1. Recognize visual concepts in a new query image.
2. Retrieve corresponding user-specific experiences from a context containing interleaved image-text dialogues.
3. Integrate this retrieved context to generate personalized responses.

A critical limitation of prior work is the reliance on "textual shortcuts," where models answer questions by matching keywords in the dialogue without genuinely recognizing the visual entity or grounding the response in visual evidence. Furthermore, existing methods often focus on simple attribute personalization (e.g., names) rather than complex, episodic memories involving fine-grained visual and textual cues.

2. Methodology: The CoViP Framework

To address these limitations, the authors propose CoViP (Contextualized Visual Personalization via Image Captioning), a unified framework that treats personalized image captioning as a proxy task to learn the underlying process of contextualized visual personalization.

2.1. Core Insight

The authors posit that diverse personalization tasks share a common underlying mechanism: interpreting a query image within the context of past user experiences. They decompose the VLM's internal process into two stages:

• Contextual Visual Encoder ($h_\theta$): Grounds the query image and context into a personalized latent representation.
• Task-Specific Generator ($g_\theta$): Produces the final response based on the latent representation and the specific prompt.

Since $h_\theta$ is shared across tasks, the authors argue that optimizing the model to generate personalized image captions (which directly reflect $h_\theta$) will holistically improve personalization capabilities across all downstream tasks.

2.2. Personalized Image Captioning Benchmark

To train and evaluate this capability, the authors constructed a challenging benchmark:

• Dataset Construction: Utilizing generative VLMs, they synthesized 2.8K training and 1.3K test samples. These involve query images containing 1–4 concepts (humans, objects, animals) and interleaved contexts with positive (relevant) and negative (distractor) dialogue samples.
• Evaluation Protocol (CapEval-QAs): To prevent models from bypassing visual recognition, the authors introduced a Multiple-Choice Question Answering (MCQA) probing protocol. An external judge model evaluates whether a generated caption contains sufficient evidence to answer factual questions derived only from the positive context dialogues.
  • Positive Accuracy ($Acc^+$): Measures if the caption correctly answers questions about relevant concepts.
  • Negative Accuracy ($Acc^-$): Measures if the caption correctly abstains (selects "cannot be determined") for questions about irrelevant (negative) concepts.

2.3. Reinforcement Learning (RL) Post-Training

The framework employs an RL-based post-training strategy to optimize the VLM for personalized captioning:

• Verifiable Reward (VR): The reward function $r(s, x, c)$ is decomposed into:
  • Recognition Reward ($r_{vis}$): An F1-based score measuring the model's ability to correctly identify which concept images from the context appear in the query image.
  • Retrieval Reward ($r_{caps}$): A contrastive reward encouraging the model to answer positive MCQs correctly while avoiding hallucinations on negative MCQs.
• Optimization: The authors utilize the GSPO (Group Sequence Policy Optimization) algorithm to maximize the expected verifiable reward, ensuring stable training on long-form generation.

2.4. Caption-Augmented Generation (CAG)

At inference time, CoViP introduces Caption-Augmented Generation (CAG). Instead of directly generating a downstream response, the model first generates a descriptive caption $s$ conditioned on the image and context. This caption is then reused as an explicit conditioning signal to guide the final personalized response generation. This allows the model to leverage the fine-grained details captured during the captioning phase.

2.5. Diagnostic Evaluation

To rigorously test generalization and rule out shortcuts, the authors designed three diagnostic downstream tasks:

1. Last-Seen Detection (LSD): Requires identifying the most recent encounter with a person in the query image from a history of multiple interactions.
2. Last-Action Recall (LAR): Requires retrieving the specific action performed during the most recent encounter.
3. Instruction-Triggered Recall (ITR): Evaluates proactive personalization where the model must recall a specific keyword instruction triggered by the visual presence of a person, even if not explicitly requested in the current turn.

3. Key Results

Extensive experiments on both the benchmark and diagnostic tasks demonstrate the efficacy of CoViP:

• Benchmark Performance: CoViP significantly outperforms existing open-source (e.g., Qwen3-VL) and proprietary (e.g., GPT-4o, Gemini-3.0 Pro) VLMs. The RL-based post-training yields an average performance gain of approximately 40% in Positive Accuracy ($Acc^+$) across varying concept complexities.
• Downstream Generalization: CoViP consistently improves performance across all diagnostic tasks (LSD, LAR, ITR). Notably, while proprietary models show unstable behavior or task-specific gains, CoViP provides holistic improvements.
• Impact of CAG: Applying CAG further amplifies performance, particularly for CoViP, by leveraging the detailed contextual information captured in the generated caption. In contrast, applying CAG to baselines without the specific post-training often leads to performance drops, highlighting the necessity of the captioning pre-training stage.
• Ablation Studies: The results confirm that both the recognition ($r_{vis}$) and retrieval ($r_{caps}$) components of the reward are necessary. Furthermore, the use of GSPO and degeneration filtering (to prevent repetitive outputs) is critical for training stability and performance.

4. Significance and Contributions

The paper claims the following contributions:

1. New Paradigm: It formalizes Contextualized Visual Personalization, shifting the focus from simple attribute retrieval to grounding visual understanding in user-specific episodic history.
2. Unified Framework (CoViP): It introduces a framework that operationalizes this concept through personalized image captioning, RL-based post-training, and caption-augmented generation, demonstrating that a single proxy task can enhance diverse downstream capabilities.
3. Diagnostic Evaluation: It proposes a suite of diagnostic tasks and evaluation protocols (CapEval-QAs) that explicitly rule out textual shortcuts, providing a systematic way to analyze the limitations of existing baselines.
4. Empirical Validation: The work demonstrates that existing VLMs lack robust contextual grounding and that CoViP achieves consistent, significant improvements, positioning it as a foundational step toward robust, generalizable personalization in VLMs.

The authors conclude that while their benchmark relies on synthetic data, it establishes a rigorous diagnostic foundation. They emphasize that to achieve robust personalization, a post-training stage focused on personalized image captioning is essential before adapting models to downstream tasks.