Technical Summary: Gate-and-Merge for Zero-Shot Compositional Personalization of VLMs

Problem Definition

Vision-Language Models (VLMs) possess strong generalization capabilities for open-world tasks but lack the ability to understand personalized, user-defined concepts (e.g., specific faces, pets, or artwork styles) that are absent from pre-training data. While existing personalization methods can learn single concepts, they struggle with compositional personalization: the ability to recognize or describe scenes containing multiple user-defined concepts simultaneously without prior co-occurrence training.

Current approaches face significant limitations:

• Parameter-based methods (e.g., learnable prompts or concept embeddings) often entangle concepts during training, making it difficult to isolate and recombine them in novel configurations.
• Retrieval-based methods avoid parameter updates but rely on external databases, raising privacy concerns and failing to internalize concepts for reasoning tasks.
• Existing solutions generally lack a mechanism to compose multiple independently learned concepts at inference time without degrading performance due to interference.

Methodology: Gate-and-Merge

The paper proposes Gate-and-Merge, a zero-shot framework designed to enable compositional personalization without requiring co-occurrence training or external databases. The framework operates in two distinct phases:

1. Single Concept Learning (Training Phase)

Each user-defined concept $c$ is learned independently as a modular, lightweight representation consisting of two components:

• Concept Token ($\langle c \rangle$): A new token added to the tokenizer vocabulary with a trainable embedding vector $e_c$ and a corresponding output head row $w_c$. This serves as the textual identifier for the concept.
• LoRA Adapter ($\delta_c$): A Low-Rank Adaptation module ($\delta_c = A_c B_c$) applied to the language model's weights. This allows the model to adapt to the visual appearance of the concept without modifying the frozen vision encoder.

The vision encoder remains frozen throughout training to maintain a stable, shared feature space, ensuring that visual features remain compatible across different concepts. The model is trained on a small support set of images and text using a standard masked language modeling objective.

2. Multi-Concept Composition (Inference Phase)

At inference, the model must handle inputs containing multiple concepts (e.g., an image with concept $A$ and concept $B$) without having seen them together during training. The framework employs a Gate-and-Merge mechanism:

• Gating (Selection): A gating mechanism determines which concept modules are relevant to the current input (image $x$ and query $q$) by evaluating two cues:

  • Textual Relevance: Checks if the concept token $\langle c \rangle$ is explicitly mentioned in the query.
  • Visual Relevance: Computes cosine similarity between the image patch features and the concept's prototype features.
  • A concept is activated if it is either textually mentioned or visually supported above a threshold $\tau$.

• Merging (Fusion): Once the relevant set of concepts $S$ is identified, their corresponding LoRA updates are merged directly in weight space. To prevent destructive interference caused by naively summing updates:

  • Sparsification: Each LoRA update is stochastically dropped (Bernoulli masking) and rescaled to retain only dominant signals, reducing cross-concept conflicts.
  • Sign-Aware Averaging: The method enforces sign consistency. For each weight coordinate, updates with conflicting signs are discarded, and only sign-consistent updates are averaged. This ensures that the fused adapter preserves the identity of each concept while combining them.

The final composed model is $M_{comp} = M \oplus \delta_{merge}$, where $\delta_{merge}$ is the fused adapter. This process is training-free at inference.

Key Contributions

1. Gate-and-Merge Framework: A modular framework for zero-shot compositional personalization that learns concepts independently and composes them at inference without co-occurrence supervision.
2. Modular Representation: A design representing each concept via a lightweight LoRA module and a unique concept token, supporting scalable and privacy-preserving personalization.
3. Training-Free Composition: A mechanism that selectively activates and merges LoRA modules based on input-dependent textual and visual cues, utilizing sparsification and sign-aware merging to mitigate interference.
4. Empirical Validation: Demonstration through extensive experiments that the method outperforms existing parameter-based personalization approaches in both single-concept and multi-concept settings.

Experimental Results

The authors evaluated the method on a benchmark constructed from a multi-concept dataset, covering 118 distinct concepts (real-world individuals, objects, fictional characters).

• Visual Recognition: In tasks requiring the identification of one or multiple concepts in an image, Gate-and-Merge achieved superior performance compared to baselines like MyVLM and Yo'LLaVA. Notably, in the challenging "Multiple-Multiple" setting (identifying multiple queried concepts simultaneously), the proposed method achieved 69.9% accuracy, significantly outperforming Yo'LLaVA (29.3%) and maintaining high balanced recall.
• Text and Visual QA: The method demonstrated robustness in both text-only reasoning and multimodal Visual Question Answering (VQA), achieving the most balanced performance across single and multiple concept settings.
• Captioning: While retrieval-based methods (RAP-MLLM) showed high recall in single-concept settings, they relied heavily on "copy-paste" of stored text. Gate-and-Merge provided more robust concept grounding, generating captions that better integrated visual evidence, though performance naturally degraded in multi-concept captioning scenarios compared to single-concept tasks.
• Ablation Studies:
  • Sparsity Ratio ($p$): Performance peaked at a sparsity ratio of 0.7–0.8, confirming that moderate sparsity effectively suppresses interference while retaining adaptation signals.
  • Gating Threshold ($\tau$): A threshold of 0.3 yielded optimal results, balancing the suppression of irrelevant activations with the retention of valid concepts.

Significance and Claims

The paper claims that Gate-and-Merge addresses the critical gap in VLM personalization by enabling compositional reasoning without the need for joint training or external data storage. By keeping concepts disentangled during training and fusing them intelligently at inference, the method allows models to handle novel combinations of user-defined concepts.

The authors position this work as a step toward scalable, modular personalization where concepts can be added, removed, or combined dynamically. They acknowledge limitations, noting that highly entangled concepts may require more expressive mechanisms and that sparsification carries a risk of losing fine-grained details. However, the results suggest that this approach offers a stable and effective solution for integrating personalized knowledge into VLMs for complex, multi-concept scenarios.