FlowComposer: Composable Flows for Compositional Zero-Shot Learning

1. Problem Statement

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen attribute-object pairs (e.g., "green apple") by recombining primitives learned from seen pairs (e.g., "red apple" and "green banana").

Recent approaches leveraging Vision-Language Models (VLMs) like CLIP typically use Parameter-Efficient Fine-Tuning (PEFT). These methods employ visual disentanglers to separate attributes and objects and manipulate token-level prompts to encode compositions. However, the authors identify two fundamental limitations in these existing PEFT-based designs:

1. Implicit Composition Construction: Composition is achieved merely by concatenating tokens or tuning prompts. This lacks an explicit operation in the embedding space, meaning the resulting composition embedding for unseen pairs may not align well with the actual image embedding.
2. Remained Feature Entanglement: Visual disentanglers fail to strictly separate features. Information from attributes leaks into object streams (and vice versa), contaminating the primitive representations and weakening the model's ability to generalize to unseen combinations.

These issues lead to a strong bias toward seen compositions, where unseen accuracy drops significantly as training progresses.

2. Methodology: FlowComposer

The authors propose FlowComposer, a model-agnostic framework that utilizes Flow Matching (FM) to explicitly model the transport of visual features in the embedding space. Instead of token manipulation, it learns velocity fields to move features toward their targets.

The framework consists of three core components:

A. Primitive Flows (Decomposition)

Two separate rectified flow models are trained:

• Attribute Flow ($v_{\theta_a}$): Learns to transport visual features toward attribute text embeddings.
• Object Flow ($v_{\theta_o}$): Learns to transport visual features toward object text embeddings.
• Mechanism: For a pair of visual feature $x_0$ and text embedding $x_1$, a linear path $x_t = (1-t)x_0 + tx_1$ is constructed. The model predicts a velocity field $v_\theta(x_t, t)$ to match the ground-truth velocity ($x_1 - x_0$).
• Inference: Uses a one-step transport scheme ($\hat{x}_1 = x_0 + v_\theta(x_0, 0)$) for efficiency, mapping image features directly to the text space without numerical integration.

B. The Composer (Explicit Composition)

To address the lack of explicit composition, a learnable Composer module fuses the primitive flows.

• Concept: Instead of concatenating tokens, the Composer learns a weighted combination of the velocity fields of the attribute and object flows.
• Operation: It predicts coefficients $(\hat{a}, \hat{b})$ to combine the normalized primitive velocities: $\hat{v}_c = \hat{a}\hat{v}_a + \hat{b}\hat{v}_o$.
• Training: The coefficients are optimized via a least-squares objective to approximate the ground-truth composition velocity ($x_c^1 - x_c^0$).
• Result: This creates an explicit, learnable rule for composing primitives in the embedding space, allowing the model to generalize to unseen pairs by recombining the learned transport directions.

C. Leakage-Guided Augmentation

Rather than treating feature entanglement (leakage between branches) as a failure, the authors exploit it as a signal.

• Strategy: The framework reuses "leaked" features from one branch (e.g., object features leaking into the attribute stream) as auxiliary supervision signals for the other branch.
• Implementation: The attribute flow is also trained to transport leaked object features toward the attribute text target, and vice versa.
• Benefit: This turns imperfect disentanglement into a constructive resource, enriching the velocity supervision and improving robustness.

3. Key Contributions

1. First Application of Flow Matching to CZSL: The paper introduces a paradigm shift from token-level manipulation to explicit embedding-space transport using flow matching.
2. Explicit Composition Rule: Replaces implicit token concatenation with a learnable Composer that explicitly fuses primitive velocity fields, addressing the generalization gap in unseen compositions.
3. Leakage-Guided Augmentation: A novel strategy that converts cross-branch feature leakage into useful supervision signals, improving performance without requiring perfect disentanglement.
4. Model-Agnostic Design: The framework is plug-and-play, operating purely in the representation space and compatible with various existing CZSL baselines (e.g., CSP, Troika).

4. Experimental Results

The method was evaluated on three standard benchmarks: MIT-States, UT-Zappos, and C-GQA, under both Closed-World and Open-World settings.

• Performance Gains: FlowComposer consistently improved performance when integrated into baselines like CSP and Troika.
  • On UT-Zappos, it achieved a +5.0% improvement in AUC over the Troika baseline.
  • On C-GQA, it improved AUC by +3.4% over Troika.
  • It achieved State-of-the-Art (SOTA) AUC scores on all three datasets in the Closed-World setting, even surpassing methods that utilize Large Language Models (LLMs) for external knowledge (e.g., PLID, LOGICZSL).
• Balanced Generalization: The method significantly reduced the bias toward seen compositions, maintaining higher unseen accuracy compared to baselines where unseen performance typically degrades during training.
• Ablation Studies:
  • Removing the Composer (using a direct regressor) or using a single flow for composition resulted in significant performance drops, validating the necessity of decomposing into primitives and explicitly composing velocities.
  • The Leakage-Guided Augmentation provided consistent gains in robustness (HM and AUC).
• Efficiency: The one-step transport scheme and lightweight Composer added negligible inference overhead (e.g., ~1.6ms increase per image on MIT-States).

5. Significance

This paper makes a significant contribution to the field of Zero-Shot Learning by demonstrating that Flow Matching is a natural and effective mechanism for compositional reasoning.

• Theoretical Insight: It establishes that the velocity fields in flow matching inherently support decomposition and recombination, aligning perfectly with the cognitive process of compositionality.
• Practical Impact: By turning the "flaw" of feature entanglement into a feature (via leakage-guided augmentation), it offers a robust solution that does not rely on perfect feature separation, which is often unattainable in practice.
• Future Direction: The success of FlowComposer suggests that explicit geometric operations in embedding spaces (transporting via velocity fields) may be superior to the current dominant paradigm of prompt engineering and token manipulation for compositional tasks.