Technical Summary: DivRL – Disentangled Self-Similarity Rewards for Diverse Subject-Driven Generation

1. Problem Statement: The Identity-Diversity Paradox

Subject-driven image generation aims to synthesize novel images of a specific subject while preserving its visual identity across diverse contexts, poses, and styles. Despite the fidelity of recent diffusion-based models, a fundamental tension exists known as the "Identity-Diversity Paradox."

• The Conflict: Models that strongly enforce identity preservation often collapse into generating near-duplicate images of the reference, exhibiting rigid poses, viewpoints, and expressions. Conversely, methods that encourage structural diversity frequently suffer from "identity drift," where the generated subject loses its defining characteristics.
• Root Cause: This paradox arises because identity-defining visual attributes are entangled with spatial structure in the reference image. When models attempt to preserve identity, they inadvertently preserve the entangled structural information, reducing diversity. When they attempt to alter structure, they risk modifying identity-defining attributes.
• Limitations of Current Rewards: Existing identity-based rewards often encourage structural mimicry, as models can maximize identity scores by reproducing the exact spatial configuration of the reference image rather than exploring novel, identity-consistent variations.

2. Methodology: DivRL Framework

The authors propose DivRL, a post-training framework based on Reinforcement Learning (specifically Flow-GRPO) that jointly optimizes identity consistency and structural diversity. The core innovation lies in disentangling these objectives through specific reward designs and an "Explore-and-Suppress" optimization strategy.

2.1 Reward Design

The framework utilizes two distinct reward signals derived from disentangled visual features (specifically using the MTG visual features from a robust similarity model):

1. Visual Semantic Matching (VSM) for Identity Consistency:

   • VSM evaluates identity preservation by computing semantic and visual similarity matrices between the reference and generated images.
   • It identifies semantically consistent regions (where semantic similarity exceeds a threshold $\tau_s$) and assesses visual consistency within those regions.
   • A higher VSM score indicates better preservation of the subject's identity under varying conditions.

2. Negative Self-Similarity Measure (nSSM) for Structural Diversity:

   • Unlike global feature difference metrics, nSSM quantifies diversity by analyzing the internal relationships of visual features.
   • It computes Self-Similarity Matrices (SSM) for the visual feature grids of both the reference and generated images. These matrices capture the intrinsic spatial organization of object parts.
   • The nSSM is defined as $1 - \rho$, where $\rho$ is the Pearson correlation coefficient between the SSMs of the reference and generated images.
   • Logic: High correlation implies similar structural layouts (low diversity), while lower correlation corresponds to structurally diverse contexts (e.g., different poses or viewpoints). Maximizing nSSM encourages the model to explore structurally different configurations.
   • Stabilization: To prevent high-frequency textural artifacts (a form of reward hacking where the model mimics local correlations without changing global structure), the visual features are downsampled via $2\times2$ average pooling before computing the SSM.

2.2 Optimization Strategy: "Explore-and-Suppress"

Directly optimizing both objectives simultaneously via linear weighting leads to unstable training and reward hacking. Instead, DivRL employs a two-stage gated optimization strategy:

• Stage 1 (Explore): The model is trained using nSSM as the sole reward. This encourages the model to freely explore the latent space and generate structurally diverse configurations, breaking the rigidity of the pre-trained model.
• Stage 2 (Suppress): The model is trained using a gated reward formulation.
  • The VSM acts as a constraint (gate).
  • If a sample's VSM score is above a threshold $s$, it receives the nSSM reward.
  • If the VSM score falls below $s$, a quadratic hinge loss penalty is applied: $R = \text{nSSM} - \lambda(s - \text{VSM})^2$.
  • Effect: This converts identity preservation from a competing objective into a feasibility constraint. The model is free to explore diverse structures as long as they remain within the identity-consistent region. This decouples the optimization, allowing both nSSM and VSM to improve jointly without the gradients of one destroying the other.

3. Key Contributions

The paper outlines three primary contributions:

1. nSSM Metric: The introduction of the negative Self-Similarity Measure, a feature-space metric that quantifies structural diversity by analyzing the self-similarity correlations of disentangled visual features, rather than global feature differences.
2. Explore-and-Suppress Strategy: A novel optimization approach that decouples diversity exploration from identity preservation using a gated reward formulation, effectively transforming identity preservation into a constraint rather than a conflicting objective.
3. Empirical Validation: Demonstration on the DreamBench++ benchmark using the Flux-Kontext backbone, showing that the method improves structural diversity while maintaining competitive identity consistency. The authors note that the nSSM formulation and two-stage optimization are backbone-agnostic.

4. Experimental Results

The method was evaluated on the DreamBench++ benchmark (150 identities, 9 prompts each) against strong baselines including Flux-IP-Adapter, OmniGen, OmniGen2, UNO, and PaCo-RL.

• Quantitative Performance:

  • Balance: DivRL achieves a superior balance between identity consistency and structural diversity compared to baselines. While optimizing for VSM alone improves consistency but hurts diversity, and nSSM alone improves diversity but hurts consistency, the combined approach maintains high consistency (comparable to Flux-Kontext) while significantly boosting diversity metrics (MTG-nSSM, DINO-nSSM).
  • Prompt Following: Interestingly, the method improves prompt alignment. The authors attribute this to the relaxation of the "geometric rigidity" induced by pre-training; by encouraging structural diversity, the model adapts the subject more effectively to contextual prompt instructions.
  • Consistency vs. Diversity Trade-off: Plots of Consistency Ratio vs. Diversity-over-Consistency show that DivRL populates the high-utility region where both objectives coexist, outperforming linear combinations of rewards which fail to improve both simultaneously.

• Qualitative Observations:

  • Baselines like Flux-Kontext and PaCo-RL often produce rigid outputs or fail to follow prompt interactions (e.g., failing to place a subject in a specific scene).
  • Other models (e.g., OmniGen) produce diverse outputs but suffer from identity drift.
  • DivRL successfully generates images with varied poses, viewpoints, and expressions while preserving fine-grained visual details of the reference subject.

5. Significance and Limitations

Significance:
The paper identifies structural mimicry as a critical failure mode in existing subject-driven generation methods. By explicitly separating structural diversity from identity preservation, DivRL resolves the Identity-Diversity Paradox. The "Explore-and-Suppress" strategy demonstrates that treating identity as a feasibility constraint rather than a competing objective allows models to explore high-quality latent spaces where identity and novelty coexist.

Limitations:
The authors modestly acknowledge several constraints:

• Metric Scope: The diversity metric (nSSM) focuses on structural variations (pose, viewpoint) and may not fully capture higher-level semantic diversity or complex compositional changes.
• Computational Cost: The reliance on Reinforcement Learning for post-training introduces additional computational overhead compared to purely feed-forward methods.
• Input Constraints: The base model (Flux-Kontext) accepts a single reference image, which may limit effectiveness when the reference contains ambiguous identity cues or complex occlusions.

The paper concludes that future work could address these limitations through richer diversity metrics, more efficient optimization strategies, and multi-reference conditioning.