Controlla: Learning Controllability via Graph-Constrained Latent Geometry

Technical Summary: Learning Controllability via Graph-Constrained Latent Geometry

Problem Statement
Current controllable multimodal generation, particularly in diffusion models and diffusion transformers, is predominantly formulated as an inference-time conditioning problem. While methods utilizing prompts, guidance, or auxiliary modules (e.g., ControlNet, DreamBooth) can generate realistic outputs, they often lack an explicit mechanism to structure how semantic attributes evolve. This leads to three primary failure modes:

1. Identity Drift: Unintended changes to the subject's identity during semantic edits.
2. Inconsistent Cross-Modal Behavior: Misalignment between text, audio, and visual outputs.
3. Semantic Drift: Unpredictable or unstable intermediate states during attribute traversal.

Existing approaches treat controllability as an external constraint imposed on implicit latent spaces rather than a property of the latent geometry itself. Consequently, semantic changes are often imposed heuristically rather than emerging from structured representations.

Methodology: Controlla
The authors propose Controlla, a modular factorized-control framework that redefines controllability as a property of structured latent geometry. Instead of relying solely on inference-time conditioning, Controlla learns a graph-structured latent space where semantic attributes evolve along graph-consistent trajectories while preserving reference-grounded identity.

The framework operates through the following components:

1. Multimodal Factorization: Given inputs (target image, reference image, text, audio), a shared representation $z$ is learned and factorized into two distinct components:

   • Attribute Factor ($z_{attr}$): Captures semantic variations (e.g., emotion).
   • Identity Factor ($z_{id}$): Encodes the reference-grounded identity from the reference image.
   • Disentanglement is enforced via orthogonality constraints ($L_\perp$) and Lipschitz regularization ($L_L$) to ensure smooth transformations.

2. Graph Priors: Two offline-constructed graphs define the relational structure:

   • Emotion Graph ($G_e$): Captures semantic relationships between affective states in a unified valence–arousal space.
   • Identity Graph ($G_i$): Encodes relationships between reference identities to encourage compact intra-identity structure.

3. Graph-Constrained Optimal Transport (Graph-OT): The core alignment mechanism uses Optimal Transport to align the learned latent factors with the graph priors:

   • Fused Gromov–Wasserstein (FGW): Aligns the attribute space with the emotion graph, ensuring that relative distances between latent attribute embeddings match the semantic distances in $G_e$.
   • Gromov–Wasserstein (GW): Aligns the identity space with the identity graph $G_i$ to maintain consistent relational structure and prevent identity drift.

4. Control Mechanism: During generation, the identity factor $z_{id}$ is held fixed, while the attribute factor $z_{attr}$ is traversed along paths consistent with the emotion graph. This induces structured, predictable semantic changes without external heuristic conditioning.

Key Contributions

1. Formulation Shift: The paper proposes a formulation where controllability is encouraged by graph-consistent latent transformations rather than inference-time conditioning alone.
2. Controlla Framework: A modular architecture comprising a multimodal factorization encoder, a graph-OT alignment module, and factorized generator-conditioning adapters.
3. AffectHuman-43K Benchmark: The construction of a new, leakage-aware multimodal benchmark designed for reference-grounded affective control. Unlike naturally co-recorded datasets, it explicitly decouples visual identity (specified by a reference image) from affective control signals (text and audio). It features identity-disjoint splits with 0% cross-split reference-identity leakage.
4. Geometry-Aware Metrics: Introduction of metrics to evaluate trajectory consistency (Geodesic Consistency), latent disentanglement (Latent Disentanglement Score), and trajectory smoothness, moving beyond simple endpoint accuracy.

Results
Experiments on AffectHuman-43K and cross-dataset evaluations (AffectNet, CelebA-HQ) demonstrate that Controlla achieves consistent improvements over strong baselines, including StyleCLIP, ControlNet++, ICEdit, FLUX.1 Kontext, and DreamBooth hybrids.

• Performance: Controlla shows superior performance in emotion accuracy (Acc), trajectory smoothness (TS), and human preference (H).
• Geometry: The method achieves higher latent disentanglement (LDS) and lower geodesic inconsistency (GC), confirming that the learned traversal follows the intended graph structure.
• Identity Preservation: The framework maintains competitive or superior identity similarity (ID-Sim) and verification AUC, even under large semantic transformations.
• Ablation Studies: Removing graph constraints or multimodal grounding leads to significant degradation in controllability and identity consistency, validating that gains stem from the structured latent geometry rather than additional conditioning alone.

Significance and Claims
The paper claims a shift from heuristic, prompt-driven control toward geometry-driven representation learning. By modeling controllability as a property of global latent geometry, Controlla enables smooth, identity-preserving transformations along graph-consistent trajectories. The authors assert that their approach addresses the fundamental under-constrained nature of current multimodal editing by explicitly structuring the latent space.

The work is presented as a conditional, representation-level justification: if graph alignment, identity stability, factor separation, and local smoothness are optimized, semantic traversal produces bounded output variation. The paper does not claim to solve all generative challenges or guarantee global optimality, but rather demonstrates that enforcing graph structure on latent directions improves controllability, identity preservation, and cross-modal alignment in a measurable and reproducible way. Future work is noted as potentially extending this framework to video and broader graph-defined control factors.