💻 computer science

LISA: Likelihood Score Alignment for Visual-condition Controllable Generation

This paper introduces LISA, a regularization method that aligns side network features with an approximated likelihood score to improve training efficiency, convergence, and feature disentanglement in visual-condition controllable generation without incurring extra inference costs.

Original authors: Yanghao Wang, Hongxu Chen, Jiazhen Liu, Zhenqi He, Rui Liu, Zhen Wang, Long Chen

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yanghao Wang, Hongxu Chen, Jiazhen Liu, Zhenqi He, Rui Liu, Zhen Wang, Long Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a master painter (the Main Network) how to paint a picture based on a specific sketch (the Visual Condition, like a pose or a depth map).

Currently, the standard way to do this is the "Dual-Branch" method. You keep the master painter frozen (they already know how to paint beautiful landscapes, faces, and textures) and hire a small, fast assistant (the Side Network) to look at the sketch and whisper instructions to the painter. The assistant says things like, "Draw the arm here," or "Make the background darker."

The Problem:
While this works, the paper argues that the assistant is working a bit blindly. The assistant is just trying to guess what to whisper to make the final picture look right, but it doesn't have a clear map of exactly what information it is supposed to provide. It's like asking a tour guide to give directions without telling them, "Your only job is to explain the route; the driver already knows how to drive the car."

The Solution: LISA (Likelihood Score Alignment)
The authors propose a new training trick called LISA. They realized that the "whisper" the assistant gives is actually a specific mathematical concept called a "Likelihood Score." In simple terms, this score represents the difference between "what the picture would look like without any sketch" and "what the picture should look like with the sketch."

Here is how LISA works, using a creative analogy:

1. The "Ghost" Comparison

Imagine the Master Painter is sitting in a room.

Step A: The painter paints a picture based on their own imagination (no sketch). This is the "Unconditional Score."
Step B: The painter is then shown the sketch and paints a second version. This is the "Conditional Score."
Step C: LISA takes the difference between Picture A and Picture B. This difference is the "Likelihood Score." It is the exact mathematical "delta" or "gap" that the sketch creates.

2. The New Training Rule

Instead of just letting the assistant guess what to say, LISA gives the assistant a training target.

The assistant looks at the sketch.
A tiny, lightweight decoder (think of it as a translator) converts the assistant's internal thoughts into a "score."
LISA checks: "Does the assistant's score match the 'Ghost Comparison' (the difference between the two paintings)?"
If they don't match, the assistant gets a gentle nudge (a regularization loss) to adjust their thinking until they perfectly understand the "gap" the sketch creates.

3. The Result: A Super-Efficient Assistant

Because the assistant is now explicitly trained to understand this specific "gap" (the likelihood score), two amazing things happen:

Faster Learning: The assistant learns much faster because it isn't guessing; it has a clear target. The paper claims this can speed up training convergence by more than 2.78 times.
Better Control: The assistant becomes better at its specific job. It learns to handle complex tasks, like combining a pose sketch with a segmentation map, without getting confused. It's like the assistant becomes a specialist who knows exactly how to translate a sketch into painting instructions without mixing up the colors.

The Best Part: Zero Extra Cost

Usually, adding a new training rule makes things slower or requires more computer power. But LISA is clever:

During Training: It adds a tiny bit of work (like adding a 0.1% extra ingredient to a recipe).
During Inference (When you actually generate the image): The "translator" and the "Ghost Comparison" are thrown away. You only use the trained assistant and the master painter. The final result is exactly the same speed as the original method, but the picture is better and the assistant learned faster.

In Summary:
LISA is a training technique that teaches the "assistant" network exactly what its job is: to calculate the precise difference between a generic image and a condition-controlled image. By giving the assistant this clear, mathematical target, it learns faster, creates better images, and handles complex combinations of conditions more easily, all without slowing down the final result.

Technical Summary: LISA (Likelihood Score Alignment)

Problem Statement

The paper addresses the challenge of visual-condition controllable generation, where models must integrate visual modalities (e.g., pose, segmentation, depth maps) to guide image and video synthesis. The prevailing approach is the dual-branch paradigm, which freezes a pretrained diffusion or flow-matching model as a "main network" and trains a separate "side network" to encode visual conditions. These side features are then fused into the main network.

Despite its success, the authors argue that the role of the side network and its training efficiency remain underexplored. Existing methods often rely on external semantic encoders for regularization (e.g., representation alignment), which bounds performance to the chosen encoder and adds complexity. The core problem identified is that the side network implicitly learns to provide a "likelihood score" (the difference between conditional and unconditional scores) but lacks explicit supervision for this specific role during training.

Methodology: LISA

The authors propose LISA (Likelihood Score Alignment), a regularization method grounded in score-based generative modeling.

1. Theoretical Decomposition

The paper reinterprets the dual-branch paradigm through Bayes' rule. The conditional score $\nabla_{x_t} \log p_t(x_t|c)$ is decomposed into:

Unconditional Score: $\nabla_{x_t} \log p_t(x_t)$ , provided by the frozen main network.
Likelihood Score: $\nabla_{x_t} \log p_t(c|x_t)$ , representing the gap between the conditional and unconditional scores.

In the standard dual-branch setup, the side network implicitly learns to bridge this gap. LISA makes this role explicit.

2. Approximated Likelihood Score Construction

To supervise the side network, LISA constructs a target "approximated likelihood score" ( $\hat{\ell}_t$ ) without requiring external encoders:

Unconditional Target: Obtained by running the frozen main network without condition injection: $s_\theta(x_t, t) \approx \nabla_{x_t} \log p_t(x_t)$ .
Conditional Target: The standard denoising target $\nabla_{x_t} \log p_t(x_t|x_0)$ serves as a tractable proxy for the conditional score.
Likelihood Target: The approximated likelihood score is calculated as the difference:
$\hat{\ell}_t(c|x_t) = \nabla_{x_t} \log p_t(x_t|x_0) - s_\theta(x_t, t)$

3. Alignment Mechanism

Feature Extraction: A specific intermediate feature layer $r^k_\phi(x_t, c, t)$ is "hooked" from the side network.
Lightweight Decoder: This feature is fed into a small, trainable decoder $D_\psi$ (approx. 0.1% of the side network size) that projects the feature into the latent score space.
Regularization Loss: The decoder's output is aligned with the approximated likelihood score target using an $L_2$ distance.
Optimization: The total objective combines the standard diffusion loss ( $\mathcal{L}_{main}$ ) and the LISA regularization loss ( $\mathcal{L}_{LISA}$ ):
$\mathcal{L}_{total} = \mathcal{L}_{main} + \lambda \mathcal{L}_{LISA}$
Inference: The lightweight decoder is discarded after training. The side network is used exactly as in the original architecture, incurring zero extra inference cost.

Key Contributions

Novel Perspective: The paper analyzes the dual-branch paradigm by decomposing the roles of the main and side networks into unconditional and likelihood scores, respectively.
LISA Framework: It introduces an effective regularization technique that explicitly aligns side network features with an approximated likelihood score, eliminating the need for external semantic encoders.
Efficiency and Performance: The method accelerates training convergence and improves final synthesis quality (both perceptual and condition fidelity) with negligible additional training cost and no inference overhead.
Disentanglement: The explicit alignment encourages the side network's features to be more disentangled, naturally improving compositional control for multi-condition generation.

Experimental Results

The authors evaluated LISA across various tasks, architectures (U-Net, DiT), and models (Diffusion, Flow Matching):

Image Generation Tasks: Experiments on Pose, Segmentation, Depth, and Low-Resolution image generation using SDXL-1.0 and Stable Diffusion 2.1.
- Convergence: LISA consistently accelerated convergence. For example, in pose-conditioned generation, LISA improved PCK (Percentage of Correct Keypoints) from 19.38% to 83.02% at 10K iterations compared to vanilla ControlNet.
- Efficiency: LISA achieved better results with fewer iterations (e.g., 4K iterations with LISA outperformed 10K iterations of ControlNet in depth map generation).
- Comparison: LISA outperformed or matched ControlNet+REPA (which relies on external DINOv2 encoders) without requiring external models.
Video Generation: Applied to pose-guided video generation (ControlVideo). LISA significantly improved FVD (Frechet Video Distance) and PCK at 5K and 30K iterations.
Architecture Agnosticism: The method generalized to Diffusion Transformers (DiT) and Optimal Transport Flow Matching (OT-FM) models (e.g., Stable Diffusion 3).
Compositional Control: When combining independently trained pose and segmentation networks, LISA demonstrated superior compositional generation capabilities compared to the baseline, suggesting better feature disentanglement.
Overhead: The method added only ~0.1% parameters and ~0.2s latency per iteration during training, with zero cost during inference.

Significance and Claims

The paper claims that LISA provides a simple yet effective solution to the training inefficiency of the dual-branch paradigm. By explicitly decomposing the generative process and aligning the side network with the likelihood score, LISA acts as a strong prior supervision that:

Accelerates Training: Significantly reduces the number of iterations required to reach high-quality results.
Improves Quality: Enhances both the perceptual quality of the output and the fidelity to the visual conditions.
Simplifies Architecture: Removes the dependency on external semantic encoders required by previous alignment methods.
Enables Compositional Control: Naturally facilitates the combination of multiple visual conditions due to the improved disentanglement of learned features.

The authors conclude that LISA is an extensible and practical approach for visual-condition controllable generation, suitable for both image and video tasks across different model architectures.