Technical Summary: TAG: Tangential Amplifying Guidance for Hallucination-Resistant Sampling

1. Problem Statement

Diffusion models have achieved state-of-the-art performance in image generation but frequently suffer from hallucinations, defined as semantic inconsistencies or violations of the data distribution (e.g., anatomical impossibilities like extra fingers, or mixed-up objects). Recent research suggests these errors stem from mode interpolation, where sampling trajectories traverse low-density valleys between distinct modes of the data distribution, leading to attribute mismatches and structural inconsistencies.

Existing inference-time guidance methods, such as Classifier-Free Guidance (CFG) and its variants, typically rely on residual scaling between conditional and unconditional branches. While effective, these methods are largely geometry-unaware; they apply a single scalar magnification to the residual without accounting for the local directional structure of the data distribution. This can inadvertently distort the denoising trajectory, causing off-manifold drift or over-smoothing. Furthermore, many existing solutions require external signals, architectural modifications, or additional denoising evaluations, adding significant computational overhead.

2. Methodology: Tangential Amplifying Guidance (TAG)

The authors propose Tangential Amplifying Guidance (TAG), a training-free, architecture-agnostic, plug-and-play guidance method that operates purely on trajectory signals. TAG is grounded in the geometric decomposition of the sampling update step.

Core Concept

Drawing on Tweedie's identity and the Gaussian annulus theorem, the authors observe that the high-probability region of a corrupted distribution lies on a thin spherical shell (a noisy data manifold, $M_k$). The sampling update can be decomposed into:

1. Radial (Normal) Component: Moves the state along the radius, dictated by the noise schedule.
2. Tangential Component: Moves the state along the manifold, preserving the radius while refining semantic structure.

The authors hypothesize that the tangential component carries rich structural and semantic information, while the normal component is largely unstructured noise. Amplifying the tangential component steers the trajectory toward higher-density regions of the data manifold without altering the radial progression.

Algorithm

Given a base solver increment $\Delta_{k+1}$ (e.g., from DDIM) at timestep $k+1$:

1. Projection: The update is decomposed using projectors onto the radial direction ($P_{M_{k+1}}$) and the orthogonal tangential direction ($P^\perp_{M_{k+1}}$) relative to the current state $x_{k+1}$.
   • $P_{M_{k+1}} = \hat{x}_{k+1}\hat{x}_{k+1}^\top$
   • $P^\perp_{M_{k+1}} = I - P_{M_{k+1}}$
2. Amplification: The tangential component is amplified by a factor $\eta \ge 1$, while the radial component remains unchanged.
   • $x_k \leftarrow x_{k+1} + P_{M_{k+1}}\Delta_{k+1} + \eta P^\perp_{M_{k+1}}\Delta_{k+1}$

Theoretical Justification

The paper provides a first-order Taylor analysis showing that amplifying the tangential component monotonically increases the first-order log-likelihood gain. Specifically, Theorem 4.1 proves that increasing $\eta$ drives the sampling trajectory toward regions of higher model density, thereby reducing off-manifold artifacts (hallucinations) while preserving the radial calibration required for proper denoising.

Extension to CFG

For Classifier-Free Guidance, the authors address the incoherence between conditional and unconditional predictions. They propose aligning the conditional prediction with the tangential residual (the difference between conditional and unconditional scores projected onto the tangent space). This reconciliation reduces the mismatch that often degrades generation quality in standard CFG.

3. Key Contributions

• Geometric Insight: The paper establishes a concrete link between the intrinsic geometry of the score function and sample quality, proving that amplifying tangential components steers sampling toward the in-distribution manifold.
• TAG Algorithm: Introduction of a theoretically grounded, computationally lightweight, and architecture-agnostic algorithm that realizes this geometric principle. It requires no retraining, no additional network evaluations, and no architectural changes.
• Plug-and-Play Compatibility: TAG is designed to integrate seamlessly with existing solvers and guidance methods (e.g., PAG, SEG, SSG) as a post-processing step on the update vector.

4. Experimental Results

The authors evaluated TAG on various Stable Diffusion backbones (v1.5, v2.1, XL, 3) and modern models (Qwen-Image, Wan2.2 video) using the COCO 2014 dataset and T2I-CompBench.

• Quantitative Improvements:
  • Unconditional Generation: TAG consistently improved FID (e.g., SD v1.5: 58.41 $\to$ 46.20) and CMMD across all backbones without increasing the number of function evaluations (NFEs).
  • Conditional Generation: TAG improved sample fidelity and prompt alignment (ImageReward, CLIP-Score) across SD v1.5, v2.1, XL, and 3.
  • Hallucination Reduction: On T2I-CompBench (Spatial and Complex subsets), TAG improved compositional faithfulness metrics, indicating a reduction in structural hallucinations (e.g., incorrect object placement or relations) rather than just perceptual style shifts.
  • Video Generation: Applied to Wan2.2, TAG improved dynamic degree, imaging quality, and temporal consistency, reducing structural artifacts in complex scenes.
• Efficiency:
  • TAG adds negligible computational overhead (approx. +9.4% latency) compared to baselines, as it only involves lightweight vector projections.
  • In contrast, methods like PAG (Perturbed Attention Guidance) nearly double the latency (+112.8%) due to extra UNet evaluations.
• Compatibility: Stacking TAG on top of existing guidance methods (PAG, SEG, SSG) yielded further improvements in FID and human preference scores, demonstrating its orthogonality to residual-based guidance.

5. Significance and Claims

The paper claims that TAG offers a geometric view of hallucinations, identifying them as a failure to stay on the data manifold during mode interpolation. By isolating and amplifying the tangential component of the update, TAG effectively corrects the sampling trajectory toward high-density regions.

The authors emphasize that TAG is a complementary refinement rather than a replacement for existing solvers. It improves sample quality and reduces hallucinations with minimal computational cost, making it suitable for integration into existing pipelines. The paper modestly notes that while the theoretical analysis establishes local first-order likelihood gain, it does not fully characterize the behavior of the reverse trajectory over the entire sampling process. Future work is suggested to explore adaptive scaling and more complex local manifold geometry estimates.