Technical Summary: Coarse-to-Fine Compositional Diffusion for Long-Horizon Planning

Problem Statement

Diffusion models excel at generating structured data but are typically trained on short-horizon scales. Many real-world tasks, such as robotic navigation over long distances, panoramic image synthesis, or long video generation, require outputs that exceed the training scale of these models. Directly training models for such long-horizon outputs is often impractical due to data scarcity and the rapid growth of modeling costs with sequence length or spatial extent.

Compositional generation offers a solution by assembling overlapping local plans from a pretrained short-horizon prior into a single long-horizon output. However, standard compositional approaches primarily enforce local consistency (agreement between neighboring local plans at their overlaps) without explicitly specifying the global structure. Consequently, locally compatible plans may fail to form a plausible global route, task sequence, or temporal evolution. Existing methods attempt to improve global coherence by repeatedly propagating local consistency signals or adding inference-time optimization (e.g., iterative search or population-based pruning). While effective, these procedures become computationally expensive as the number of local plans or their dimensionality increases.

Methodology: Coarse-to-Fine Compositional Diffusion (CoFi)

The authors propose CoFi, an inference-time sampler that decouples global structure formation from local detail refinement. Unlike methods that rely on repeated message passing or expensive inner-loop optimization, CoFi operates in two distinct stages:

1. Coarse Global Scaffold Construction

The first stage aims to align local denoised estimates around a shared coarse structure to establish global coherence.

• Scaffold Alignment: For each denoising step $t$, the method computes a "Tweedie estimate" $\hat{X}^i_{0|t}$ for each local plan. To enforce global alignment, it solves a simplified optimization objective that balances local fidelity with a shared global component. This yields a closed-form solution where each local estimate is shrunk toward a shared scaffold mean $\bar{X}_{0|t}$:
  $$\bar{X}^i_{0|t} = (1 - \lambda_t) \bar{X}_{0|t} + \lambda_t \hat{X}^i_{0|t}$$
  Here, $\lambda_t$ is a schedule-dependent coefficient that controls the trade-off. Early in the diffusion process, $\lambda_t$ is small, forcing local plans to align strongly with the global scaffold. As $t$ decreases, $\lambda_t$ increases, allowing local details to emerge.
• Guided Update: The reverse diffusion step is updated using this aligned estimate, biased toward a globally coherent but locally coarse scaffold:
  $$X^i_{t-1} = \text{DDIM}_\theta(\bar{X}^i_{0|t}, t) + \sigma_t \nabla \hat{r}_t(X^i_t, X^{i+1}_t)$$
  The first term enforces global alignment, while the second term (local guidance) ensures boundary consistency between adjacent segments.

2. Structure-Preserving Refinement

The second stage restores the fine local details suppressed during the coarse stage while preserving the established global structure.

• Re-noising: The coarse global plan $Y^c_0$ is diffused forward to an intermediate noise level $t^*$ using a shared noise realization. This is critical; applying independent noise to local plans would break the alignment achieved in the first stage.
  $$Y^r_{t^*} = \sqrt{\bar{\alpha}_{t^*}} Y^c_0 + \sqrt{1 - \bar{\alpha}_{t^*}} \epsilon$$
• Refinement: Starting from $t^*$, the sampler performs a second reverse diffusion pass using the same pretrained local prior and local guidance. This stage recovers high-frequency local details and texture while the global scaffold anchors the overall structure.

Key Contributions

1. Decoupled Design: CoFi separates global structure formation from local detail refinement, avoiding the need for expensive iterative message passing or search-based optimization used in prior compositional methods.
2. Shared Scaffold Mechanism: By introducing a coarse alignment stage that projects local estimates onto a shared scaffold, the method ensures long-range coherence without requiring complex reward functions that are difficult to design for high-dimensional spaces.
3. Structure-Preserving Refinement: The use of a shared noise realization during the re-noising step ensures that the refined local plans remain compatible views of the same global object, preventing the "drift" often seen in long-horizon generation.
4. Efficiency: The method requires significantly fewer denoiser evaluations (NFE) compared to baselines that rely on inner iterations or population search.

Experimental Results

The authors evaluated CoFi on three domains: long-horizon robotic planning, panoramic image generation, and long video generation.

• Robotic Planning (OGBench): On the challenging "Giant" and "Scene-Play" tasks, CoFi achieved success rates of 96% (PointMaze-Giant) and 63% (Scene-Play), outperforming the previous state-of-the-art CDGS (82% and 51%, respectively). The scaffold stage was found to be crucial for resolving long-range inconsistencies in maze-like trajectories.
• Panoramic Image Generation: CoFi achieved the best scores across all metrics (Intra-LPIPS, Intra-Style loss, and CLIP score) compared to baselines like Multi-Diffusion, SyncDiffusion, and CDGS. It produced panoramas with more stable global styles and fewer abrupt transitions across patch boundaries.
• Long Video Generation: In generating 273-frame videos from 50-frame priors, CoFi improved subject consistency (94.11 vs. 91.67 for CDGS) and aesthetic quality, approaching the quality of native short-horizon generation. It effectively prevented subject drift over long temporal horizons.
• Computational Efficiency: CoFi demonstrated a 2–8× reduction in the number of function evaluations (NFE) compared to CDGS, while maintaining or improving sample quality. This is attributed to replacing iterative search with a single coarse-to-fine pass.

Significance and Claims

The paper claims that CoFi provides a simple and effective inference-time sampler that satisfies both local consistency and global coherence without the computational overhead of existing methods. By separating the global scaffold construction from local refinement, CoFi addresses the limitation where local consistency alone fails to guarantee a plausible global structure. The authors position this as a practical alternative for scaling short-horizon priors to long-horizon tasks in robotics, vision, and video generation, offering a favorable trade-off between performance and computational cost.

The authors modestly note limitations: the quality of the output is bounded by the expressiveness of the pretrained local prior, and the hyperparameters (scaffold schedule $\lambda_t$ and refinement depth $t^*$) require per-domain selection. Additionally, the current consensus mechanism applies uniform weighting to all local plans, which may be suboptimal if segments vary in structural importance.