FLAT: Feedforward Latent Triangle Splatting for Geometrically Accurate Scene Generation

Technical Summary: FLAT (Feedforward Latent Triangle Splatting)

Problem Statement
Generating explorable 3D scenes from a single image requires balancing strong generative priors with geometrically accurate representations suitable for downstream applications like simulation and standard graphics pipelines. While current video diffusion models offer high-quality generation and implicit multi-view geometric structure, existing feedforward decoders typically output volumetric 3D Gaussians. These Gaussian primitives lack well-defined surfaces, making them unsuitable for most graphics engines that rely on opaque surface representations (e.g., triangles or meshes). Furthermore, directly optimizing opaque triangle or mesh-based representations via feedforward prediction is challenging due to the non-differentiability of standard rendering and the sensitivity of flat primitives to orientation errors, which often leads to poor gradient flow and training instability.

Methodology
FLAT introduces a feedforward pipeline that decodes explicit, non-volumetric triangle splats directly from the latent space of a frozen video diffusion model in a single forward pass. The system operates as follows:

1. Pipeline Architecture: The method augments a frozen, camera-conditioned latent video diffusion model (specifically Uni3C based on Wan-2.1) with a lightweight scene decoder. Given a single RGB image and a target camera trajectory, the video model generates a denoised latent. The decoder then maps this latent directly to explicit scene parameters (triangle splats) without per-scene optimization.
2. Ray-Centered Triangle Parameterization: To address the instability of regressing flat primitives, FLAT employs a novel parameterization. Instead of predicting unconstrained 3D vertices or full world-space rotations, each decoder token predicts a triangle relative to a ray-centered local frame:
   • Position: Defined by a depth value along an anchor ray.
   • Shape: Derived from a canonical 2D equilateral triangle transformed by a constrained lower-triangular matrix (Cholesky-style), ensuring strictly positive area and avoiding degenerate solutions.
   • Orientation: Parametrized by residual tilt and spin angles around the ray-aligned frame, rather than a full 3D rotation (e.g., quaternion), which improves numerical stability.
3. Differentiable Rendering & Window Function: To enable gradient flow during training, FLAT utilizes a modified window function for differentiable triangle rendering. Unlike previous formulations that use a max reduction or hard boundaries, FLAT introduces a "product window function." This function extends the influence of the triangle beyond its exact boundary and routes gradients to all three vertices simultaneously, significantly improving gradient flow and training stability.
4. Camera Conditioning: Camera information is encoded as dense per-pixel ray embeddings (using a Ray-Point-Perpendicular-Component or RPPC parameterization) and injected into the decoder via zero-initialized convolutional blocks, ensuring alignment with visual latents.
5. Post-Processing: For compatibility with game engines, a lightweight test-time refinement step converts the predicted semi-opaque "triangle soup" into a fully opaque, coherent mesh. This involves refining depth and geometry, pushing opacity toward binary values, and stitching boundaries.

Key Contributions

• First Direct Decoding of Surface Primitives: The paper demonstrates, for the first time, that explicit, non-volumetric surface primitives (triangles) can be decoded directly from compressed video diffusion latents in a single forward pass.
• Stable Parameterization and Rendering: The authors introduce the specific technical ingredients required to make this feasible: a ray-centered local triangle parameterization with constrained shape and residual orientation, and a novel product window function that stabilizes training by improving gradient flow.
• Systematic Representation Analysis: The authors provide the first systematic comparison of 3D Gaussian Splatting (3DGS), 2D Gaussian Splatting (2DGS), and triangle splatting under identical training conditions. This analysis characterizes the trade-offs between rendering quality, geometric accuracy, and downstream mesh compatibility.

Results
Evaluated on the RealEstate10K and DL3DV datasets, FLAT achieves the following:

• Geometric Accuracy: The triangle-based model produces significantly more accurate geometry compared to Gaussian variants. It achieves a cosine similarity of 0.853 with ground-truth normals (Metric3D), outperforming 2DGS (0.587) and 3DGS (which yields near-random normals due to volumetric nature).
• Visual Quality: While 3DGS remains the strongest baseline for pure visual fidelity (PSNR), FLAT's triangle and 2DGS variants maintain competitive visual quality, with PSNR scores comparable to state-of-the-art methods like Lyra and Wonderland.
• Mesh Conversion: The proposed post-processing step converts the feedforward prediction into an opaque mesh that achieves a PSNR improvement of over 7 dB compared to meshes extracted from 3DGS/2DGS representations, demonstrating superior compatibility with standard graphics pipelines.

Significance
The paper claims that FLAT establishes a new paradigm for feedforward scene generation by showing that latent video generation models can serve as powerful priors not just for image synthesis, but for direct 3D scene prediction with explicit geometric structure. By successfully decoding non-volumetric primitives, the method bridges the gap between generative AI and standard graphics pipelines, enabling the creation of game-engine-ready assets without the computational overhead of per-scene optimization. The work highlights that with careful parameterization and rendering formulation, the stability issues previously associated with feedforward flat-primitive decoding can be overcome.