Technical Summary: DéjàView – Looping Transformers for Multi-View 3D Reconstruction

Problem Statement

Traditional multi-view 3D reconstruction relies on iterative Structure-from-Motion (SfM) pipelines involving feature matching, pose estimation, and bundle adjustment. Recent advances have shifted toward feed-forward transformers that regress 3D geometry in a single pass. However, these state-of-the-art models have scaled to over a billion parameters, following the trend of increasing model capacity in computer vision.

The authors posit that the performance gains from depth in these large transformers may stem from implicit iterative refinement rather than unique representational capacity. Evidence suggests that contiguous transformer layers often behave like repeated applications of similar operations, and decoder layers progressively refine predictions. Current architectures pay for this "iteration" inefficiently by using unique parameters for every layer. The paper addresses the need for an architecture that makes this iterative process explicit, allowing for high-quality reconstruction with significantly fewer parameters and the ability to trade compute for accuracy at inference time.

Methodology: DéjàView

Core Architecture

DéjàView replaces the deep, feed-forward transformer stack with a single looped transformer block applied recurrently $K$ times.

1. Initialization: Per-view features are initialized using a pretrained DINOv2 encoder. Learnable register tokens and camera tokens are prepended to the token sequence.
2. Recurrent Loop: A single shared transformer block is applied $K$ times to the evolving state $z_k$. The block consists of two attention sub-blocks:
   • Frame Attention: Processes each view independently using 2D rotary position embeddings.
   • Global Attention: Operates over the joint sequence of all tokens across all views, enabling cross-view reasoning.
3. Time Conditioning: Unlike prior weight-tied transformers that condition on discrete step indices, DéjàView conditions the block on a continuous time interval $(t_k, t_{k+1})$. The update is formulated as:
   $$z_{k+1} = f_\theta(z_k, t_k, t_{k+1})$$
   This allows a single set of weights to cover a range of step counts $K$ without retraining.
4. Gated Updates: The block employs channel-wise scale vectors ($s_{attn}, s_{mlp}, s_{out}$) controlled by a zero-initialized MLP conditioned on the time interval. This design modulates the attention, MLP, and residual stream updates.

Training Strategy

• Variable-$K$ Training: During training, the number of refinement steps $K$ is sampled per batch from a Beta distribution scaled to $[K_{min}, K_{max}]$ (e.g., $[8, 16]$). This creates a single checkpoint that functions as a "compute knob."
• Loss Function: The model is supervised directly on predicted geometry using five loss terms: depth, rays, world points, and camera parameters. Predictions are normalized via inverse scale factors before computing $\ell_2$ and $\ell_1$ losses.
• Two-Stage Depth Training:
  • Stage 1: End-to-end training with a linear pixel-shuffle depth head and plain $\ell_2$ loss.
  • Stage 2: The depth head is swapped for a convolutional head (to eliminate patch-boundary artifacts), and the network is fine-tuned with a confidence-weighted loss while freezing other parameters.

Inference

At inference, the user selects a step count $K_{inf}$. The model runs the single shared block $K_{inf}$ times. If $K_{inf}$ is within the trained range $[K_{min}, K_{max}]$, the model maintains high accuracy; performance degrades if $K_{inf}$ is pushed significantly beyond the trained budget.

Key Observations on Dynamics

The authors characterize the recurrence as directional refinement. The state does not converge to a fixed point in feature space (the norm $\|z_k\|$ grows monotonically). Instead, the direction of the state vector stabilizes, with the cosine similarity to the final state approaching 1 and the relative update norm decaying significantly. This allows the decoder (which uses pre-norm) to absorb the norm growth while the decoded representation converges in direction.

Key Contributions

1. DéjàView Model: A looping transformer architecture that applies a single shared block recurrently to DINOv2-initialized features, conditioned on continuous time intervals.
2. Variable-$K$ Training Recipe: A method to train a single checkpoint that supports a range of inference step counts, exposing compute as a tunable knob without retraining.
3. State-of-the-Art Efficiency: The model achieves competitive or superior reconstruction quality across five benchmarks (indoor, outdoor, object-centric, driving) using a fraction of the parameters and comparable or lower compute than much larger feed-forward baselines.
4. Inductive Bias Validation: Experiments demonstrate that the shared-weight formulation outperforms an identical variant with independent per-step parameters, suggesting that explicit iteration with weight sharing is a stronger inductive bias for multi-view 3D reconstruction than parameter scaling.

Experimental Results

The paper evaluates DéjàView on five benchmarks: DTU, ETH3D, 7-Scenes, ScanNet++, and nuScenes.

• Accuracy: DéjàView matches or surpasses significantly larger models (e.g., VGGT-1B, Pi3, DA3-G) in pointmap accuracy (Relative $\ell_2$ and Inlier Ratio) and camera pose accuracy (AUC@3° and AUC@30°). For instance, on ScanNet++, it outperforms VGGT-Ω-1B by nearly 50 points in AUC@3°.
• Efficiency:
  • Parameters: DéjàView uses 117M parameters, compared to baselines ranging from 356M to over 1.2B.
  • Compute: It requires 75.9 TFLOPs for 24 views, comparable to or lower than baselines that require 2–10x more compute.
  • Memory: Peak inference memory is under 5 GiB on an A100, significantly lower than many baselines.
• Ablation Studies:
  • Weight sharing alone improves performance over decoupled blocks despite having 16x fewer parameters.
  • Time-conditioned residual gates and state gates provide monotonic improvements.
  • Variable-$K$ training matches fixed-$K$ performance within ~2-3% at the same inference budget, validating the single-checkpoint flexibility.

Significance and Claims

The paper claims that parameter scaling is not the only path forward for 3D reconstruction. By making the iterative refinement process explicit through a shared, time-conditioned loop, DéjàView demonstrates that:

1. Iteration can be explicit: The benefits of deep networks can be achieved by reusing a single block, avoiding the redundancy of unique layer parameters.
2. Compute is a knob: A single model can adapt to different inference budgets (latency/accuracy trade-offs) without retraining.
3. Shared weights are superior: The shared-weight formulation acts as a stronger inductive bias than independent parameters, suggesting that the "depth" in current SOTA models is largely a proxy for iterative refinement rather than unique representational capacity.

The authors note limitations: the model does not extrapolate well beyond its trained step range ($K_{max}$), and it does not explicitly handle dynamic scenes. However, the work establishes that high-quality multi-view reconstruction can be achieved with a fraction of the computational cost and model size of current feed-forward giants.