💻 computer science

TRACE-GS: On-Policy Trajectory Distillation with Privileged Geometric Conditioning for Sparse-View 3DGS Restoration

TRACE-GS introduces an on-policy trajectory distillation framework that leverages privileged geometric conditioning during training to align denoising directions and correct cumulative errors in sparse-view 3D Gaussian Splatting restoration, achieving superior generalization without requiring additional views at deployment.

Original authors: Linlian Jiang, Yuchen Xi, Sadman Rakib Pinon, Ruigang Yang, Yang Wang, Xinxin Zuo

Published 2026-08-12

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Linlian Jiang, Yuchen Xi, Sadman Rakib Pinon, Ruigang Yang, Yang Wang, Xinxin Zuo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect 3D model of a room, but you only have a handful of blurry photos taken from a few angles. In the world of computer vision, this is a classic puzzle called "sparse-view 3D reconstruction." The goal is to fill in all the missing pieces—the walls, the furniture, the shadows—so you can walk around inside the model and see it from any angle. Recently, a technique called 3D Gaussian Splatting (3DGS) has become a superstar for this job because it can render these 3D scenes incredibly fast, almost like a video game. However, when you don't have enough photos, the computer gets confused. It might invent floating blobs of color, blur the edges, or make the geometry look like melted wax. To fix this, scientists have started using "diffusion models," which are like AI artists trained on millions of images. These AIs can guess what the missing parts should look like, but they often make mistakes when they have to guess step-by-step without enough clues.

Enter TRACE-GS, a new method that acts like a master coach training a student athlete. The researchers realized that while AI models are getting smarter, they are still making a specific kind of mistake: they are being taught to guess based on random, isolated snapshots, but then they have to make a long, connected journey to create the final image. It's like teaching a student to solve a math problem by looking at the answer key for random, unrelated problems, and then expecting them to solve a long chain of equations without help. The student gets lost along the way. TRACE-GS solves this by changing the training game entirely. Instead of just giving the student random clues, it uses a "privileged" teacher who has access to a full, high-quality 3D model during training (because the training data has more photos than the real-world problem does). This teacher guides the student along the exact path the student will take when it's working alone, correcting its course at every single step. The result is a system that can take a few blurry photos and turn them into a sharp, detailed 3D scene, even in difficult situations where other methods fail.

The Problem: Getting Lost on the Way

When you try to reconstruct a 3D scene from just a few photos (say, 3, 6, or 9), the computer has to fill in huge gaps. Existing methods try to use AI "diffusion" models to guess the missing details. Think of this like a game of "telephone" where you are trying to draw a picture based on a whispered description. Most current AI methods are trained by looking at the final picture and asking, "What does this look like if we add some noise?" They learn to remove that noise in isolation.

But here is the catch: when the AI actually tries to build the image in the real world, it doesn't start from a clean slate and jump to the finish. It takes a long, step-by-step journey, making a guess, then using that guess to make the next guess. If the first guess is slightly wrong (which it often is, because the photos are sparse), that error gets passed down the line. The next step builds on a mistake, and the next builds on that mistake. By the time the AI finishes, the errors have piled up, and the image is distorted. The paper calls this an "off-policy mismatch." It's like a GPS that was trained on perfect maps but is trying to navigate a city with missing streets; it keeps recalculating based on bad data, and you end up driving in circles.

The Solution: The Privileged Coach

The authors of TRACE-GS realized that the training data often contains more photos than the real-world problem does. During training, they have access to a "dense" set of views (many photos), but at deployment (when the user actually uses the tool), they only have the "sparse" set (few photos).

They created a system with two characters:

The Student: This is the model that will be used in the real world. It only sees the few, sparse photos.
The Teacher: This is a copy of the student, but during training, it gets to see the rich, dense set of photos. This is the "privileged information."

The magic happens in how they train. Instead of just comparing the student's random guesses to a target, the researchers let the student take its own journey (its "rollout") to generate an image. At every single step of this journey, the Teacher—who has the advantage of seeing the full, high-quality geometry—steps in to say, "No, that's not right. Here is the correct direction to go next."

Crucially, the Teacher doesn't take its own journey. It just looks at where the Student is right now and gives a correction. This ensures the Student learns to navigate the specific path it will actually travel, rather than just learning to fix random, isolated errors. This is called "on-policy trajectory distillation."

The Results: Sharper Details, Fewer Ghosts

The paper tested this method on various datasets, including real-world scenes and synthetic ones. They compared TRACE-GS against other top methods like Difix3D+, GenFusion, and GSFixer.

The results were impressive. In tests with only 3 input views (the hardest scenario), TRACE-GS consistently produced better images than the competition. For example, on the DL3DV-Benchmark dataset, it achieved a PSNR (a score measuring image quality) of 16.92 with 3 views, beating the next best method (GSFixer) which scored 16.21. As the number of views increased to 6 and 9, TRACE-GS continued to lead or tie for the top spot.

The visual difference is striking. In scenes with reflective surfaces (like a shiny store window) or complex textures, other methods often produced blurry blobs or "ghosts" (floating artifacts). TRACE-GS, however, managed to restore sharp details. In one example, while other methods blurred the text on a sign or the contours of a chair, TRACE-GS kept the edges crisp and the structure logical.

The researchers also ran "ablation studies" (experiments where they removed parts of the system to see what happened). They found that:

Privileged Geometry Matters: If the Teacher didn't have access to the extra photos (the privileged geometry), the system performed worse than the baseline. The extra views were essential for providing the correct guidance.
On-Policy Matters: If they trained the system using the old "off-policy" method (random steps instead of the student's actual path), the quality dropped. This proved that correcting the student along its own journey is the key to success.

Why It Works

The core idea is that the "gap" between what the student sees (sparse views) and what the teacher sees (dense views) is used as a teaching tool. The teacher knows what the scene should look like because it has more information. By guiding the student step-by-step along the student's own path, the teacher prevents the small errors from turning into big disasters.

Once the training is done, the Teacher is discarded. The Student, now a master of its own path, is deployed. It takes the sparse photos, runs its own denoising journey, and produces a high-quality 3D scene that can be refined further. The paper suggests this approach is a significant step forward for sparse-view 3D restoration, offering a way to get high-quality results without needing expensive, dense camera setups. It turns a difficult, error-prone guessing game into a guided tour, ensuring that even with limited information, the final destination is clear and sharp.

Technical Summary: TRACE-GS

Problem Statement

Sparse-view 3D Gaussian Splatting (3DGS) reconstruction suffers from severe under-constrained geometry when input views are limited (e.g., 3–9 images). This leads to artifacts such as floaters, fragmented geometry, and blurred textures. While recent approaches utilize pretrained diffusion models as generative priors to restore novel-view renderings for 3DGS refinement, they face a fundamental training-inference mismatch:

Off-Policy State Mismatch: Existing methods supervise the model at independently forward-noised states derived from clean targets. However, during inference, the model follows a sequential rollout driven by its own denoising predictions. Consequently, the model encounters self-generated states not covered by the training objective.
Compounding Geometry Bias: In sparse-view settings, the initial geometry is under-constrained. Artifacts in early denoising steps bias subsequent directions, causing deviations to compound along the rollout. Off-policy supervision fails to correct these deviations because it does not supervise the actual states visited during inference.

Methodology: TRACE-GS

TRACE-GS (On-Policy Trajectory Distillation with Privileged Geometric Conditioning) addresses these issues by formulating sparse-view restoration as a Learning Using Privileged Information (LUPI) problem. The framework employs a teacher-student architecture where the teacher has access to "privileged" geometric information available only during training.

1. Geometry-Asymmetric LUPI Formulation

Setup: For a given scene, two 3DGS reconstructions are generated: one from the sparse input views ( $V_s$ ) and one from all available training views ( $V_d$ , where $V_s \subset V_d$ ).
Privileged Information: The dense-view reconstruction ( $R_d$ ) provides richer, more reliable geometry and fewer artifacts compared to the sparse-view reconstruction ( $R_s$ ). This geometric fidelity gap constitutes the privileged information.
Architecture: A single velocity predictor backbone ( $v_\theta$ $v_{θ}$ ) is shared between a Teacher and a Student.
- Teacher: Conditioned on the dense-view renderings ( $R_d$ ). It acts as a privileged guide.
- Student: Conditioned on the sparse-view renderings ( $R_s$ ). It is the deployable model.
- Both share the same frozen backbone and trainable LoRA adapters (initialized identically), differing only in their conditioning inputs.

2. Training Strategy

The training process consists of two main phases to align the student with the privileged teacher:

Privileged Direction Alignment (Warm-up):
At independently sampled forward-noised states, the student's prediction ( $v_S$ ) is aligned with the teacher's prediction ( $v_T$ ). This reduces the initial direction bias caused by the sparse geometry gap.
$\mathcal{L}_{align} = \mathbb{E}[\|v_S - \text{sg}[v_T]\|_2^2]$
(where sg denotes stop-gradient).
On-Policy Trajectory Distillation:
Unlike prior methods, TRACE-GS supervises the model along the student's own rollout trajectory ( $Z_S$ ).
- Student Rollout: The student generates a trajectory of states ( $z^S_{t'}$ ) starting from noise, using its own predictions.
- Teacher Query: At each state visited by the student, the frozen teacher is queried using the same state and the privileged dense geometry ( $R_d$ ) to generate a target velocity ( $u^T_i$ ). The teacher does not perform its own rollout.
- Trajectory Loss: The student's velocity predictions are aligned with the teacher's targets at these visited states:
  $\mathcal{L}_{traj} = \mathbb{E}\left[\frac{1}{N}\sum_{i=1}^N \|u^S_i - \text{sg}[u^T_i]\|_2^2\right]$
On-Policy Retrieval Alignment:
To ensure the student utilizes cross-view evidence correctly beyond just velocity direction, the framework aligns the cross-attention retrieval responses of the student and teacher at visited states.
$\mathcal{L}_{ret} = \mathbb{E}\left[\frac{1}{N}\sum_{i=1}^N \|\bar{F}^S_i - \text{sg}[\bar{F}^T_i]\|_2^2\right]$
where $\bar{F}$ represents normalized retrieval responses.

3. Deployment

At deployment, only the sparse-conditioned student is retained. It restores artifact-laden renderings from the sparse-view 3DGS. These restored renderings serve as pseudo-observations to refine the 3DGS representation iteratively. No privileged geometry or teacher computation is required during inference.

Key Contributions

Identification of Coupled Limitations: The authors identify that off-policy mismatch and geometry-induced bias compound in sparse-view 3DGS restoration, necessitating supervision at the actual states visited during inference.
Geometry-Asymmetric LUPI: The paper formulates a novel LUPI setting where knowledge is transferred from a teacher with richer geometric conditioning to an architecturally identical student with sparse conditioning.
First On-Policy Supervision for Sparse 3DGS: TRACE-GS is the first method to derive on-policy supervision from privileged dense-view geometry, aligning denoising directions and retrieval responses along the student's rollout without requiring a teacher rollout.

Experimental Results

The method was evaluated on three datasets: DL3DV-Benchmark (in-domain), Mip-NeRF 360, and NeRF-Busters (out-of-domain), across 3, 6, and 9 input views.

Quantitative Performance: TRACE-GS achieved state-of-the-art performance in PSNR, SSIM, and LPIPS across all datasets and sparsity levels.
- On DL3DV-Benchmark, it outperformed the previous best (GSFixer) by 0.47 dB in PSNR (3-view setting).
- On out-of-domain datasets, it consistently led in all metrics, particularly in the challenging 3-view setting (e.g., +0.73 dB over GSFixer on Mip-NeRF 360).
Qualitative Improvements: Visual comparisons show TRACE-GS recovers sharper details and coherent structures (e.g., text, reflections, object contours) where baselines produce blurring or geometric fragmentation.
Ablation Studies:
- Privileged Geometry: Increasing the number of views used for the teacher (from 3 to all available) monotonically improved performance, confirming the value of the geometric gap.
- On-Policy vs. Off-Policy: Replacing on-policy supervision with off-policy (independently sampled states) resulted in lower performance, particularly in later denoising steps where errors accumulate. This validates the necessity of supervising visited states.
- Retrieval Alignment: Adding retrieval alignment provided consistent gains, ensuring better cross-view consistency.

Significance and Claims

The paper claims that TRACE-GS represents a fundamental shift in addressing sparse-view 3DGS restoration. By moving away from pursuing increasingly sophisticated restoration architectures, it addresses the core limitation of training-inference mismatch in diffusion-based priors.

The significance lies in the Learning Using Privileged Information (LUPI) framework applied to 3DGS:

It leverages training-time geometric advantages (dense views) to guide a model that operates solely on sparse views at deployment.
It ensures that the guidance is applied exactly where the model needs it most: along its own inference trajectory.
The method is presented as a "drop-in" refinement stage that requires no additional views or computational overhead during deployment, achieving strong generalization across diverse datasets and sparsity levels.