💻 computer science

JSGS: JPEG State-Guided Supervision for 3D Gaussian Splatting from Mixed-Quality Views

The paper proposes JSGS, a novel method that leverages JPEG quantization tables to construct view-specific observation operators and frequency-aware supervision, effectively mitigating compression artifacts and improving reconstruction quality for 3D Gaussian Splatting trained on mixed-quality JPEG images.

Original authors: Jinhua Cui, Anhong Wang, Kai Hu, Donghan Bu, Peihao Li, Tammam Tillo, Hao Jing, Shiao Xu

Published 2026-08-11

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Jinhua Cui, Anhong Wang, Kai Hu, Donghan Bu, Peihao Li, Tammam Tillo, Hao Jing, Shiao Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect 3D model of a city using only photographs. In the world of computer graphics, there's a super-fast, super-sharp technique called 3D Gaussian Splatting. Think of it like a digital artist throwing millions of tiny, fluffy, colored clouds (called Gaussians) into a virtual space. The computer figures out exactly where each cloud should float, how big it should be, and what color it needs to be so that when you look at the model from any angle, it looks exactly like a real photo.

Usually, this magic trick works best when every single photo you feed the computer is crystal clear and perfect. But in the real world, photos are rarely perfect. They get squished and saved as JPEGs to save space on your phone or hard drive. This "squishing" process, called compression, is like trying to pack a giant, fluffy pillow into a tiny suitcase. To make it fit, the suitcase (the JPEG file) has to crush the pillow, creating weird blocky patterns and fuzzy rings around edges. These are called "artifacts." If you try to build your 3D city using a mix of some perfect photos and some crushed, blocky JPEGs, the computer gets confused. It tries to fix the blocky parts, but in doing so, it messes up the perfect parts, resulting in a 3D model that looks glitchy and strange.

This is the puzzle that a team of researchers tackled in a new paper called JSGS. They asked a simple question: What if the computer didn't just try to ignore the bad photos, but actually understood exactly how they were crushed? Instead of treating a low-quality JPEG as just a "bad" picture, JSGS treats it like a puzzle with a specific set of rules. The paper suggests that by using the hidden "instruction manual" (called a quantization table) stored inside every JPEG file, the computer can learn exactly how that specific photo was distorted. By simulating that same distortion on its own 3D model before comparing it to the photo, the computer can stop fighting the artifacts and start learning from them. The result? A 3D model that looks incredibly sharp and realistic, even when built from a messy mix of high-quality and low-quality photos.

The Problem: The "Bad Photo" Confusion

Imagine you are a teacher trying to grade a student's drawing. You have a perfect reference photo of a bicycle. But the student only has access to a photocopy of that photo that was printed on a cheap, grainy printer. The photocopy has weird blocky squares around the wheels and fuzzy rings around the handlebars.

If you tell the student, "Draw the bike exactly like this photocopy," they will draw the blocks and the fuzz. If you then show them a different photo of the same bike that is crystal clear, and tell them to fix their drawing, they might get confused. Should they keep the blocks because the first photo had them? Should they remove them because the second photo doesn't?

This is exactly what happens in standard 3D Gaussian Splatting. The computer tries to build a single 3D world using many different photos. If one photo is a high-quality JPEG and another is a low-quality, blocky JPEG, the computer gets a headache. It tries to update the 3D "clouds" to match the blocky photo, which ruins the smooth parts of the model that other photos were trying to build. The paper argues that the standard approach is like trying to grade a student without looking at the specific instructions the printer used to make the photocopy.

The Solution: The "Magic Translator" (JSGS)

The authors of this paper, Jinhua Cui and his team, came up with a clever solution they call JSGS (JPEG State-Guided Supervision). Instead of ignoring the "crushing" that happened to the photos, JSGS uses the secret instructions hidden inside every JPEG file to guide the computer.

Here is how it works, broken down into three fun steps:

1. The "Fake-It-Til-You-Make-It" Translator (JPEG Observation Operator)
Every JPEG file has a hidden note inside it called a quantization table. This table is like a recipe card that says, "For this photo, we squished the bright colors this much and the dark colors that much."
JSGS takes the computer's 3D model, renders a picture from it, and then runs that picture through a "fake" JPEG process using the exact same recipe card found in the original photo. It's like the computer saying, "Okay, I'll draw the bike, but then I'll intentionally crush my drawing to look exactly like your low-quality photo." Now, when it compares its crushed drawing to your crushed photo, they match perfectly! The computer isn't fighting the blocks; it's learning to speak the language of the blocks.

2. The "Frequency Detective" (Domain-Matched DCT Supervision)
Photos are made of different types of details. Some are big, blurry shapes (low frequency), and some are tiny, sharp edges (high frequency). The "squishing" process usually messes up the middle-sized details the most.
JSGS acts like a detective who knows exactly which parts of the photo were squished the hardest. It uses the recipe card to weigh the errors. If the photo was squished heavily in a certain area, the computer knows to be gentle there. If the photo was clear, the computer knows to pay close attention. It's like a teacher who knows, "Oh, this student's handwriting is shaky in the middle of the page, so I'll focus my grading on the top and bottom where it's clear." This helps the computer learn the right details without getting distracted by the noise.

3. The "Cloud Manager" (Gaussian Controller)
Sometimes, the computer still sees a disagreement between the photo and the model. Maybe the photo has a weird blocky square that the model doesn't have.
JSGS has a special manager that looks at these disagreements. If the computer sees a tiny, fuzzy cloud (a Gaussian) that is causing a mess in a blocky area, the manager tells it to shrink or smooth out. It's like a traffic cop directing the tiny clouds away from the trouble spots so they don't cause a pile-up. This keeps the 3D model clean and prevents the "bad photo" from ruining the whole scene.

What They Found

The team tested their new method on seven different 3D scenes, ranging from a bicycle to a dinosaur, using three different ways of mixing high-quality and low-quality photos.

The Results: JSGS was a huge success. It produced 3D models that looked much more realistic than previous methods. Specifically, it had the lowest error rate (called LPIPS) in every single test, meaning the models looked the most like the real thing. It also had the highest structural similarity (SSIM), meaning the shapes and details were preserved best.
The Speed: One of the coolest things is that JSGS didn't slow things down. It could render the 3D scenes at about 150 frames per second (FPS). That is fast enough to make the 3D world feel like a real video game, even while it's doing all this complex math to fix the bad photos.
The Trade-off: While JSGS was the best at making the images look realistic and structurally correct, another method called FDS-GS was slightly better at measuring raw pixel brightness (PSNR). However, the authors suggest that looking "real" (low LPIPS) is often more important for human eyes than just matching pixel numbers perfectly.

The Bottom Line

The paper concludes that by understanding how a photo was compressed, rather than just treating it as a bad image, we can build much better 3D worlds. JSGS is like a translator that helps the computer understand the "accents" of different JPEG photos.

However, the authors are honest about the limits. Their method works great for photos that have been compressed once. It doesn't quite know how to handle photos that have been compressed, saved, and then compressed again (double compression), because the second compression overwrites the original instructions. Also, they tested this on photos they made themselves in a lab, so they aren't 100% sure yet how it will handle every single photo taken by a random camera in the real world.

But for now, JSGS offers a playful and powerful new way to turn a messy pile of photos into a stunning, high-speed 3D experience. It proves that sometimes, to build something perfect, you have to understand the imperfections.

Technical Summary: JSGS – JPEG State-Guided Supervision for 3D Gaussian Splatting from Mixed-Quality Views

Problem Statement

Standard 3D Gaussian Splatting (3DGS) operates under the assumption that every input training image faithfully samples scene radiance. However, in practical image collections, input images often exist in mixed JPEG qualities. This violates the standard assumption because compression-induced artifacts—specifically blocking and ringing—vary in strength and spatial support across different views. Since 3DGS jointly updates a shared set of geometry, opacity, and appearance parameters based on all views, these structured residuals from low-quality views can corrupt the shared 3D representation, leading to degradation in reconstruction quality.

Existing approaches to degraded 3DGS typically treat compression as a generic degradation type or use scalar quality labels to weight frames. They generally fail to leverage the specific per-view quantization tables stored in JPEG file headers, which contain precise information about the quantization steps applied at different block-DCT coordinates. Consequently, how to condition the supervision of shared Gaussians using this specific per-view JPEG state remains an open question.

Methodology

The authors propose JSGS (JPEG State-Guided Supervision), a framework that utilizes the luminance and chrominance quantization tables ( $T^Y_i, T^C_i$ ) parsed from each JPEG file header to guide the optimization process. The method operates on three levels: observation matching, frequency-domain supervision, and Gaussian regularization.

1. JPEG Observation Operator (JO)

To ensure domain-matched comparison, JSGS introduces a differentiable JPEG Observation Operator. For each view $i$ , the rendered image $\hat{I}_i$ is passed through a differentiable JPEG encoding and decoding process using the specific quantization tables $T^Y_i$ and $T^C_i$ associated with that view.

Process: The operator applies a fixed 4:2:0 transform, performs element-wise soft rounding on quantized DCT coefficients, and reconstructs the image.
Loss: The output $\hat{I}^J_i$ is compared with the decoded input image $I_i$ using a standard photometric loss ( $\ell_{obs}$ ). This ensures the renderer learns to produce images that, when compressed by the specific view's JPEG state, match the observed input.

2. Domain-Matched DCT Supervision (DS)

JSGS performs supervision in the frequency domain to distinguish between reliable and unreliable DCT coefficients based on the quantization tables.

Frequency Bands: The method separates DCT residuals into a low-frequency (LF) band (anchoring coarse structure) and a middle-frequency (MF) band.
Weighting: A continuous reliability weight $\omega_i(u, v)$ is derived from the luminance quantization step $T^Y_i(u, v)$ . Smaller quantization steps (higher quality) yield larger weights.
Loss Calculation:
- Low Frequency ( $L_{low}$ ): Uses a fixed mask to anchor the coarse structure.
- Middle Frequency ( $L_{mid}$ ): Applies the continuous weights $\omega_i(u, v)$ to the residuals within a fixed MF band. This redistributes supervision, prioritizing DCT coordinates with higher reliability (smaller quantization steps) and down-weighting those prone to heavy compression artifacts.

3. Gaussian Controller (GC)

The weighted disagreement in the middle frequency band is projected back to the 3D Gaussian primitives to regularize the representation.

Block Disagreement: The weighted MF error is aggregated per 8x8 JPEG block to compute a block-level disagreement score $r^b_i$ .
Regularization: This score is projected to visible Gaussians. A specific loss term ( $L_{fro}$ ) regularizes small primitives with high opacity in regions where disagreement remains high. This prevents the optimization from overfitting to compression artifacts in specific views by penalizing primitives that contribute to inconsistent regions.

Overall Objective

The final training objective combines the standard photometric loss, the JPEG observation loss, the low and middle frequency DCT losses, and the Gaussian regularization loss:
$L_i = \lambda_{base}\ell_{ph} + \lambda_{obs}L_{obs} + \lambda_{low}L_{low} + \lambda_{mid}L_{mid} + \lambda_{fro}L_{fro}$

Key Contributions

Formulation of Mixed JPEG Quality: The paper formulates mixed JPEG quality as a problem of unequal observation fidelity in 3DGS, where multiple views with varying compression states jointly update a shared representation.
JPEG Observation Operator & Complementary Losses: The authors design a differentiable operator to match the rendering domain to the specific JPEG state of each view. They introduce complementary losses in the low and middle frequency bands, using continuous quantization-table weights to redistribute residual contributions.
Gaussian Regularization via Disagreement: The method utilizes the resulting block disagreement to guide a Gaussian Controller, which regularizes small primitives in regions of high disagreement, effectively mitigating the influence of low-quality views on the shared geometry.

Experimental Results

The authors evaluated JSGS on seven scenes (four from LLFF, three from Mip-NeRF 360) under three mixed-quality schedules (Uniform, Gaussian, Descending) and fixed-quality controls ( $q=30, 50$ ).

Perceptual Quality: JSGS achieved the lowest mean LPIPS in all 21 combinations of scenes and schedules, indicating superior perceptual fidelity compared to baselines like Gsplat, 3DGS, SRGS, and FDS-GS.
Structural Similarity: JSGS achieved the highest mean SSIM across all three mixed-quality schedules.
PSNR: While FDS-GS achieved slightly higher mean PSNR, JSGS prioritized structural and perceptual metrics, which the authors argue are more relevant for handling compression artifacts.
Efficiency: JSGS maintains a rendering speed of approximately 150 FPS, matching the baseline Gsplat and significantly outperforming other robust methods (which run at ~80 FPS). Training time is approximately 10 minutes, faster than standard 3DGS and other robust variants.
Ablation: Removing any component (JO, DS, or GC) resulted in performance degradation, confirming the complementary nature of observation matching, frequency weighting, and regularization.

Significance and Limitations

The paper claims that JSGS successfully addresses the challenge of mixed-quality views by leveraging the specific quantization information inherent in JPEG files, rather than treating compression as a generic noise source. By aligning the observation domain and reweighting frequency supervision, JSGS achieves state-of-the-art perceptual quality and structural similarity without sacrificing rendering speed.

Limitations:

Double Compression: The method does not account for double JPEG compression, as the final file header only reflects the last encoding stage.
Data Source: The experiments rely on controlled JPEG encoding of existing scene datasets. The authors note that the method has not yet been tested on data generated by diverse real-world communication devices, encoders, and transmission links.