💻 computer science

NeR-SC: Adapting Neural Video Representation to Screen Content

The paper proposes NeR-SC, a neural video representation framework specifically designed for screen content that introduces a learnable color palette, a multi-gate dense fusion module, and an embedding-level frame skip strategy to significantly outperform existing neural methods and traditional codecs like H.264/H.265 in terms of quality and decoding efficiency.

Original authors: Ruohan Shi, Jiaoyan Zhao, Haogang Feng

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruohan Shi, Jiaoyan Zhao, Haogang Feng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a video of a computer screen (like a Zoom call, a coding session, or a cloud game) over the internet.

The Problem:
Standard video compression tools (like the ones used for movies on Netflix) are designed for "natural" video. They expect smooth gradients, like a sunset or a person's skin. But computer screens are different. They are full of sharp text, solid blocks of color, and things that stay exactly the same for minutes at a time. Trying to compress a computer screen with tools made for movies is like trying to pack a box of Lego bricks using a vacuum sealer meant for a bag of marshmallows. It works, but it's inefficient and leaves a lot of space wasted.

The Solution: NeR-SC
The authors of this paper built a new tool called NeR-SC (Neural Representation for Screen Content). Think of it as a custom-tailored suit for computer screen videos, rather than a "one-size-fits-all" outfit.

Here is how they made it work, using three clever tricks:

1. The "Color Palette" Trick

The Analogy: Imagine you are painting a picture of a computer screen. A standard painter tries to mix millions of tiny shades of blue to get the right color for a button. But a computer screen only uses a specific set of colors (like a limited box of crayons).
What NeR-SC does: Instead of guessing every shade, NeR-SC learns a specific "box of crayons" (a Learnable Color Palette) just for that video. When it needs to draw a blue button, it doesn't mix paint; it just picks the exact blue crayon from its box. This makes the description of the image much shorter and sharper because it stops trying to invent colors that don't exist on the screen.

2. The "Team Huddle" Trick

The Analogy: Imagine a construction crew building a house. In the old method (previous AI models), the workers passed instructions down a line: Worker A tells Worker B, who tells Worker C. If Worker A makes a mistake, Worker C might not know about it until it's too late.
What NeR-SC does: NeR-SC uses a Multi-Gate Dense Fusion module. Instead of a line, it's like a team huddle. All the workers (different layers of the AI) talk to each other at the same time. They share information instantly across the whole team. This ensures that the sharp edges of text and the smooth backgrounds get built perfectly together, without any blurry corners.

3. The "Skip the Boring Part" Trick

The Analogy: Imagine you are watching a video of a static whiteboard with a teacher writing on it. For 90% of the video, the board doesn't move. A standard player would try to redraw the whole whiteboard for every single frame, even though nothing changed.
What NeR-SC does: It uses an Embedding-Level Skip Strategy. It takes a tiny "fingerprint" of the current frame and compares it to the last one. If the fingerprints match (meaning the screen hasn't changed), it simply says, "I know what this looks like; I'll just reuse the last picture." It skips the heavy lifting of redrawing the image. This happens instantly and costs no extra effort to learn.

The Results

The authors tested this new system on real screen-content videos (like online classes and remote desktops).

Quality: It produced clearer, sharper images than previous AI methods and even beat traditional video standards (like H.264 and H.265) when the file size was kept small.
Speed: Because of the "Skip the Boring Part" trick, the video can be decoded in real-time (about 61 frames per second) without losing any quality.

In Summary:
NeR-SC is a smarter way to compress computer screen videos. It stops trying to force screen videos to look like movies. Instead, it learns the specific rules of screens (limited colors, sharp edges, and static moments) and uses those rules to create smaller, higher-quality files that play back instantly.

Technical Summary: NeR-SC – Adapting Neural Video Representation to Screen Content

1. Problem Statement

Implicit neural representations (INR) have emerged as a competitive paradigm for video compression, particularly for natural scenes. However, existing methods (e.g., NeRV, SNeRV) struggle with screen content video (common in remote desktop, online education, and cloud gaming). Screen content exhibits distinct statistical properties that natural video codecs fail to exploit:

Discrete Color Palettes: Sharp edges and UI elements use a limited set of colors rather than continuous gradients.
Strong Temporal Redundancy: Large portions of the frame remain static across sequences, with changes confined to small regions (e.g., cursors).
Multi-scale Structure: Text and graphical elements require precise handling of sharp transitions.

Current neural video representation methods regress continuous RGB values, utilize sequential feature fusion that may be suboptimal for graphical content, and decode every frame independently, ignoring temporal redundancy. This results in a performance gap compared to traditional codecs (like H.264/H.265) and specialized deep learning codecs (like DSCVC) for screen content.

2. Methodology

The authors propose NeR-SC (Neural Representation for Screen Content), a framework built upon the SNeRV backbone. The architecture decomposes input frames using a single-level 2D Haar wavelet transform. The low-frequency (LL) sub-band is processed by a heavy ConvNeXt encoder-decoder, while high-frequency (HF) sub-bands are reconstructed via lightweight convolutional heads.

NeR-SC introduces three specific innovations to adapt this backbone for screen content:

A. Learnable Color Palette

Recognizing that screen content relies on discrete colors, the authors replace unconstrained RGB regression for the LL sub-band with a per-pixel classification approach.

A trainable palette matrix $P \in \mathbb{R}^{K \times 3}$ (where $K=64$ ) is learned.
The decoder outputs logits for $K$ color indices, which are converted to probabilities via a temperature-scaled softmax.
The reconstructed LL sub-band is a convex combination of these palette entries.
High-frequency sub-bands remain unconstrained to preserve edge fidelity and gradients.

B. Multi-Gate Dense Fusion (MGF) Module

The original SNeRV uses a sequential pair-wise fusion strategy (MRF) which restricts information flow to adjacent stages. NeR-SC replaces this with dense, simultaneous fusion of the last three decoder stages ( $F_3, F_4, F_5$ ).

Features are projected to a common dimension and aligned via bilinear interpolation.
A residual block fuses the concatenated features.
A Squeeze-and-Excitation (SE) block applies per-channel attention gating in a residual manner, allowing the network to learn the importance of features across stages dynamically.

C. Embedding-Level Frame Skip Strategy

To address strong temporal redundancy, the authors introduce a mechanism to bypass the decoder for static frames.

Since the decoding function is deterministic, frames with similar latent embeddings yield near-identical reconstructions.
During decoding, the current frame's latent embedding $z_i$ is compared to the last decoded frame's embedding $z_k$ using a normalized L1 distance.
If the distance falls below a threshold $\tau$ (default 0.005), the decoder is skipped, and the previous frame is reused.
This strategy incurs zero training overhead and requires storing only a single scalar threshold.

3. Key Contributions

Learnable Color Palette: Converts continuous regression to discrete classification for the low-frequency sub-band, effectively modeling the limited color palettes of screen content while maintaining expressiveness for high-frequency details.
Multi-Gate Dense Fusion (MGF): Replaces sequential fusion with dense, attention-gated cross-stage interaction, improving multi-scale feature integration and reducing parameter count.
Embedding-Level Skip Strategy: A zero-overhead mechanism that detects temporal redundancy via compact latent embeddings, enabling real-time decoding by skipping redundant decoder invocations.
Performance Validation: Demonstrates that NeR-SC outperforms representative neural video representation methods and, at low bitrates, surpasses traditional codecs (H.264, H.265) on screen content benchmarks.

4. Experimental Results

The method was evaluated on two datasets: DSCVC (diverse screen content) and VCD (video conferencing).

Quantitative Performance:
- NeR-SC achieved average PSNR of 40.32 dB on DSCVC and 41.73 dB on VCD.
- It outperformed the SNeRV backbone by 0.54 dB (DSCVC) and 0.11 dB (VCD).
- At low bitrates, NeR-SC surpassed H.264 and H.265, demonstrating superior rate-distortion performance tailored to screen content statistics.
- In model size ablation studies, NeR-SC at 1.5M parameters (38.54 dB) already exceeded HNeRV at 6.0M parameters (36.81 dB).
Qualitative Performance:
- Visual comparisons showed NeR-SC produced sharper structures and more consistent color reproduction than NeRV and HNeRV.
- Compared to SNeRV, NeR-SC better preserved content elements (e.g., dark rectangular blocks) and boundaries, attributed to the palette and MGF modules.
Efficiency:
- The embedding-level skip strategy increased decoding throughput by 20.7 FPS, achieving 61.7 FPS (real-time) with no loss in PSNR.
- Ablation studies confirmed the Palette module added +0.22 dB PSNR, MGF added +0.17 dB PSNR while reducing parameters, and the Skip strategy provided the throughput gain.

5. Significance and Claims

The paper claims that NeR-SC successfully bridges the gap between general-purpose neural video representation and the specific statistical needs of screen content. By explicitly modeling discrete colors, enhancing multi-scale fusion, and leveraging temporal redundancy at the embedding level, the framework achieves state-of-the-art performance among neural methods.

The authors emphasize that neural representation methods, when properly adapted to the target content type, can compete with and even outperform traditional codecs for screen content, particularly at low bitrates. The work also highlights the potential of zero-overhead skip strategies to enable real-time decoding without compromising quality, addressing a critical bottleneck in deploying neural codecs for interactive applications like cloud gaming and remote desktop.

Future work is noted to focus on extending NeR-SC to higher resolutions and integrating it with traditional Screen Content Coding (SCC) tools for hybrid pipelines.