⚡ electrical engineering

Data-driven Video Codec with Implicit Neural Representations

This paper proposes a novel data-driven video codec that represents video and audio as the weights of a single overfitted sinusoidal neural network, which is then compressed via knowledge distillation, 16-bit quantization, and lossless encoding to achieve a 2.61x compression ratio while maintaining competitive reconstruction quality compared to traditional standards like H.264 and MP3.

Original authors: Nishan Khanal, Saugat Neupane, Abhinav Chalise, Nimesh Gopal Pradhan, Dinesh Baniya Kshatri

Published 2026-07-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Nishan Khanal, Saugat Neupane, Abhinav Chalise, Nimesh Gopal Pradhan, Dinesh Baniya Kshatri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a massive, high-definition movie to a friend. In the world of digital media, we usually treat video and audio like a giant mosaic made of tiny, colored tiles (pixels) and a separate stream of sound waves. Traditional video codecs are like master archivists who look at this mosaic, find all the boring, repeating patterns, and write a very clever set of instructions on how to rebuild it using fewer tiles. They are incredibly efficient, but they rely on a fixed grid of pixels.

Now, imagine a different approach. Instead of storing the tiles themselves, what if you stored a single, super-smart recipe? This recipe is a mathematical function that, when you ask it "What color is the pixel at this specific spot at this specific time?", it instantly answers with the exact color and sound. This is the world of Implicit Neural Representations (INRs). Think of it like teaching a robot to memorize a painting by learning the brushstrokes rather than taking a photo of the canvas. If the robot knows the recipe perfectly, you only need to save the recipe (the robot's "brain" or weights) to recreate the entire movie later. This paper explores whether we can use this "recipe" method to store both video and audio together in one single brain, and if we can shrink that brain down small enough to be useful.

The "One-Brain" Video Codec

The researchers from Tribhuvan University in Nepal asked a bold question: Can we build a single neural network—a digital brain—that learns to be both a video player and a sound system at the same time? Usually, video and audio are handled by separate systems. Here, they built a unified network called a SIREN (Sinusoidal Representation Network).

Think of this network as a magical translator. You feed it a set of coordinates: where you are in the picture (x, y), which frame of the movie it is (t), and the exact moment in time for the sound (T). In return, the network spits out the color of that pixel and the volume of the sound at that exact instant. Instead of saving a file full of millions of pixels, they save the network's "weights"—essentially the mathematical settings that make the translator work.

However, there was a catch. Video and audio are very different beasts. Video changes slowly (a person walking), while audio changes incredibly fast (a violin string vibrating). The researchers found that a single "first layer" of the network couldn't handle both well. It was like trying to use one pair of glasses to read a book and listen to a symphony; neither worked clearly. So, they gave the network two separate "glasses" (initialization layers) at the start—one tuned for the slow-moving video and one for the fast-moving audio—before merging them into a shared brain.

The Compression Magic Trick

The biggest problem with this idea is that the "recipe" (the trained network) is often huge. In fact, for a short video, the recipe can be bigger than the video file itself! To fix this, the team used a three-step compression pipeline:

The Teacher and the Student: First, they trained a large, complex "Teacher" network to memorize the video perfectly. This teacher was huge (about 9.05 MiB). Then, they used a technique called Knowledge Distillation. Imagine the Teacher explaining the video to a smaller, simpler "Student" network. The Student doesn't just learn the raw video; it learns to mimic the Teacher's answers. This shrunk the model down significantly.
Quantization (The Rounding Game): The network's weights are usually stored with high precision (like 32-bit numbers). The researchers tried rounding these numbers down to fewer bits, like 16-bit integers. It's like taking a very precise recipe that says "add 1.000001 grams of salt" and simplifying it to "add 1 gram." They tested everything from 1-bit (very rough) to 32-bit (very precise). They found that once they hit 16 bits, the quality stopped improving. Going higher didn't make the video look better, but going lower made it look terrible.
Lossless Packing: Finally, they used a standard compression tool called LZMA2 (the same kind used in .xz files) to zip up the remaining data without losing any information.

The Results: A Mixed Bag

The team tested this on several short videos. Here is what they found:

The Size Win: For a test video that was 6.08 MiB in size, their final compressed model was 2.33 MiB. That is a compression ratio of 2.61. They successfully squeezed the video and audio into a file less than half the size of the original.
The Quality Trade-off: The video quality was decent but not amazing. The best video they tested got a PSNR of 28.72 dB (a measure of quality where higher is better) and an SSIM of 0.75. The audio was okay, with a PSNR of 24.18 dB.
The "Short Video" Problem: This is the most important finding. Because the network has a fixed size (it needs a certain amount of "brain" to work), short videos actually get bigger when you compress them. For a tiny 5-second clip, the compressed file was larger than the original. The method only starts to save space when the video is long enough to justify the fixed cost of the network.
Comparison to the Giants: When they compared their method to the industry standards H.264 and HEVC (the codecs used by YouTube and Netflix), their method lost. The traditional codecs were much better at compressing video, often by a factor of 10 or more. The new method only matched the quality of traditional codecs when those traditional codecs were set to their lowest, most "grainy" quality settings.

Why This Matters (Even if It's Not Perfect)

The authors are very honest about the limitations. They admit that for now, this method is not ready to replace YouTube or Netflix. It is too slow to train (it takes thousands of hours of computer time to "teach" the network a single video) and it struggles with fast-moving scenes.

However, they discovered something unique: The file size is constant. In traditional video, if you double the length of the movie, the file size doubles. With this method, the file size stays roughly the same whether the video is 1 minute or 1 hour long (once it passes a certain minimum length). This suggests that for very long, high-resolution content, this approach might one day become very efficient.

They also built a working prototype that runs in a web browser, showing that it is possible to train, send, and play these "neural video recipes" over the internet using standard web technologies.

In short, this paper didn't invent a new video format that beats the current champions. Instead, it proved that you can store a video and its soundtrack inside a single mathematical brain, and that you can shrink that brain down to a usable size without destroying the picture. It's a promising step toward a future where we don't just send pixels, but we send the instructions to imagine them.

Technical Summary: Data-driven Video Codec with Implicit Neural Representations

Problem Statement
Conventional video codecs (e.g., H.264, HEVC) compress data by removing spatial and temporal redundancy from a fixed grid of pixel samples. While mature and efficient, these methods face increasing complexity as resolution scales to 4K and 8K. Implicit Neural Representations (INRs) offer an alternative by training a neural network to map coordinates (space and time) directly to signal values (pixel colors or audio amplitudes), storing the signal as the network's weights. However, prior INR-based work typically treats video and audio as separate modalities, requiring distinct networks or hybrid architectures. This paper investigates whether a unified, sample-wise representation using a single Multi-Layer Perceptron (MLP) is practical for joint audio-video compression and quantifies the associated costs.

Methodology
The proposed system encodes a video and its audio track into the weights of a single Sinusoidal Representation Network (SIREN). The architecture and pipeline are defined as follows:

Unified Architecture: The network takes normalized space-time coordinates $[(x,y,t), T]$ $[(x, y, t), T]$ as input, where $(x,y)$ $(x, y)$ is the pixel position, $t$ $t$ is the frame index, and $T$ $T$ is the audio timestep.
- Initialization: Audio and video inputs pass through separate initialization layers. This is critical because audio requires a much wider weight range $[-25, 25]$ to capture high-frequency content, whereas video requires a narrower range $[-2/3, 2/3]$ to avoid graininess.
- Shared Layers: The outputs of the initialization layers are concatenated and passed through a stack of shared fully connected hidden layers using sine activations.
- Output Branches: The network splits into three branches: one for video (RGB output) and two Siamese audio branches. The two audio branches produce slightly different estimates of the same amplitude; their difference is used to estimate reconstruction noise, which is subsequently removed via spectral gating during decoding.
Compression Pipeline:
1. Overfitting: A "teacher" network is trained to overfit the specific video/audio pair.
2. Knowledge Distillation: The teacher (2.37M parameters) is compressed into a smaller "student" network (1.30M parameters) using response-based distillation, where the student learns from both ground truth and the teacher's outputs.
3. Quantization: Weights and biases are symmetrically quantized to 16-bit integers.
4. Lossless Encoding: The quantized weights are compressed using LZMA2 (via xz).
Decoding: The receiver decompresses the weights, dequantizes them, and runs inference over the coordinate grid to reconstruct frames and audio. The audio denoising step utilizes the difference between the two Siamese branches.

Key Contributions

Unified Audio-Video SIREN: A single MLP architecture that jointly stores pixels and audio amplitudes using separate initialization layers and shared hidden layers, a configuration not previously explored in a plain MLP format.
Compression Pipeline: A complete pipeline demonstrating that a 9.05 MiB teacher network can be reduced to 2.33 MiB (a 2.61x compression ratio on a 6.08 MiB test video) through distillation, 16-bit quantization, and LZMA2 encoding.
Quantization Analysis: An empirical study sweeping bit-widths from 1 to 32 bits, identifying that reconstruction quality saturates at 16 bits, below which quality collapses, and above which no visible gains occur.
End-to-End Prototype: A browser-based system using FastAPI and WebRTC that trains, transfers, and decodes these models peer-to-peer.

Results

Performance: On a 6.08 MiB test video, the quantized student achieved a video PSNR of 28.72 dB (SSIM 0.75) and an audio PSNR of 24.18 dB (Log Spectral Distance 10.69 dB).
Comparison with Standards: The approach trails H.264, HEVC, and MP3 in compression efficiency by one to two orders of magnitude at comparable quality levels. The codec is only competitive with conventional codecs at their most aggressive (lowest quality) settings (e.g., CRF 51).
Fixed Size Property: A defining characteristic is that the stored size is constant regardless of video length or resolution. While this results in expansion (compression ratio < 1) for short clips, the relative efficiency improves as input size increases.
Artifacts: The Siamese noise estimation works well for music but can misclassify quiet speech pauses as noise, leading to gaps in the spectrogram. Additionally, distillation sometimes shifts capacity toward video at the expense of audio quality.

Significance and Limitations
The paper modestly claims that while this unified INR approach is not yet a replacement for conventional codecs in terms of raw compression ratio, it offers a unique property: a single, differentiable, resolution-independent representation where storage size is decoupled from content duration. The authors acknowledge four primary limitations:

Fixed Size Floor: The minimum model size (~2.33 MiB) makes the codec impractical for short clips.
Dynamic Content: Reconstruction quality degrades with increased motion and frame rates.
Distillation Trade-offs: Current distillation strategies may compromise audio quality to preserve video quality.
Training Latency: The requirement to train a network per video for thousands of epochs precludes real-time application.

The authors conclude that the approach is a proof of concept for unified modal representation, with future work needed to lower the size floor, handle high-motion content, and reduce training costs.

The "One-Brain" Video Codec

The Compression Magic Trick

The Results: A Mixed Bag

Why This Matters (Even if It's Not Perfect)

Technical Summary: Data-driven Video Codec with Implicit Neural Representations

More like this