⚡ electrical engineering

TVRN: Invertible Neural Networks for Compression-Aware Temporal Video Rescaling

This paper proposes TVRN, an end-to-end invertible neural network framework that combines a multi-input multi-output temporal wavelet transform with a surrogate network for lossy codecs and a learning-to-rank strategy to achieve robust, compression-aware temporal video rescaling with superior reconstruction quality.

Original authors: Xinmin Feng, Li Li, Dong Liu, Feng Wu

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinmin Feng, Li Li, Dong Liu, Feng Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a high-definition, high-speed video of a hummingbird's wings flapping. It's beautiful, but it's too heavy to send over a slow internet connection.

The Old Way:
Traditionally, to send this video, you'd just throw away most of the frames (like keeping only every 4th picture) to make it smaller. When the receiver gets it, they try to guess what the missing pictures looked like using a standard "fill-in-the-blank" tool.

The Problem: This is like trying to reconstruct a complex puzzle by guessing the missing pieces without looking at the picture on the box. The result is often blurry, and if you compress the video further (like zipping a file), the "guessing" tool gets confused and the video looks even worse.

The New Solution (TVRN):
The authors of this paper, TVRN, propose a smarter way to handle this. Think of their method as a magic two-way street that treats the video differently before and after it travels.

Here is how it works, broken down into simple steps:

1. The "Magic Split" (Invertible Architecture)

Instead of just deleting frames, TVRN uses a special "splitter" (called MIMO-TWT).

The Analogy: Imagine you have a thick, heavy book (the high-speed video). Instead of tearing out pages and throwing them away, you use a magic machine that separates the book into two parts:
1. The Story: A thin, easy-to-read version of the book (the low-frame-rate video) that you can send easily.
2. The Secret Notes: A hidden layer of "high-frequency" details (like the exact speed of the wings or fine textures) that are too complex to send directly.
Why it's cool: The machine is invertible. This means the process is perfectly reversible. If you have the "Story" and the "Secret Notes," you can put them back together to get the exact original book back. No information is ever truly lost; it's just hidden.

2. The "Secret Decoder" (Surrogate Network)

Real-world video compression (like sending a video via WhatsApp or YouTube) is a "black box." It's a rigid machine that doesn't understand math, so computers can't easily learn how to fix the damage it causes.

The Problem: You can't teach a computer to fix a broken video if you don't know exactly how the "black box" broke it.
The Solution: TVRN builds a fake twin of the compression machine (called a Surrogate Network). This twin acts like a perfect mimic. It pretends to be the real compression software, allowing the main system to learn, "Oh, when the video gets compressed, it loses this specific type of detail." This lets the system train itself to survive the compression journey.

3. The "Motion Guide" (Optical Flow)

When you split the video, the "Secret Notes" (the high-frequency details) are often out of sync because things are moving fast.

The Analogy: Imagine trying to glue a torn piece of a moving car back together. If you just look at the pieces, they might not line up.
The Solution: TVRN uses a motion guide (bi-directional optical flow). It's like having a GPS that tells the system exactly how the car moved between frames. This helps the system align the "Secret Notes" perfectly before gluing them back onto the "Story."

4. The "Smart Filter" (Compression-Aware Features)

Sometimes, the video gets compressed so heavily that it looks like static noise. A standard fixer might try to "clean" the video but accidentally wipe out the "Secret Notes" you need to rebuild the high-speed version.

The Solution: TVRN uses a smart filter that knows exactly how much the video was compressed. It's like a chef who knows exactly how much salt was added to a soup and adjusts the seasoning accordingly, rather than just adding more salt blindly. This ensures the system knows what to save and what to discard, even under heavy compression.

The Result

When you put all these pieces together, TVRN acts like a time-traveling video courier:

It splits the video into a "travel-friendly" package and a "hidden treasure map."
It learns exactly how the delivery truck (the internet compression) damages the package.
It uses a motion guide to keep the treasure map aligned.
At the destination, it uses the map and the package to perfectly reconstruct the original high-speed video, even if the package got a little beat up during the trip.

In short: TVRN is a system that doesn't just guess missing video frames; it hides the missing details in a clever way, learns how the internet breaks them, and uses a smart guide to put them back together perfectly, resulting in a much clearer video than previous methods.

Technical Summary: TVRN: Invertible Neural Networks for Compression-Aware Temporal Video Rescaling

Problem Statement
The rapid growth of high-frame-rate (HFR) video content creates a bottleneck for transmission over bandwidth-constrained networks. A common solution involves temporally downscaling HFR videos to low-frame-rate (LFR) formats for transmission, followed by upscaling (frame interpolation) on the client side. Existing methods typically treat downscaling and upscaling as separate tasks or link them only through training objectives, failing to fully exploit their reciprocal nature. This often leads to high-frequency information loss. Furthermore, current approaches frequently overlook the impact of lossy video codecs (e.g., HEVC, VVC, AV1) on LFR videos. Most learned methods rely on rounding operations that limit compatibility with standard lossy codecs, and existing surrogate networks struggle to accurately model inter-frame compression artifacts, particularly those arising from motion compensation. Consequently, achieving effective, high-quality frame-rate rescaling under non-differentiable lossy compression remains a significant challenge.

Methodology
The authors propose TVRN (Temporal Video Rescaling Network), an end-to-end framework designed for compression-aware frame-rate rescaling. The core architecture and components include:

Invertible Architecture with MIMO-TWT: To regularize information loss during downscaling, TVRN adopts an invertible neural network (INN) structure. It introduces a Multi-Input Multi-Output Temporal Wavelet Transform (MIMO-TWT). Unlike traditional temporal wavelets that require additional motion cues, MIMO-TWT employs a prediction-first lifting scheme using a lightweight video frame interpolation network for prediction and Dense3D-T blocks for updates. This decomposes HFR videos into temporal low-frequency (LFR) frames and high-frequency (HF) residuals without needing to transmit extra motion information.
High-Frequency Reconstruction: To recover the discarded HF components during upscaling, the framework utilizes a bidirectional optical flow-guided reconstruction module. This module aligns contextual features from neighboring LFR frames using bidirectional optical flow to address spatial misalignment caused by motion, refining the initial HF estimates via a context-aware U-Net.
Surrogate Network for Lossy Codecs: To enable end-to-end training through non-differentiable lossy codecs, the authors design a surrogate network ( $G$ ). This network mimics compression distortions by degrading HF components. It incorporates Q-Invertible Blocks, which are compression-aware invertible coupling layers that condition their transformation functions on quantization parameters (QP) derived from the codec. This allows the network to simulate inter-frame prediction artifacts and residual encoding loss more accurately than previous Dense3D-T based surrogates.
Asymmetric Architecture with Compression-Aware Features: Recognizing that symmetric invertible architectures are sensitive to distribution shifts caused by varying compression levels, TVRN extends to an asymmetric design. It integrates Video Quality Enhancement (VQE) modules. Crucially, instead of applying VQE directly to remove artifacts (which can distort embedded HF information), the framework treats upscaling as a decoupling process. It incorporates compression-aware features learned via a pairwise learning-to-rank strategy. These features allow a single model to adaptively control enhancement strength across different QPs, improving robustness without requiring QP-specific models.
Alternate Training Strategy: The framework employs an alternate optimization strategy where the surrogate network and the rescaling network are trained in alternating steps to ensure stable convergence.

Key Contributions

Invertible Framework: The design of MIMO-TWT and a bidirectional optical flow-guided HF reconstruction module, which effectively regularizes and reconstructs high-frequency information lost during downscaling within an invertible architecture.
Compression-Aware Surrogate: The introduction of a surrogate network with compression-aware invertible blocks to estimate gradients of non-differentiable lossy codecs, enabling end-to-end optimization under realistic compression settings.
Asymmetric Extension: The identification of challenges in integrating VQE methods and the proposal of an asymmetric architecture that uses learning-to-rank derived compression-aware features to enhance robustness across varying compression levels.
Performance: The demonstration that TVRN outperforms existing frame-skipping and learning-based methods in reconstruction quality under industrial video compression settings.

Experimental Results
Extensive experiments were conducted on standard benchmarks (UCF-101, Vimeo90K, SNU-FILM) using H.265, VVC, and AV1 codecs.

Reconstruction Quality: TVRN achieves state-of-the-art performance in HFR reconstruction, outperforming the best existing VFI methods (e.g., GIMM-VFI) by significant margins in PSNR and SSIM. For instance, on the Vimeo90K test set, it achieves 35.65 dB PSNR compared to 35.23 dB for GIMM-VFI.
Rate-Distortion (RD) Performance: TVRN is the first learnable frame-rate rescaling method to surpass state-of-the-art VFI methods in RD performance. It achieves BD-rate improvements of -7.22% (PSNR) and -16.40% (SSIM) on UCF-101 compared to GIMM-VFI.
Surrogate Effectiveness: Ablation studies show that replacing the gradient estimator of a competing method (CSTVR) with the proposed surrogate network significantly improves its performance, validating the surrogate's ability to model codec distortions.
Efficiency: A lightweight variant, TVRN-S, reduces FLOPs to 6.54% and upscaling time to 35.75% of GIMM-VFI while maintaining competitive quality.
Subjective Quality: A Mean Opinion Score (MOS) study with 25 participants indicates a clear preference for TVRN (4.02) over the second-best method (3.69).

Significance and Claims
The paper claims that TVRN represents a significant advancement by successfully integrating invertible neural networks with compression-aware mechanisms for temporal video rescaling. The authors state that this is the first learnable frame-rate rescaling method to outperform state-of-the-art VFI methods under modern lossy codecs. By modeling the reciprocal nature of downscaling and upscaling and accurately simulating lossy compression, TVRN addresses the critical gap between theoretical frame-rate rescaling and practical deployment in bandwidth-constrained streaming environments. The work highlights that jointly optimizing these processes, rather than treating them separately, is essential for preserving high-frequency motion information and achieving high-fidelity reconstruction.

1. The "Magic Split" (Invertible Architecture)

2. The "Secret Decoder" (Surrogate Network)

3. The "Motion Guide" (Optical Flow)

4. The "Smart Filter" (Compression-Aware Features)

The Result

More like this