💻 computer science

FedVSR: Towards Model-Agnostic Federated Learning in Video Super-Resolution

The paper introduces FedVSR, a model-agnostic and stateless federated learning framework that utilizes a lightweight Discrete Wavelet Transform-based loss and a loss-aware aggregation strategy to significantly enhance the perceptual quality of video super-resolution while maintaining near-zero computational and communication overhead.

Original authors: Ali Mollaahmadi Dehaghi, Hossein KhademSohi, Reza Razavi, Steve Drew, Mohammad Moshirpour

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Mollaahmadi Dehaghi, Hossein KhademSohi, Reza Razavi, Steve Drew, Mohammad Moshirpour

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Blurry Group Project"

Imagine you have a group of friends who each have a blurry, low-quality video on their phones. You want to combine their knowledge to create one single, crystal-clear, high-definition version of that video.

In the old way of doing this (Centralized Learning), everyone would have to upload their entire raw video files to a central super-computer. The computer would then train a "master brain" to fix the blur.

The Catch: This is a privacy nightmare. You don't want to send your private home videos or sensitive surveillance footage to a stranger's server. Plus, uploading 8K video files takes forever and eats up all your data plan.

Federated Learning (FL) was invented to solve this. Instead of sending the videos, everyone keeps their data on their own phone. They train a small part of the "master brain" locally and only send the lessons learned (math updates) back to the server. The server mixes these lessons to improve the master brain.

The New Problem: While this protects privacy, standard Federated Learning is terrible at fixing videos. When the server mixes the lessons from everyone, the result is often a blurry, mushy mess. It's like asking 100 people to describe a painting from memory and then trying to reconstruct the painting from their descriptions; you lose all the fine details, textures, and sharp edges.

The Solution: FedVSR (The "Sharpness Specialist")

The authors created FedVSR, a new system specifically designed to fix videos without breaking privacy or losing detail. Think of it as a specialized coach for the group project who ensures the final result isn't blurry.

FedVSR does two main things to fix the "blurry mess" problem:

1. The "Wavelet" Ear (The Local Trainer)

In video super-resolution, the most important parts are the high-frequency details—the sharp edges of a tree branch, the texture of a brick wall, or the flicker of a light. Standard training often ignores these in favor of just getting the "general shape" right, which leads to blur.

The Analogy: Imagine you are trying to teach a student to draw a detailed map. A normal teacher says, "Make sure the river is in the right place." The student draws a smooth, wavy line.
FedVSR's Approach: FedVSR adds a special "ear" to the student's training process. It uses a mathematical tool called 3D Discrete Wavelet Transform (DWT). Think of this as a pair of glasses that lets the student see the texture and cracks in the map, not just the lines.
How it works: Before the student sends their lesson back to the group, this "ear" checks: "Did you capture the sharp edges? Did you keep the texture?" If the answer is no, the student is forced to try harder to capture those high-frequency details.
Crucial Note: The paper found that this "ear" is only useful in this group setting. If you use it on a single computer with all the data, it actually makes things worse. It's a special tool designed specifically to fix the problems caused by splitting the work up.

2. The "Smart Mixer" (The Server Aggregator)

Once the students (clients) send their lessons back, the server has to mix them together.

The Old Way (FedAvg): The server just takes a simple average. "Okay, Client A says the edge is here, Client B says it's there. Let's put it in the middle." This often results in a muddy, average result that satisfies no one.
The FedVSR Way: The server acts like a Smart Mixer. It listens to how well each student did on their own local test.
- If a student had a very low error (they learned well), their lesson gets a louder voice in the final mix.
- If a student had a high error (they were confused), their lesson gets a quieter voice.
- The Safety Net: The system is smart enough not to ignore the "quiet" students entirely. It uses a mathematical "safety valve" (based on something called Hellinger distance) to ensure that if everyone is doing roughly the same, it just averages them out normally. But if there's a big difference in quality, it prioritizes the best performers to stop the whole group from getting dragged down.

Why This Matters (The Results)

The authors tested FedVSR against other methods (like FedAvg, FedProx, and SCAFFOLD) using real video datasets.

The Outcome: FedVSR produced videos that were significantly sharper and clearer.
- PSNR (A measure of clarity): Up to +0.89 dB better.
- SSIM (Structural similarity): Up to +0.0370 better.
- LPIPS (Perceptual quality): Lower is better, and they reduced it by 0.0347.
- VMAF (Video quality score): Improved by nearly 5 points.
The Cost: The best part? It did all this with almost zero extra cost.
- It didn't require sending extra data (communication overhead).
- It didn't require the phones to do heavy extra math (computation overhead).
- It works with any video model (Model-Agnostic), meaning you don't have to rebuild the whole system to use it.

Summary

FedVSR is a new way to train AI to fix blurry videos without ever seeing the private videos themselves. It solves the "blurry group project" problem by:

Giving each device a special "texture-checking" tool (3D DWT loss) to ensure they don't forget the fine details.
Using a "smart mixer" at the server to weigh the best lessons more heavily than the bad ones.

The result is a high-quality, privacy-safe video enhancement system that works efficiently on everyday devices.

Technical Summary: FedVSR

Problem Statement

Video Super-Resolution (VSR) aims to reconstruct high-resolution (HR) video sequences from low-resolution (LR) counterparts by leveraging both spatial and temporal information. While deep learning has advanced VSR, current state-of-the-art approaches rely on centralized training with massive datasets. This centralization poses significant challenges:

Privacy and Regulation: Raw video data often contains sensitive or personally identifiable information (PII), making centralized collection prohibited by regulations like GDPR or CCPA.
Bandwidth and Logistics: Uploading raw video data (e.g., 8K footage) for training creates prohibitive bandwidth costs and logistical hurdles.
Performance Degradation in Federated Settings: Existing Federated Learning (FL) frameworks, designed primarily for high-level tasks like classification, fail when applied to low-level vision tasks like VSR. Direct application of general FL methods (e.g., FedAvg, FedProx) results in blurry outputs with lost textures and poor temporal coherence. This is attributed to data heterogeneity, communication constraints, and the inability of standard aggregation to preserve the high-frequency details essential for VSR.

Methodology: FedVSR

The authors propose FedVSR, a model-agnostic and stateless framework designed specifically to bridge the gap between VSR quality and FL privacy. The framework operates without modifying the internal architecture of the underlying VSR models or requiring clients to store historical weights. It consists of two primary components:

1. Frequency-Aware Local Training (3D DWT Loss)

To address the loss of high-frequency details (textures and edges) during federated training, FedVSR introduces a lightweight loss function based on the 3D Discrete Wavelet Transform (DWT).

Mechanism: The method applies a separable Haar wavelet transform to the video frames, decomposing them into eight sub-bands (one low-frequency and seven high-frequency).
Loss Function: The training objective includes a standard VSR task loss (e.g., L1/L2) combined with a high-frequency helper loss ( $L_{HiFr}$ ). This helper loss computes the Charbonnier loss between the concatenated high-frequency sub-bands of the predicted frame and the ground truth.
Rationale: Unlike centralized training where standard losses suffice, the FL environment suffers from limited communication budgets that often smooth out high-frequency signals. The DWT loss explicitly forces clients to prioritize these details. The authors note that this loss is specific to the FL environment; applying it in centralized training actually degrades performance due to over-regularization.

2. Adaptive Loss-Aware Aggregation

To mitigate the effects of data heterogeneity and noisy clients, FedVSR employs a hybrid aggregation strategy that dynamically adjusts client weights based on their local training quality.

Loss Tracking: Clients transmit their average total local loss ( $L_{total}$ ) alongside model weights.
Weighting Strategy: The server computes a loss-based distribution ( $l_i$ ) by normalizing the inverse of the local losses. This is combined with a uniform distribution ( $u_i$ ) via a convex combination.
Adaptive Mixing: The mixing coefficient is determined by the Hellinger distance between the uniform and loss-based distributions. A soft threshold ensures that loss-based prioritization is only activated when there is a significant divergence in client update quality. If losses are similar, the system defaults to uniform averaging, preventing the marginalization of diverse but valuable data.

Key Contributions

First FL Framework for VSR: FedVSR is the first framework specifically designed for Federated Video Super-Resolution, addressing the unique challenges of temporal coherence and high-frequency preservation in decentralized settings.
Model-Agnostic and Stateless Design: The framework treats VSR models as black boxes, requiring no internal architectural changes or storage of past weights (stateless). This ensures scalability, low memory usage, and compatibility with diverse VSR architectures (e.g., VRT, RVRT, IART).
Novel Loss and Aggregation: The introduction of the 3D DWT-based loss and the adaptive loss-aware aggregation mechanism creates a holistic solution that improves training robustness without adding significant computational overhead.

Experimental Results

Extensive experiments were conducted across three VSR models (VRT, RVRT, IART) and two datasets (REDS, Kinetics-400) under both less heterogeneous (IID-like) and more heterogeneous (Non-IID) scenarios.

Performance Gains: FedVSR consistently outperformed existing FL baselines (FedAvg, FedProx, SCAFFOLD, FedMedian).
- In less heterogeneous settings, it achieved improvements of +0.82 dB PSNR, +0.0337 SSIM, −0.0334 LPIPS, and +3.62 VMAF on average.
- In more heterogeneous settings, gains were even more pronounced: +0.96 dB PSNR, +0.0404 SSIM, −0.0359 LPIPS, and +6.34 VMAF.
Qualitative Improvements: Visual comparisons showed that FedVSR successfully recovered fine textures and sharp edges that were blurred or distorted by other FL methods.
Robustness and Scalability:
- Communication Failures: FedVSR maintained performance even with up to 75% client upload failure rates, whereas FedAvg degraded significantly.
- Client Population: The method scaled effectively when the number of clients increased by 3x and 6x, and remained robust under extremely low participation rates (1.7%).
- Efficiency: FedVSR incurred close to zero additional computation and communication overhead compared to FedAvg. In contrast, competitors like SCAFFOLD and FedSR introduced significant overhead (e.g., SCAFFOLD required ~2x communication overhead due to control variates; FedSR required massive aggregation overhead).

Significance and Claims

The paper claims that FedVSR sets a new benchmark for federated learning in low-level vision tasks. Its significance lies in:

Bridging the Gap: It successfully bridges the trade-off between privacy, efficiency, and perceptual quality, proving that high-fidelity VSR is achievable in a decentralized manner.
Practical Deployment: By being model-agnostic and stateless, it offers a practical solution for real-world, privacy-sensitive applications (e.g., surveillance, medical imaging) where data cannot be centralized.
Foundational Insight: The work highlights that standard FL techniques are insufficient for regression-based, low-level vision tasks and demonstrates that frequency-aware constraints are necessary to counteract the specific failure modes of federated aggregation in this domain.

The authors conclude that while FedVSR establishes a strong baseline, a performance gap remains between FL-based VSR and centralized approaches, suggesting future work should focus on task-specific aggregation strategies and further loss-guided optimization.