⚡ electrical engineering

ScalablePromptus: Scalable and High-Fidelity Prompt-Based Video Streaming

ScalablePromptus is a robust video streaming framework that overcomes the vulnerability of existing prompt-based methods to network fluctuations by employing a dropout training strategy and advanced interpolation techniques, enabling high-fidelity video reconstruction from arbitrarily truncated prompts and reducing performance degradation by 82%–95% under lossy conditions.

Original authors: Zehao Cao, Bowei Xu, Xun Cao, Zhan Ma, Hao Chen

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zehao Cao, Bowei Xu, Xun Cao, Zhan Ma, Hao Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a secret message to a friend across a stormy ocean. In the old days, you would try to send a giant, heavy crate filled with a perfect, pixel-by-pixel photograph of a sunset. But the waves are rough, the crate is too heavy, and if even a single plank breaks, the whole photo gets ruined. This is how traditional video streaming works: it tries to squeeze every tiny detail of a picture into a file, but when the internet connection wobbles, the picture turns into a blurry, blocky mess.

Now, imagine a smarter way. Instead of sending the heavy crate of pixels, you send a tiny, magical note that says, "Draw a sunset with orange clouds and a purple sky." Your friend, who has a super-smart artist robot at home, reads the note and paints the picture instantly. This is the idea behind "prompt-based video streaming." It's a new frontier in computer science where we stop sending the picture itself and start sending the instructions to create the picture. The goal is to make video so small it can fly through the thinnest internet connections without getting lost. But there's a catch: if the ocean gets too stormy and the note gets torn in half, the robot might draw a purple sun or a sky made of soup. That's the problem this paper sets out to solve.

The Problem: When the Note Gets Torn

The researchers started with a cool system called "Promptus," which was already pretty good at sending those magical "draw this" notes instead of heavy picture files. But Promptus had a major weakness: it was fragile. If the internet connection hiccuped and the note arrived only halfway, the robot at the other end would panic. It would try to draw with a broken instruction, and the result would be a catastrophic mess—a total quality collapse where the video becomes unrecognizable.

Think of it like a recipe. If you send a recipe for a cake that says "Add 2 cups of flour, 1 cup of sugar, and 3 eggs," but the mailman drops the bag and only the "2 cups of flour" part arrives, your friend can't just make a half-cake. They might try to bake with just flour and end up with a brick. The old system (Promptus) was like that recipe; it needed the whole list to work, and if you lost the end, the whole thing failed.

The Solution: A "Rank-Ordered" Recipe

The team at Nanjing University, led by Zehao Cao and Hao Chen, built a new system called ScalablePromptus. Their big idea was to make the instructions "rank-ordered."

Imagine you are writing a recipe, but you write it in a special way. The first few lines are the most important: "Flour, Sugar, Eggs." The next lines are details: "A pinch of salt, a dash of vanilla." The last lines are fancy garnishes: "Sprinkle of gold dust." In the old system, if you lost the last line, you were fine, but if you lost the first line, you were doomed. In ScalablePromptus, the system is trained so that no matter how much of the note gets torn off, the remaining part still makes sense.

If the stormy ocean only delivers the first two lines ("Flour, Sugar"), your friend can still bake a decent cake. If it delivers the first five lines, the cake gets even better. If it delivers the whole thing, it's a masterpiece. This is called a "rank-ordered representation." The system forces the most critical information to be at the very beginning of the note, so even a tiny, truncated note can produce a recognizable video.

How They Did It: Three Magic Tricks

To make this work, the team used three clever techniques:

The "Smart Artist" Training (Dropout): They trained their AI robot using a game of "hide and seek." During training, they would randomly hide parts of the instructions (like covering up the last few words of the recipe) and force the robot to learn how to draw a good picture with whatever was left. This taught the robot to put the most important details at the very front of the note. This is the core of their "dropout training strategy."
The "Color Check" (Semantic and Color Loss): Sometimes, the robot would draw a sunset that looked right but felt weird—maybe the sky was too gray or the colors were dull. To fix this, they added a "color check" and a "meaning check." They made sure the robot didn't just copy pixels, but actually understood the vibe and the colors of the scene. They also fixed a math problem where the robot's instructions would get "squished" when moving between frames, making the video look washed out. They replaced a straight-line math trick with a curved, "spherical" one (called Slerp) that keeps the colors rich and the shapes sharp.
The "No-Reset" Upgrade: In the old system, if your internet got faster in the middle of a video, the system had to throw away the low-quality instructions it had already sent and start over with a new, bigger set. That was wasteful. With ScalablePromptus, if the connection gets better, the sender just sends the rest of the note (the "gold dust" and "vanilla") to add to what was already there. No restarting, no waste.

The Results: Tough as Nails

The researchers tested their new system on various videos, from nature landscapes to people moving around. They simulated stormy internet conditions where packets of data were lost or cut off.

The results were impressive. When the internet connection was perfect, their system made slightly better videos than the old one. But when the connection was bad and the notes were chopped up, the difference was huge. While the old system's video quality would crash and burn, ScalablePromptus kept chugging along. The paper reports that their method reduced the performance drop caused by these cuts by 82% to 95%.

In simple terms: if the old system lost 100% of its quality when the note was torn, the new system only lost a tiny fraction. It proved that you can send video over a shaky, unpredictable connection without the picture turning into soup.

Why It Matters

This isn't just a lab experiment; it's a step toward making video streaming work in the real world, where internet connections are never perfect. Whether you are in a crowded stadium, on a moving train, or in a remote area with spotty Wi-Fi, ScalablePromptus suggests that we can finally stream high-quality video without it freezing or pixelating, simply by sending smarter, more resilient instructions instead of heavy picture files. It turns a fragile system into a robust one, ready for the messy reality of the internet.

Technical Summary: ScalablePromptus

Problem Statement

Prompt-based video streaming represents a paradigm shift from traditional pixel-compression (e.g., H.264/H.265) to transmitting compact semantic prompts for generative reconstruction. While the state-of-the-art framework, Promptus, achieves significant bandwidth reduction by inverting video frames into low-rank prompt embeddings and using Stable Diffusion for regeneration, it suffers from two critical limitations that hinder real-world deployment:

Suboptimal Quality: Promptus relies solely on pixel-level losses (MSE and LPIPS), failing to fully exploit the semantic capacity of the prompt space. Furthermore, its use of linear interpolation (Lerp) for intermediate frames causes "norm collapse" within the hyperspherical embedding space, leading to desaturated colors and loss of detail.
Vulnerability to Network Fluctuation: Promptus lacks mechanisms to handle packet loss or bandwidth variations. It requires pre-fitting multiple prompts at different ranks, which is storage-intensive. Crucially, if a transmitted prompt is truncated due to network loss, the receiver cannot reconstruct meaningful video, resulting in catastrophic quality collapse. Conversely, if bandwidth increases mid-stream, the system cannot incrementally upgrade quality without discarding received data and re-downloading.

Methodology

The authors propose ScalablePromptus, an enhanced framework designed to improve fidelity and ensure robustness against truncation. The pipeline consists of sender-side prompt inversion, transmission, and receiver-side generation.

1. Semantic and Geometric Prompt Enhancement

To address quality issues, the authors introduce a composite loss function and a geometric interpolation strategy:

Semantic Similarity Loss: A cosine similarity loss is added between the semantic embeddings of the generated frame and the ground truth, extracted via a frozen CLIP ViT-B/32 encoder. This ensures semantic alignment beyond pixel-level discrepancies.
Color Statistics Alignment Loss: To correct global color shifts (e.g., graying), a loss function matches the per-channel (RGB) means and standard deviations between generated and ground-truth frames.
Spherical Linear Interpolation (Slerp): Replacing linear interpolation, Slerp is used for both training and inference to generate intermediate frames. Since prompt embeddings reside on a hyperspherical manifold, Slerp travels along the geodesic, preserving the embedding norm. This prevents the "norm collapse" associated with Lerp, maintaining conditioning strength and preventing color desaturation.

2. Dropout Training for Scalable Streaming

The core contribution is a training strategy inspired by Nested Dropout to produce rank-ordered representations:

Mechanism: During training, a random subset of prompt dimensions (rank $k$ ) is dropped in each iteration. The model is forced to encode the most critical visual information in the leading dimensions ( $1 \dots k$ ) and progressively finer details in the trailing dimensions.
Rank-Ordered Representation: This forces the prompt factors to be ordered by importance. The first few dimensions contain coarse, essential content, while later dimensions refine details.
Training Techniques:
- Warmup: Full-rank training is performed initially to ensure all dimensions are initialized meaningfully before random truncation begins.
- Auxiliary Regularization: A small penalty is applied to the magnitudes of non-selected (trailing) dimensions to prevent them from drifting or introducing noise when not in use.
- Smoothed Worst-Step Selection: An exponential moving average (EMA) of the loss is used to select frames for training, mitigating bias caused by random truncation variance.
Inference: The receiver can reconstruct video from any truncated prefix of the prompt (the first $k$ dimensions) without retraining or adaptation. If bandwidth increases, trailing dimensions can be appended incrementally without discarding previously received data.

Key Results

The authors evaluate ScalablePromptus on the QST and UVG datasets (512×512 resolution, 30 fps) against H.265, H.266, VAE-based compression, and the original Promptus.

Stable Networks: Under ideal transmission, the semantic and geometric enhancements yield modest but consistent quality improvements over the baseline Promptus.
Truncation Resilience: When prompts are truncated (simulating bandwidth drops), ScalablePromptus degrades gracefully. Quantitative results show that the method reduces performance degradation (measured by LPIPS increase) by 82%–95% compared to the baseline. While the baseline suffers catastrophic failure when the effective rank drops below the trained rank, ScalablePromptus maintains structurally meaningful output even at low ranks (e.g., $k=4$ ).
Packet Loss Resilience: Under Gilbert-Elliott burst loss models (simulating 5%–20% packet loss), ScalablePromptus maintains significantly lower LPIPS scores than Promptus. For instance, at 20% loss, the baseline LPIPS rises to 0.4491, whereas ScalablePromptus remains at 0.3417.
Incremental Upgrade: The system supports lightweight, incremental quality upgrades. As bandwidth becomes available, the sender transmits only the missing trailing dimensions, which the receiver appends to the existing prompt, avoiding the need to re-download the entire prompt.

Significance and Claims

The paper claims that ScalablePromptus makes prompt-based video streaming robust enough for real-world deployment in volatile network environments.

Practicality: By eliminating the need for pre-fitting multiple rank-specific prompts and enabling graceful degradation under truncation, the system overcomes the primary barrier to adopting generative streaming in lossy networks.
Efficiency: The framework requires only a single trained model per keyframe to support arbitrary rank truncation on demand, reducing storage and training overhead compared to multi-bitrate scalable coding approaches.
Performance: The combination of semantic/color-aware inversion and spherical interpolation addresses the inherent geometric and semantic limitations of previous prompt-based methods, while the dropout strategy provides a novel solution for resilient transmission without requiring complex error correction or retransmission protocols.

The authors conclude that ScalablePromptus demonstrates that scalable prompt streaming is both practical and robust, achieving a balance between ultra-low-bitrate communication and high-fidelity reconstruction even under adverse network conditions.