🤖 AI

SteerVTE: Seamless Video Text Editing with Style and Glyph Control

SteerVTE is a unified framework that enables precise video text editing with style and glyph control by steering a frozen video diffusion model through specialized encoders, a novel loss function, progressive training, and a large-scale synthetic dataset.

Original authors: Kai Zeng, Moran Li, Zhengwei Wang, Yingchen Yu, Yiheng Lin, Ruichuan An, Ming Lu, Qi She, Wentao Zhang

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kai Zeng, Moran Li, Zhengwei Wang, Yingchen Yu, Yiheng Lin, Ruichuan An, Ming Lu, Qi She, Wentao Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a home video where a sign in the background says "Cafe," but you want to change it to "Bar" without making the video look weird, flickery, or fake. Doing this with just a photo is hard enough; doing it with a moving video is like trying to repaint a moving train while it's speeding down the track, ensuring every wheel, window, and reflection stays perfectly in sync.

This paper introduces SteerVTE, a new AI tool designed specifically to solve this "moving train" problem. Here is how it works, explained through simple analogies:

The Problem: Why Current Tools Fail

The authors explain that existing video editing tools are like clumsy painters:

Frame-by-frame editing: If you edit every single picture in the video separately (like painting 30 photos a second), the letters might look great in one frame but jump around or change font in the next. It's like a flickering light.
Image-to-Video: If you edit the first frame and tell the AI to "keep going," the AI often gets confused. It might hallucinate new movements or change the background scenery because it doesn't know exactly what the text should look like in the later frames.
General Video Editors: These are great at changing a person's shirt color or adding a cat, but they are terrible at writing words. They often produce gibberish or misspelled letters.

The Solution: SteerVTE

SteerVTE is like a specialized surgical team for video text. Instead of retraining the entire AI brain from scratch (which is expensive and risky), they take a powerful, pre-trained video AI (the "Brain") and freeze it. Then, they attach a lightweight, custom "Headset" (called the Text Context Adapter) that gives the Brain three specific instructions:

The "Where" (The Mask): A precise map telling the AI exactly which pixels to touch and which to leave alone. It's like putting a stencil on the video so the paint only goes on the sign, not the sky.
The "Look" (The Style Encoder): The AI needs to copy the exact font, color, and texture of the original sign. The authors use a special "native-resolution" vision model (like a high-end camera that doesn't squish the image) to capture the style perfectly, ensuring the new text looks like it belongs there.
The "Shape" (The Glyph Encoders): This is the secret sauce. The AI looks at the new text in two ways:
- Line-level: "Where does the whole word go?"
- Character-level: "What does every single letter stroke look like?"
  This ensures the letters are spelled correctly and have the right curves, not just blurry blobs.

The Training: A Three-Step School

You can't teach a student to run a marathon by throwing them into a race on day one. The authors used a Three-Stage Curriculum to train SteerVTE:

Stage 1 (The Basics): The AI learns on simple, computer-generated images. It just learns how to match shapes and letters.
Stage 2 (The Real World): The AI moves to real-world photos with messy fonts and complex backgrounds. It learns to handle the "real" look of text.
Stage 3 (The Marathon): Finally, the AI practices on actual videos. It learns to keep the text steady and consistent as the camera moves.

To make sure the AI pays attention to the tiny details of the letters, they invented a special scoring system called GLAS Loss. Think of this as a teacher who ignores the rest of the page and only grades the student on the specific letters they were asked to write, ensuring they get every stroke perfect.

The Data: Building a Giant Practice Library

Since there weren't enough real videos with text to edit, the team built their own massive library called SteerVTE-1M.

They created 1 million practice examples using a computer pipeline. They took random videos, pasted different text styles on them, and used an automated checker to make sure the text was clear and readable.
They also mixed in real-world photos to ensure the AI didn't just learn to look like a cartoon.

The Results: The Gold Standard

The team created a new test called VTE-Bench (the first ever for this specific task) to see how SteerVTE compares to others.

Accuracy: SteerVTE got the spelling right 77% of the time on real videos, while the next best competitor only got 32%.
Consistency: The text stayed steady and didn't flicker.
Background: The rest of the video remained untouched, preserving the original scene perfectly.

In short, SteerVTE is the first tool that can seamlessly swap text in a moving video, keeping the spelling perfect, the style matching, and the background intact, without the video looking glitchy or fake.

Technical Summary: SteerVTE

Problem Statement

Visual text editing requires precise modification of text within images or videos while maintaining stylistic consistency and visual realism. While significant progress has been made in the image domain using diffusion models, video text editing remains largely unexplored and challenging. The core difficulties stem from the task's localized nature, which demands stroke-level precision within small text regions. This complexity is compounded by the need for:

Cross-frame accuracy: Ensuring text remains correct across frames.
Temporal coherence: Preventing flickering and style drift.
Stylistic fidelity: Preserving the original font, color, and layout.

Existing approaches fail to balance these requirements. Frame-by-frame image editing ignores cross-frame dependencies, causing flickering. Image-to-video (I2V) propagation enforces temporal consistency but often hallucinates motion or breaks fidelity to the original scene. General video-to-video (V2V) models, optimized for scene generation rather than fine-grained text rendering, struggle with character-level precision and often produce garbled text.

Methodology

The authors introduce SteerVTE, a unified framework designed to steer a frozen video diffusion model to perform precise video text editing through style and glyph control. The approach avoids retraining the heavy backbone, instead infusing editing awareness via a lightweight Text Context Adapter.

1. Model Architecture

SteerVTE is built on a frozen pre-trained text-to-video Diffusion Transformer (DiT). The adapter injects three complementary signals via cross-attention:

Spatiotemporal Mask: A mask ( $M$ ) decomposes the source video into foreground and background components, enabling precise region localization while preserving background fidelity.
Style Encoder: To capture visual attributes (stroke thickness, spacing, aspect ratio) without spatial distortion, the authors employ a native-resolution Vision-Language Model (VLM) (specifically Qwen2.5-VL). Unlike standard encoders (e.g., CLIP) that resize inputs to squares, the VLM tokenizes images at their original aspect ratio, preserving intrinsic spatial statistics of the text.
Dual-Granularity Glyph Encoders: To ensure accurate rendering, the model encodes target text at two levels:
- Line-level: Provides a layout prior dictating where glyphs appear.
- Character-level: Uses a pre-trained OCR network to encode individual character strokes, ensuring topological fidelity.
  Both glyph branches use a fixed neutral font to isolate content and structure, leaving style modeling to the VLM encoder.

2. Optimization Objective: GLAS Loss

To counteract the weak text rendering priors of video foundation models, the authors propose the GLyph-Aware Spatial-focal (GLAS) Loss, which operates in a dual-space paradigm:

Spatial-Focal Loss ( $L_{SF}$ ): Reweights the standard flow-matching loss in the latent space using a normalized focal map derived from the spatial mask. This concentrates optimization gradients on text regions, preventing them from being suppressed by the dominant background.
Glyph-Aware OCR Loss ( $L_{OCR}$ ): Imposes supervision in the pixel space. The predicted latent is decoded, and an off-the-shelf OCR recognizer evaluates the output against the ground truth. This loss combines stroke-level fidelity ( $L_{text}$ ) and character-level recognizability ( $L_{CTC}$ ) to enforce structural correctness.

3. Training Strategy: Three-Stage Curriculum

Recognizing that video foundation models lack strong text priors, the authors implement a progressive three-stage curriculum:

Feature Alignment: Trains only the linear connectors of the adapter on synthetic image data (SynthTE-Easy) to establish initial visual-textual mapping.
In-the-Wild Image Fine-Tuning: Jointly optimizes connectors and LoRA modules on real-world image data (RealTE) to bridge the synthetic-to-real domain gap and adapt to complex typographical layouts.
Temporal Video Fine-Tuning: Fine-tunes on video data (SynthTE-Hard) using the full GLAS loss to enforce kinematic coherence and fine-grained glyph fidelity.

4. Data Synthesis

To address the lack of large-scale training data, the authors constructed SteerVTE-1M, a dataset of one million triplets. It combines:

SynthTE: Automatically synthesized data with varying difficulty (fonts, text length, background complexity) using a pipeline that ensures 100% OCR recognition accuracy.
RealTE: Real-world image data derived from OCR datasets, filtered for legibility and stylistic consistency.

Key Contributions

SteerVTE Framework: The first end-to-end model for seamless video text editing, utilizing a frozen DiT backbone augmented with a VLM-based style encoder and dual-granularity glyph modules.
GLAS Loss & Curriculum: A novel loss function and a three-stage training strategy that effectively counteracts the deficient text priors in video foundation models, improving performance on both sentence-level and small-region text editing.
Resources: The construction of SteerVTE-1M (a million-triplet dataset) and VTE-Bench (the first dedicated benchmark for video text editing), addressing the scarcity of training data and standardized evaluation metrics.

Experimental Results

Extensive experiments on VTE-Bench (comprising synthetic and real-world videos) demonstrate that SteerVTE substantially outperforms existing baselines, including frame-wise image editors (FLUX-Text), mask-based video editors (VACE), and instruction-guided models (VIVA, UniVideo).

Text Accuracy: SteerVTE achieves 77% Sentence Accuracy on real-world videos, more than doubling the performance of the proprietary model Seedance 2.0 (32%). It also shows a 34-point improvement in Normalized Edit Distance (NED) over Seedance 2.0 (95% vs. 61%).
Style & Temporal Coherence: The method achieves a Style-Similarity score of 59% (vs. 34% for Seedance 2.0) and an FVD of 14.55, significantly lower than the flickering observed in frame-by-frame methods (FVD > 300).
Background Preservation: The model maintains near-perfect background preservation (SSIMbg > 0.96), avoiding unintended modifications to non-text regions.
User Study: In a quality assessment on real-world videos, SteerVTE achieved a 71% Quality Rate, compared to 23% for Seedance 2.0.

Significance

The paper positions SteerVTE as a fundamental rethinking of video text editing, moving away from the trade-offs inherent in existing methods. By freezing the generative backbone and injecting precise editing signals, the framework achieves a balance between text accuracy, style consistency, and temporal coherence that previous approaches could not attain. The work highlights that video foundation models, while powerful for scene generation, require specific architectural adaptations and curriculum learning to master the fine-grained demands of text rendering. The introduction of SteerVTE-1M and VTE-Bench provides the necessary infrastructure to advance research in this previously underexplored domain.