VibeFlow: Versatile Video Chroma-Lux Editing through Self-Supervised Learning

1. Problem Statement

Video Chroma-Lux Editing involves modifying the illumination and color of a video while strictly preserving its structural integrity (geometry, object identity) and temporal coherence (flicker-free motion).

• Current Limitations: Existing methods typically rely on supervised training using expensive, synthetic paired datasets (e.g., same scene under different lighting generated via virtual engines). These approaches often require:
  • Massive datasets (millions of images).
  • Heavy computational resources (e.g., 8+ H20 GPUs).
  • Auxiliary physical priors (depth maps, normal maps, albedo).
• The Gap: Specialized methods struggle to generalize to diverse, real-world ("in-the-wild") scenarios without dedicated retraining. General video editing models often fail to maintain strict structural fidelity during photometric changes, leading to artifacts or identity shifts.

2. Methodology: VibeFlow

VibeFlow is a self-supervised framework that leverages pre-trained video generation models (specifically Diffusion Transformers) to edit video color and light without paired data or external physical supervision.

A. Core Architecture

• Backbone: Utilizes a pre-trained video-to-video generation model (e.g., Wan2.1) fine-tuned via LoRA (Low-Rank Adaptation).
• Input: A source video $\{I_{src}\}$ and a reference image $I_{ref}$ (providing target color/light cues).
• Goal: Reconstruct the source video but with the photometric attributes of the reference image, while keeping the structure identical.

B. Key Technical Components

1. Disentangled Self-Supervised Training Pipeline
Instead of learning from paired data, VibeFlow creates a "proxy task" to force the model to learn the separation of structure and appearance:

• Perturbation Strategy:
  • Structure Perturbation ($T_{high}$): Applied to the reference image (e.g., elastic transformation, Gaussian blur) to corrupt structural details.
  • Color/Light Perturbation ($T_{low}$): Applied to the source video (e.g., color jitter) to corrupt photometric details.
• Training Objective: The model is trained to reconstruct the original, unperturbed source video using the perturbed source video and perturbed reference image as conditions.
• Result: This forces the model to learn to extract structural information exclusively from the source video and color/illumination cues exclusively from the reference image, achieving robust disentanglement without paired data.

2. Residual Velocity-Guided Structure Preservation
Flow-based models (like Rectified Flow) suffer from discretization errors when using discrete solvers (e.g., Euler), leading to structural degradation even in identity mappings.

• Residual Velocity Fields ($V_{res}$): The authors define a residual velocity field derived from the reconstruction deviation under original conditions:
  $$V_{res} = V_{gt} - V_{recon}$$
  where $V_{gt}$ is the ground truth velocity and $V_{recon}$ is the predicted velocity.
• Trajectory Rectification: During inference, this residual field is added to the target editing velocity to correct the sampling trajectory:
  $$V_{edit}^{corrected} = V_{edit} + \gamma \cdot V_{res}$$
  This injects fine-grained structural details from the source into the edited video, compensating for solver errors.

3. Structural Distortion Consistency Regularization
To ensure the residual velocity field captures only structural information and not color/light dependencies:

• Dual-Branch Training: A second training branch swaps the source and perturbed conditions.
• Consistency Loss: The model minimizes the discrepancy between the residual velocities computed from both branches.
• Effect: This enforces invariance, ensuring the structural compensation mechanism works robustly regardless of the specific color or lighting conditions being applied.

3. Key Contributions

1. VibeFlow Framework: A novel self-supervised approach for video chroma-lux editing that eliminates the need for costly paired datasets and auxiliary physical priors.
2. Disentangled Perturbation Pipeline: A training strategy that forces pre-trained models to implicitly learn the separation of structure and photometry, enabling zero-shot adaptation.
3. Residual Velocity Mechanism: A trajectory rectification method using residual velocity fields and consistency regularization to solve discretization errors, ensuring high-fidelity structural preservation and temporal coherence.
4. Zero-Shot Generalization: The method generalizes to diverse tasks (relighting, recoloring, day-night translation, low-light enhancement) without task-specific fine-tuning.

4. Experimental Results

• Datasets: Trained on OpenVid-1M (67k samples) and evaluated on DAVIS-val.
• Metrics: Outperforms state-of-the-art (SOTA) specialized methods (RelightVid, TC-Light, ColorMNet) and general editing models (Wan2.1, AnyV2V) across:
  • Subject Consistency (SC): Lower MSE (0.0817 vs. 0.12+).
  • Structural Fidelity (SF): Higher SSIM (0.9402 vs. 0.9370).
  • Temporal Coherency (TC): Higher CLIP similarity.
  • Motion Consistency (MC): Higher optical flow SSIM.
• Qualitative Performance:
  • Successfully preserves fine-grained details (e.g., clothing wrinkles, facial features) that other methods distort.
  • Eliminates temporal flickering and unnatural artifacts.
  • Handles complex tasks like day-night translation and object-specific color editing effectively.
• Efficiency: Training takes ~8 hours on 8 A100 GPUs; inference takes 3-5 minutes per video on a single A100.

5. Significance

VibeFlow represents a paradigm shift in video editing by moving away from data-hungry supervised learning toward self-supervised exploitation of pre-trained model priors.

• Accessibility: It democratizes high-quality video editing by removing the barrier of needing massive, expensive synthetic datasets or rendering pipelines.
• Quality: It achieves superior structural fidelity, a critical bottleneck in current video editing, by mathematically correcting the discretization errors inherent in flow-based solvers.
• Versatility: It provides a unified solution for multiple photometric editing tasks (lighting, color, time-of-day) with a single framework, making it highly practical for cinematic post-production, photography retouching, and AI-driven environment design.