MemCam: Memory-Augmented Camera Control for Consistent Video Generation

1. Problem Statement

Interactive video generation aims to create videos based on user-defined control signals (e.g., camera trajectories). However, existing methods face a critical limitation: long-term scene inconsistency.

• The Challenge: When generating long videos with dynamic camera movements (especially large rotations like 360°), current models tend to "forget" previously generated content.
• The Symptom: When the camera returns to a previously visited viewpoint, the scene often appears distorted, hallucinated, or structurally different from the original.
• Root Causes:
  • Limited Context: Methods like DFoT rely on fixed-length context windows, which cannot retain information from the distant past.
  • Lack of Explicit Memory: Methods like CameraCtrl rely solely on camera parameters without explicit memory of past visual content.
  • Reconstruction Errors: Approaches using 3D reconstruction (e.g., GeometryForcing) suffer from error accumulation and inherent model limitations.

2. Methodology: MemCam

MemCam proposes a memory-augmented framework that treats previously generated frames as external memory to condition future predictions. The architecture is built upon a Diffusion Transformer (DiT) and consists of three core components:

A. Context Compression Module

To handle the computational cost of processing long video histories, MemCam introduces a compression mechanism:

• Architecture: A Convolutional Neural Network (CNN) replaces the standard patchify layer of the base model to process context tokens.
• Compression Strategy: The network compresses spatial dimensions (height and width) by a factor of 2, reducing the token sequence length to 1/4 of the original. The temporal dimension remains uncompressed to preserve frame order.
• Initialization: Weights are initialized by extending the pre-trained model's patchify layer to ensure stable early training.
• Positional Encoding: Rotary Positional Encoding (RoPE) is used. Context tokens receive down-sampled spatial positional encodings to align with the prediction sequence.

B. Co-Visibility Based Context Selection

Instead of using all historical frames or a fixed window, MemCam dynamically selects the most relevant frames based on co-visibility:

• Mechanism: For every frame in the current prediction sequence, the system calculates the overlap between its Field of View (FOV) and the FOVs of all historical frames.
• Calculation: Co-visibility is estimated via Monte Carlo sampling of 3D points. It is defined as the Intersection over Union (IoU) of visible point sets between two camera poses.
• Selection: Frames with the highest IoU scores are selected as context. This ensures that when the camera rotates back to a specific area, the model retrieves the specific historical frames that originally captured that view.

C. Training and Inference Strategy

• Training:
  • A fixed first frame serves as a baseline context.
  • For prediction frames, context frames are selected from the scene based on co-visibility.
  • Selected context frames are encoded by a 3D VAE and passed through the Context Compression Module.
  • A 10% probability of zero-padding context is used to simulate scenarios with no history (Image-to-Video).
• Inference:
  • Uses a segment-wise generation approach.
  • For each predicted frame, the system retrieves the historical frame with the highest co-visibility score.
  • The memory buffer is updated after each segment is generated.

3. Key Contributions

1. MemCam Framework: A novel architecture that explicitly uses historical frames as external memory, enabling scene-consistent generation over long temporal horizons and large camera rotations without requiring 3D reconstruction modules.
2. Context Compression & Selection: The design of a compression module that reduces computational overhead while a co-visibility selection strategy ensures the retrieval of the most semantically and spatially relevant historical context.
3. State-of-the-Art Performance: Demonstrated significant improvements in scene consistency metrics compared to existing baselines, particularly in 360° rotation scenarios.

4. Experimental Results

The method was evaluated on the Context-as-Memory dataset and RealEstate10K (for zero-shot generalization).

Evaluation Metrics:

• Round-trip Generation: 90° and 360° camera rotations returning to the start.
• Metrics: PSNR, SSIM, LPIPS (for consistency), and FVD (for overall video quality).

Quantitative Findings (Table I):

• 360° Scenario: MemCam significantly outperforms baselines.
  • FVD (Lower is better): MemCam achieved 167.87 on Context-as-Memory (360°), compared to 852.05 for GeometryForcing (GF) and 1188.34 for DFoT.
  • PSNR/SSIM: MemCam achieved the highest scores (14.81 PSNR, 0.423 SSIM), indicating superior structural preservation.
• Comparison:
  • I2V (Baseline): Failed to maintain consistency (FVD ~988).
  • DFoT: Struggled with long sequences due to fixed context windows.
  • GF: Suffered from error accumulation in 360° rotations.

Qualitative Findings:

• MemCam faithfully reconstructs the original scene appearance when the camera returns to the starting viewpoint.
• Baselines exhibited "drift" and content hallucination (e.g., changing textures or layouts) after large rotations.

Ablation Studies:

• Selection Strategy: The co-visibility strategy ("Ours") outperformed "Recent" (fails in 360°), "Random," and "TopK" strategies, proving that dynamic relevance is crucial for long-term memory.
• Compression: The compression module allowed the model to use 76 context frames with inference speeds nearly 5x faster than uncompressed concatenation, while maintaining or improving quality.

5. Significance and Conclusion

MemCam addresses a fundamental bottleneck in interactive video generation: long-term memory retention. By treating past frames as a retrievable memory bank and compressing them efficiently, the method enables the generation of long, coherent videos with complex camera movements (e.g., full 360° rotations) that were previously impossible for diffusion models.

• Impact: This approach bridges the gap between short-form video generation and long-form scene simulation, which is critical for applications in gaming, virtual reality, and world simulation.
• Future Work: The authors plan to address the current inference speed limitations (due to bidirectional attention) via diffusion distillation and scale up training with larger datasets.

Code Availability: The implementation is publicly available at https://github.com/newhorizon2005/MemCam.