🤖 AI

A $^2$ RD: Agentic Autoregressive Diffusion for Long Video Consistency

The paper introduces A $^2$ RD, an agentic autoregressive diffusion framework that ensures long-video consistency through a closed-loop retrieve-synthesize-refine-update cycle and multimodal memory, outperforming state-of-the-art methods by up to 30% in consistency and 20% in narrative coherence.

Original authors: Do Xuan Long, Yale Song, Min-Yen Kan, Tomas Pfister, Long T. Le

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Do Xuan Long, Yale Song, Min-Yen Kan, Tomas Pfister, Long T. Le

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to tell a very long, complex story using a magic camera that generates video frames one by one. The biggest problem with current "magic cameras" is that they suffer from amnesia and drift. If you ask them to make a 10-minute movie, by minute 5, the main character might have changed their hair color, the background might have shifted to a different city, or the story might have looped back on itself nonsensically. They get the first few seconds right, but they lose the thread as the story gets longer.

The paper introduces A2RD (Agentic Autoregressive Diffusion), which is like giving that magic camera a super-intelligent director, a memory bank, and a strict editor to work together.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Drifting" Camera

Think of current video generators as a student taking a long exam. They answer the first question perfectly. But for the second question, they only remember the answer to the first one, not the whole exam instructions. By the time they get to question 50, they've forgotten who the main character was, what the setting looks like, or even the plot. This is called "semantic drift." The story falls apart.

2. The Solution: The "Agentic" Director

A2RD doesn't just generate video; it acts like an agent (a smart assistant) that manages the whole process. It breaks the long movie into small chunks (segments) and manages them in a loop: Retrieve, Synthesize, Refine, Update.

Here are the three main tools this "Director" uses:

A. The "Multimodal Video Memory" (The Director's Script & Photo Album)

Instead of just looking at the last frame to guess what comes next, A2RD keeps a detailed memory bank.

The Analogy: Imagine a director who keeps a giant binder. Inside, they have:
- Text Notes: "Clara is wearing a red dress and has a scar on her left cheek."
- Photos: High-quality reference images of the character and the room.
- Video Clips: Short clips of how the character moves.
How it helps: Before generating a new scene, the AI checks this binder. It doesn't just guess; it retrieves the exact details of who Clara is and where she is supposed to be. This prevents the character from accidentally turning into a different person or the room from changing colors.

B. The "Adaptive Segment Generation" (The Smart Switch)

The AI has to decide how to connect one scene to the next. It has two modes:

Extrapolation (Guessing Forward): "The character is walking out the door; let's guess what happens next." This is good for natural movement but risky because the AI might hallucinate (make things up).
Interpolation (Connecting the Dots): "The character starts at the door and ends at the car; let's fill in the middle." This is very safe and consistent but can feel stiff.
The A2RD Trick: The AI acts like a smart editor. It looks at the story and automatically switches between these two modes. If the scene is a simple walk, it guesses forward. If the scene jumps to a new location, it connects the dots strictly to ensure the transition makes sense.

C. The "Hierarchical Self-Improvement" (The Strict Editor)

This is the most unique part. Before the video is final, the AI critiques its own work and fixes it.

The Analogy: Imagine you write a paragraph, then you read it aloud to a strict editor. The editor says, "Wait, you said the car was red, but in the picture, it's blue. Also, the character is facing the wrong way."
The Process:
1. Generate: The AI makes a video clip.
2. Check: An AI "Judge" looks at the clip and compares it to the memory bank and the story. It gives it a score on things like "Is the character's face the same?" and "Is the physics real?"
3. Fix: If the score is low, the AI rewrites its own instructions (prompts) and tries again. It does this twice for every single clip.
4. Learn: It remembers what mistakes it made so it doesn't make them again later in the movie.

3. The New Test: LVbench-C

To prove this works, the authors created a new, very difficult test called LVbench-C.

The Analogy: Most video tests are like asking someone to draw a picture of a cat. Easy. This new test is like asking someone to draw a cat, then a dog, then a cat again (but the cat is now wearing a hat and is wet), then a dog again (but the dog is now old and tired), all while keeping the background consistent.
It tests "cyclical" consistency: Can the AI remember that the character appeared 10 minutes ago, disappeared for a while, and now needs to reappear with specific changes?

The Results

When they ran the tests:

Consistency: A2RD was up to 30% better at keeping characters and environments looking the same throughout the video compared to the best existing methods.
Storytelling: It was 20% better at keeping the story logical and not repeating itself.
Human Feedback: When humans watched the videos, they rated A2RD much higher for smooth transitions and character consistency.

Summary

In short, A2RD is a system that stops video AI from "forgetting" the story halfway through. It does this by giving the AI a detailed memory, letting it choose the best way to connect scenes, and forcing it to critique and fix its own mistakes before showing the final result. It turns a chaotic, drifting video generator into a disciplined, long-form storyteller.

Technical Summary: A2RD: Agentic Autoregressive Diffusion for Long Video Consistency

Problem Statement

Synthesizing consistent and coherent long-form videos (ranging from minutes to hours) remains a fundamental challenge in generative AI. While recent diffusion models have achieved remarkable fidelity for short clips (seconds), scaling to long horizons introduces two critical failure modes: temporal consistency (the inability to track and preserve entities, environments, and motion dynamics over time) and narrative coherence (the inability to evolve stories meaningfully without semantic drift or repetition).

Existing state-of-the-art approaches generally follow a passive, open-loop paradigm:

Frame-based Autoregressive (FAR) models generate frames or chunks sequentially. While they preserve local continuity, they freeze generated frames as fixed conditioning for subsequent steps. This causes errors to propagate uncorrected, leading to semantic drift and narrative collapse over long durations.
Segment-based methods (both parallel and autoregressive) offer better narrative control but struggle with inter-segment consistency. They often rely on visual-only conditioning, which is insufficient for reliably tracking entity identities and narrative states across segments, resulting in persistent inconsistencies in character appearance, object states, and environment layouts.

Methodology: A2RD Architecture

The authors propose A2RD (Agentic Autoregressive Diffusion), a training-free architecture that reframes long video synthesis as a closed-loop, agentic process. A2RD decouples creative synthesis from consistency enforcement, operating through a Retrieve–Synthesize–Refine–Update cycle for each video segment. The system integrates three core components:

1. Multimodal Video Memory (MVMem)

Unlike prior works that store only visual references, MVMem explicitly tracks evolving video world states across three complementary modalities to enforce long-range dependencies:

Textual States ( $T_j$ ): A structured representation capturing Visual Arcs (entity identity, changes, and motion), Spatial Relations (subject-relation-object triplets for geometric layouts), and Camera states (viewport trajectories). These are extracted hierarchically from both boundary frames and full segments.
Frames ( $\mathcal{F}_j$ ): Global reference frames (user-provided or synthesized) and segment keyframes (begin/end) indexed by captions to anchor concrete visual details.
Videos ( $V_j$ ): Synthesized segments stored for motion dynamics verification and refinement.

Initialization: Before generation, the agent plans global references for entities and environments, constructs a dependency graph to determine synthesis order, and synthesizes these references using a Text-to-Image (TI2I) model.

2. Adaptive Segment Generation

A2RD adaptively selects the generation mode for each segment based on narrative context:

Extrapolation: Used when the narrative progresses naturally within a continuous environment, synthesizing forward from a begin frame.
Interpolation: Used when transitioning between distinct environments or shots, synthesizing a segment to connect a fixed begin and end frame.
The agent uses a Multimodal Large Language Model (MLLM) to infer the spatio-temporal continuity of the storyline and select the appropriate mode, balancing natural progression with strict consistency.

3. Hierarchical Test-Time Self-Improvement (HITS)

To prevent error propagation, A2RD employs a closed-loop refinement process at two levels:

Boundary Frame Self-Improvement: Synthesized begin/end frames are verified against a rubric covering entity/environment consistency, narrative progression, spatial logicalness, and physical plausibility. Based on feedback, the agent either edits the frame or regenerates it with an optimized prompt.
Video Segment Self-Improvement: Synthesized segments undergo similar verification. If an end frame is unavailable, the system co-optimizes the end frame and the video prompt sequentially.
Memory-Augmented Prompt Optimization (MAPO): A novel algorithm that leverages a database of past successful and failed refinement cases. By contrasting positive and negative examples, MAPO derives actionable guidelines to refine prompts, addressing root causes of failure rather than applying generic corrections.

Key Contributions

A2RD Architecture: The first agentic autoregressive framework for long video synthesis that integrates multimodal memory, adaptive generation strategies, and hierarchical self-improvement. It significantly mitigates semantic drift and content collapse, scaling to ultra-long videos (up to 10 minutes) while maintaining high consistency.
LVbench-C Benchmark: A new, challenging benchmark designed to stress-test long-horizon consistency. Unlike existing benchmarks, LVbench-C features non-linear transitions where entities and environments appear, disappear, and reappear ("cyclic") with optional state evolutions (e.g., changing clothing, object conditions, or environmental states).
Comprehensive Evaluation: Extensive experiments on public benchmarks (VBench-Long) and LVbench-C, demonstrating state-of-the-art performance in both automatic metrics and human evaluations.

Results

Experiments were conducted on 1-minute (single-scene), 3-minute, 5-minute, and 10-minute (multi-scene) video generation tasks.

Automatic Metrics: On VBench-Long (1-minute), A2RD outperformed the best baseline (Naive-AR) by 20% in narrative coherence (0.90 vs. 0.75) and 30% in character consistency (0.74 vs. 0.57). It achieved a motion consistency score of 0.9935, indicating transitions nearly as smooth as single-shot generation.
Long-Horizon Performance: On LVbench-C (3-5 minutes), A2RD maintained superior consistency, outperforming the strongest baseline (VideoMemory) by 16% (3-min) and 13% (5-min) in character consistency. It also achieved 0.95 narrative coherence on 5-minute videos, significantly higher than baselines.
Human Evaluation: Human raters (7 evaluators) scored A2RD an average of 4.68/5.00, substantially outperforming the best baseline (VideoMemory at 3.93). A2RD excelled in character consistency (4.89), narrative coherence (4.75), and transition smoothness (4.34).
Generalization: A2RD demonstrated strong generalization across different open-source video diffusion backbones (LTX-Video and Wan 2.2), consistently outperforming naive autoregressive baselines without retraining.
Scaling: Analysis showed that while baselines degrade monotonically as the number of scenes increases, A2RD maintains high consistency (e.g., >96% character consistency) even over 40 scenes.

Significance and Claims

The paper claims that A2RD represents a significant step forward in long-form video generation by addressing the fundamental limitations of passive, open-loop paradigms. By introducing an agentic, closed-loop process, the system actively enforces temporal consistency and narrative coherence, preventing the semantic drift that plagues existing methods.

The authors emphasize that their approach is training-free, relying on the orchestration of existing foundation models (MLLM, TI2I, TI2V) through sophisticated memory and refinement mechanisms. The introduction of LVbench-C highlights a gap in current evaluation standards, arguing that true long-horizon consistency requires testing non-linear, cyclic state transitions rather than simple linear progression.

While acknowledging computational overhead due to iterative self-improvement, the authors argue this cost is justified given the substantial improvement in generative yield and the ability to autonomously scale to ultra-long narratives without manual intervention. The work suggests that future progress in long-form video synthesis depends on shifting from passive generation to active, consistency-enforcing agentic systems.