Lumos-1: On Autoregressive Video Generation with Discrete Diffusion from a Unified Model Perspective

Lumos-1 is a unified, LLM-based autoregressive video generation model that introduces MM-RoPE for balanced spatiotemporal modeling and Autoregressive Discrete Diffusion Forcing to address training inefficiencies, achieving state-of-the-art performance with significantly fewer computational resources than existing methods.

The Gist

Imagine you are trying to teach a brilliant, well-read librarian (a Large Language Model, or LLM) how to paint moving pictures instead of just writing stories. The librarian is great at words, but when you ask them to draw a video, they get confused. They try to paint the video one tiny dot at a time, in a strict line, like writing a sentence. This is slow, and the dots often don't connect well, resulting in a blurry, jerky mess.

The paper introduces Lumos-1, a new way to teach this librarian to paint videos. It's like giving the librarian a magical new set of tools that let them paint the whole picture at once, understand how time flows, and fix their mistakes as they go.

Here is how Lumos-1 works, broken down into three simple magic tricks:

1. The "3D Compass" (MM-RoPE)

The Problem:
Standard LLMs use a "1D compass" (called RoPE) to know where words are in a sentence. It's like a ruler that only measures length. But a video isn't just a line; it's a 3D object with Time (how long it lasts), Height (up and down), and Width (left and right).
If you try to use a flat ruler to measure a 3D cube, the measurements get messy. The librarian gets confused about whether a bird is moving up or forward in time.

The Solution:
The authors invented MM-RoPE, a "3D Compass."

• The Analogy: Imagine the librarian's brain is a giant orchestra. In the old system, the instruments for "Time," "Height," and "Width" were all crowded into the same small section of the room, playing at the same speed, causing a cacophony.
• The Fix: MM-RoPE rearranges the orchestra. It gives the "Time" instruments their own spacious section, and it splits the "Height" and "Width" instruments into alternating seats so they can play together without stepping on each other's toes. This allows the model to perfectly understand the complex dance of a video, knowing exactly where an object is in space and how it moves through time.

2. The "Group Painting" Trick (Autoregressive Discrete Diffusion)

The Problem:
Old video models try to paint a video like a human writing a letter: they write one word (or pixel), then the next, then the next. This is called "next-token prediction." For a 25-second video, that's thousands of tiny steps. It's incredibly slow, and if you make a mistake on the first word, the whole sentence gets ruined.

The Solution:
Lumos-1 uses a technique called Discrete Diffusion, which is more like a "Group Painting" party.

• The Analogy: Imagine you have a canvas covered in blank white squares. Instead of painting one square at a time, you cover the entire canvas with blank squares first. Then, you reveal a few squares at a time, let the model guess what goes there, and then reveal more. You repeat this until the whole picture is clear.
• The Twist: The authors realized that if you just reveal random squares, the model gets lazy. It looks at the previous frame and just copies it, because that's the easiest way to fill in the blanks.
• The Fix (AR-DF): They introduced a rule called Temporal Tube Masking. Imagine a tube going straight through the video from start to finish. If you hide a spot in the first frame, you must hide that exact same spot in every single frame that follows.
  • Why this works: The model can no longer just "copy-paste" from the previous frame. It has to actually predict how that specific spot changes over time. It forces the model to learn the physics of motion, not just the pattern of pixels.

3. The "Practice Run" (Inference Strategy)

The Problem:
Even with the Group Painting trick, if you try to generate a video without any "practice," the model gets confused. It sees a perfect first frame, but then tries to guess the second frame with no hints, leading to a jumpy, broken video.

The Solution:
The authors realized that during training, the model always saw some of the previous frames. So, during the actual creation (inference), they do the same thing.

• The Analogy: Imagine you are taking a test. During practice, you are allowed to peek at the previous question's answer to help you with the next one. But on the real test, you are forced to do it blind. You will fail.
• The Fix: When generating the video, Lumos-1 intentionally "blinds" itself to a portion of the frame it just created. It pretends it doesn't know everything, forcing itself to use the "Group Painting" logic to fill in the gaps. This keeps the motion smooth and the video consistent, just like it did during practice.

The Result?

By combining these three tricks, Lumos-1 can generate high-quality videos (Text-to-Video, Image-to-Video) using a standard LLM architecture.

• Efficiency: It's much faster than previous methods because it doesn't paint pixel-by-pixel.
• Quality: It understands motion and space better because of the 3D Compass.
• Simplicity: It doesn't need a massive, separate brain to understand text; it uses the same brain for both reading and painting.

In short, Lumos-1 takes a text expert, gives them a 3D map, teaches them to paint in groups instead of lines, and forces them to practice with the lights dimmed. The result is a model that can turn a simple sentence into a beautiful, moving movie.

Technical Summary

1. Problem Statement

The paper addresses the challenges of adapting Autoregressive (AR) Large Language Models (LLMs) for video generation. While AR models have unified language tasks, their application to video faces three critical bottlenecks:

1. Architectural Mismatch: Standard LLMs use 1D Rotary Position Embeddings (RoPE) designed for sequential text, which are ill-suited for the complex 3D spatiotemporal correlations of video data.
2. Inefficiency of Next-Token Prediction: The standard AR paradigm predicts tokens one by one. For video, this is computationally prohibitive and fails to model the unique properties of video: spatial bidirectionality (pixels within a frame relate to each other) and temporal causality (frames follow a sequence).
3. Training Imbalance: Naive mask-based training (discrete diffusion) leads to loss imbalance. Later frames in a video are easier to predict because spatial information leaks from previous frames, causing the model to "copy" history rather than learn genuine temporal dynamics.

Existing solutions either diverge from standard LLM architectures, rely on heavy external text encoders, or suffer from high latency.

2. Methodology

The authors propose Lumos-1, a unified model based on the Llama architecture that integrates Discrete Diffusion for efficient video generation. The methodology consists of three core innovations:

A. MM-RoPE (Multi-Modal Rotary Position Embedding)

To handle spatiotemporal data within an LLM framework, the authors identified that naive 3D RoPE suffers from imbalanced frequency spectra (temporal dimensions dominate high frequencies while spatial dimensions are relegated to near-zero frequencies, causing aliasing and loss of resolution).

• Solution: They propose MM-RoPE, which features:
  • Distributed Frequency Allocation: Instead of assigning specific channel blocks to time, height, and width, MM-RoPE interleaves these dimensions across the frequency spectrum. This ensures comprehensive modeling of both local and global dependencies.
  • Strategic Scaling: To balance the vastly different scales of text (long captions) and visual tokens (compressed latent frames), the 3D positions are scaled by the compression ratio (e.g., 4x8x8). This aligns the positional resolution of visual tokens with the text modality.
  • Compatibility: Text tokens retain standard LLM RoPE, preserving language capabilities, while visual tokens utilize the new MM-RoPE.

B. Autoregressive Discrete Diffusion Forcing (AR-DF)

To overcome the inefficiency of next-token prediction and the loss imbalance issue, Lumos-1 adopts a parallel, mask-based discrete diffusion approach.

• Training (Temporal Tube Masking): The authors introduce Temporal Tube Masking. Instead of random global masking, a specific mask pattern is generated for the first frame and repeated across the temporal axis for all subsequent frames. This prevents the model from exploiting spatial redundancy in previous frames to predict masked tokens in later frames, forcing it to learn genuine temporal dynamics.
• Inference (Partial Observation): To align inference with training, the model generates frames using a partial observation strategy. After generating a frame, a portion of its tokens is randomly masked (using the same ratio as training) before being fed into the next frame generation. This prevents error accumulation and maintains motion coherence.

C. Unified Architecture

• Backbone: Built on Llama (0.5B, 1.5B, and 3.6B parameter versions).
• Tokenization: Uses the Cosmos Tokenizer for visual data (8x8x4 spatiotemporal compression) and a discrete tokenizer for text, resulting in a unified codebook of ~129k tokens.
• Training Strategy: A stage-wise approach involving text-to-image pre-training, joint image-video training, and supervised fine-tuning (SFT) on high-quality data.

3. Key Contributions

1. Lumos-1 Architecture: The first pure LLM-based unified model for AR video generation that utilizes discrete diffusion, eliminating the need for external text encoders or hybrid architectures.
2. MM-RoPE: A novel positional encoding scheme that solves the frequency imbalance of standard 3D RoPE, enabling effective spatiotemporal modeling while maintaining compatibility with text.
3. AR-DF: A training and inference framework that resolves loss imbalance via temporal tube masking and ensures generation quality via inference-time partial masking, significantly improving efficiency over next-token prediction.
4. Efficiency: Achieves state-of-the-art results using only 48 GPUs for pre-training and fine-tuning, with limited data (60M images, 10M videos) compared to competitors requiring thousands of GPUs or billions of samples.

4. Experimental Results

Lumos-1 was evaluated on standard benchmarks (GenEval, VBench-I2V, VBench-T2V) and compared against leading Diffusion models (e.g., OpenSoraPlan, CogVideoX) and AR models (e.g., Show-o2, EMU3).

• Text-to-Image (GenEval): Lumos-1 (3.6B) achieved a score of 0.791, outperforming Show-o2 (0.76) and EMU3 (0.66), despite using a discrete tokenizer and fewer training images.
• Image-to-Video (VBench-I2V): Lumos-1 (3.6B) scored 84.72, surpassing COSMOS-Video2World (84.16) and VideoCrafter, despite COSMOS using 10x more data and significantly more compute resources.
• Text-to-Video (VBench-T2V): Lumos-1 (3.6B) achieved 78.90, outperforming OpenSoraPlan (77.23) and CogVideoX (81.91 is higher but uses a much larger model and continuous tokens; Lumos-1 is competitive with discrete tokens).
• Efficiency: The model demonstrates superior training efficiency, achieving comparable or better results with significantly fewer GPU-days than diffusion-based counterparts.
• Ablation Studies: Confirmed that MM-RoPE and AR-DF are critical; removing them leads to significant performance drops and slower convergence.

5. Significance

• Unification: Lumos-1 demonstrates that a single LLM architecture can effectively handle both text understanding and complex video generation, moving closer to a true "Unified Model" for multimodal tasks.
• Efficiency: By replacing sequential next-token prediction with parallel discrete diffusion and optimizing positional encoding, the paper offers a pathway to scalable video generation that is computationally feasible for smaller research teams (48 GPUs vs. thousands).
• Future Direction: The work suggests that the limitations of AR video generation are not inherent to the paradigm but rather stem from architectural mismatches (RoPE) and training objectives (loss imbalance), which can be solved with targeted innovations like MM-RoPE and AR-DF.

In summary, Lumos-1 represents a significant leap in making autoregressive video generation efficient, high-quality, and compatible with the unified model paradigm, challenging the dominance of diffusion-only approaches for video tasks.