Mirai: Autoregressive Visual Generation Needs Foresight

The paper introduces Mirai, a framework that injects foresight signals from future tokens into autoregressive visual generation models to align with their internal representations, thereby significantly accelerating convergence and improving image quality without architectural changes or inference overhead.

The Gist

The Big Problem: The "Blind Jigsaw Puzzle"

Imagine you are trying to assemble a massive jigsaw puzzle, but you are blindfolded. You can only see the piece you are holding right now and the pieces you have already placed. You have to guess what the next piece looks like based only on the immediate neighbor.

This is how current AI image generators (called Autoregressive or AR models) work. They build an image one tiny pixel-block (token) at a time, like a line of dominoes.

• The Issue: Because they can't see the whole picture, they often get lost. They might draw a bird's head perfectly, but then forget where the body is supposed to be, resulting in a bird with a disconnected head or a rocket with smoke blowing in the wrong direction. They are great at local details but terrible at the "big picture."

The Solution: "Mirai" (The Crystal Ball)

The authors introduce a new method called Mirai (which means "Future" in Japanese).

Think of Mirai as giving the blindfolded puzzle solver a crystal ball or a map of the finished puzzle.

• The Twist: The solver still has to place the pieces one by one (so the process remains fast and simple), but while they are placing a piece, they get a gentle nudge from the crystal ball saying, "Hey, remember that the sky is blue and the mountain is over there? Make sure your current piece fits that future plan."

This "nudge" is called Foresight. It allows the AI to plan ahead without actually changing how it generates the image.

How Mirai Works: Two Ways to Look Ahead

The paper tests two different ways to give the AI this "foresight":

1. Mirai-E (The "Self-Reflection" Method)

• The Analogy: Imagine the AI is a student taking a test. Usually, they just write the answer. With Mirai-E, the student is also given a slow-moving, slightly blurry version of their own future answers.
• How it works: The AI looks at a "future version" of itself (created by averaging its past self) to see where the image is heading. It uses this to correct its current step. It's like looking at your own reflection in a mirror that shows you 5 seconds into the future to make sure you aren't walking into a wall.

2. Mirai-I (The "Expert Mentor" Method)

• The Analogy: Imagine the student is taking a test, but a super-smart professor (a pre-trained AI that has seen millions of images) is sitting next to them. The professor can see the entire finished image instantly.
• How it works: The professor whispers to the student, "You are drawing a cat's ear right now, but remember, the whole cat needs to be sitting on the mat." The student doesn't copy the professor's drawing; they just use the professor's "big picture" advice to make their own drawing better.

Why This is a Game-Changer

The paper proves that giving the AI this "foresight" is like giving it a superpower:

1. Speed: It learns 10 times faster.
   • Analogy: If a normal student takes 10 hours to learn a subject, the Mirai student learns it in 1 hour because they aren't wasting time guessing the wrong path.
2. Quality: The images look much more coherent.
   • Analogy: Instead of a puzzle where the sky is on the ground and the grass is in the clouds, the whole image makes sense. The smoke from a rocket goes up, not sideways.
3. No Extra Cost: The best part? When the AI actually creates the image for you, it doesn't need the crystal ball or the professor anymore. It just uses the knowledge it learned. So, it generates images just as fast as before, but the results are much better.

The Key Takeaway

The paper argues that AI needs to "think ahead" to draw well.

By teaching these models to peek at the future (even just during training), they stop being myopic (short-sighted) and start understanding the global structure of the image. It's the difference between a robot that just follows orders step-by-step and a human artist who has a vision of the final masterpiece before they even pick up the brush.

In short: Mirai teaches AI to plan ahead, resulting in faster training and much more beautiful, logical images.

Technical Summary

1. Problem Statement

Autoregressive (AR) visual generation models treat images as sequences of discrete tokens (typically in raster order) and are trained using a strictly causal Next-Token Prediction (NTP) objective. While successful in language modeling, this paradigm faces significant challenges in vision:

• Global Incoherence: Strictly causal supervision relies only on immediate past tokens. This forces the model to learn global structure only through the slow propagation of local cues, often resulting in images that are locally consistent but globally misaligned (e.g., disconnected body parts, inconsistent backgrounds).
• Slow Convergence: The lack of global planning signals forces the model to take many training steps to learn long-range dependencies, leading to slower convergence compared to diffusion models or masked generative models.
• Limitations of Existing Fixes: Attempts like Multi-Token Prediction (MTP) in language models often fail in vision because predicting multiple future discrete tokens simultaneously introduces gradient interference and target competition in the high-dimensional token space.

The authors hypothesize that AR models lack foresight—training signals derived from future tokens that allow the model to "plan ahead" and anticipate upcoming structures without violating causal decoding at inference time.

2. Methodology: The Mirai Framework

The authors propose Mirai (Japanese for "future"), a general training framework that injects future information into AR training without altering the model architecture or inference cost. The core idea is to align the AR model's internal representations with foresight representations derived from future tokens.

The total loss function is:
$$L_{Mirai} = L_{NTP} + \lambda L_{Foresight}$$

Key Design Choices (Diagnostic Findings)

Through extensive ablation studies, the authors identified three critical design principles:

1. Injection Level: Foresight must be injected at the internal representation level (hidden states), not the output prediction level. Predicting future tokens directly at the output head causes gradient interference. Aligning hidden states regularizes the model's internal state without forcing it to emit discrete tokens prematurely.
2. Spatial Layout: Foresight must be positioned in a 2D grid (spatial nearest neighbors) rather than a 1D raster scan. Visual geometry relies on 2D spatial coherence; 1D scanning pairs spatially irrelevant tokens, weakening the supervisory signal.
3. Source of Foresight: Foresight can be Explicit (from a unidirectional model) or Implicit (from a bidirectional encoder).

Two Instantiations of Mirai

• Mirai-E (Explicit Foresight):
  • Source: Uses the Exponential Moving Average (EMA) of the AR model itself as a unidirectional foresight encoder.
  • Mechanism: Aligns the current hidden state with the representations of specific future tokens (e.g., tokens at spatial offsets like "right" and "below") in the 2D grid.
  • Alignment: Uses distinct projection heads for each future position to ensure explicit, position-indexed supervision.
• Mirai-I (Implicit Foresight):
  • Source: Uses a frozen, pretrained bidirectional encoder (e.g., DINOv2) that processes the full image.
  • Mechanism: Aligns the AR model's hidden state at position $n$ with the representation at the same position $n$ from the bidirectional encoder. Since the bidirectional encoder sees the whole image, its features implicitly contain global context and "foresight."
  • Alignment: Uses a single projection head to align the current state with the globally informed feature.

Inference: During inference, the foresight encoders and projection heads are discarded. The model decodes token-by-token strictly causally, maintaining the same computational cost as the baseline AR model.

3. Key Contributions

1. Systematic Investigation of Foresight: The paper provides a comprehensive analysis of where (internal vs. output), how (1D vs. 2D), and from where (explicit vs. implicit) to inject foresight, establishing that 2D-aligned internal representation alignment is the optimal strategy for visual AR.
2. The Mirai Framework: A novel, architecture-agnostic framework that accelerates AR training and improves quality by aligning internal states with future-aware signals.
3. Significant Performance Gains: Demonstrates that foresight is not just a minor tweak but a fundamental requirement for high-quality AR visual generation, achieving state-of-the-art results among AR methods.

4. Experimental Results

Experiments were conducted on the ImageNet 256×256 benchmark using the LlamaGen family of models (B, L, XL scales).

• Convergence Speed: Mirai dramatically accelerates training.
  • Mirai-I achieves the same FID as the baseline LlamaGen-B in 40 epochs (vs. 400 epochs for baseline), representing a 10× speedup.
  • Mirai-E achieves similar gains with an 80-epoch training (5× speedup).
• Generation Quality (FID):
  • LlamaGen-B (300 epochs): FID improved from 5.34 (baseline) to 4.34 (Mirai-I) and 4.49 (Mirai-E).
  • LlamaGen-XL (300 epochs): Mirai-I achieved a best FID of 2.59, outperforming all other AR-based methods and competing with top diffusion models.
• Qualitative Improvements: Visualizations of internal representations (t-SNE) show that Mirai models maintain smooth, spatially coherent feature fields, whereas baselines exhibit scrambled patterns and structural breaks (e.g., disconnected parrot heads, misaligned smoke).
• Generalization: The method works across different model sizes (B to XXL) and different AR architectures (including Parallelized AR), and remains effective in low-resource settings.
• Compute Efficiency: Despite the added training overhead (6.6% for Mirai-I, 38.2% for Mirai-E per image), the massive reduction in required epochs results in a 9.4× reduction in total training compute for Mirai-I to reach the same performance level.

5. Significance

This work fundamentally challenges the assumption that autoregressive visual generation must rely solely on strictly causal, local supervision. By introducing foresight, the authors demonstrate that AR models can learn global structure and spatial coherence much faster, closing the performance gap with diffusion models while retaining the benefits of AR (e.g., exact likelihood, flexible sampling). The findings suggest that global planning signals are essential for visual autoregression, and the Mirai framework offers a simple, effective, and inference-free solution to integrate these signals.