🤖 machine learning

MIRO: MultI-Reward cOnditioned pretraining improves T2I quality and efficiency

The paper proposes MIRO, a multi-reward conditioned pretraining method that enhances text-to-image generation quality and training efficiency by directly learning user preferences from multiple rewards, thereby outperforming existing post-training paradigms on key benchmarks.

Original authors: Nicolas Dufour, Lucas Degeorge, Arijit Ghosh, Vicky Kalogeiton, David Picard

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nicolas Dufour, Lucas Degeorge, Arijit Ghosh, Vicky Kalogeiton, David Picard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Three-Stage Factory" Bottleneck

Imagine you want to train a robot chef to make the perfect meal. Currently, the industry standard involves a clumsy three-stage process:

The Raw Kitchen (Pretraining): You throw the robot into a chaotic kitchen with every ingredient imaginable—rotten vegetables, burnt toast, and Michelin-star dishes. The robot learns what food looks like, but it doesn’t know what tastes good.
The Filter (Fine-tuning): You hire a strict critic who throws away all the "bad" food. The robot only sees the good stuff now. But by throwing away the bad food, you’ve also thrown away the lessons on why some things are bad. The robot loses its understanding of the full range of cooking.
The Reward Game (RLHF): Finally, you play a game where the robot makes a dish, and you give it a single score (e.g., "Deliciousness"). The robot tries to maximize this one score. The problem? It might start making only one type of dish over and over again because that’s the easiest way to get a high score. It loses creativity (diversity) and might ignore other important things, like whether the dish actually matches your request (semantic fidelity).

The Result: A robot that is slow to learn, wastes data, and produces boring, repetitive results.

The Solution: MIRO (The "Smart Label" Approach)

The authors propose MIRO (MultI-Reward cOnditioned pretraining). Instead of the three-stage factory, they use a single, smarter training method.

Think of it like this: Instead of throwing away the bad food or playing a post-game scoring round, they stick a multi-colored sticker on every single ingredient in the raw kitchen.

One color represents "Aesthetic Beauty."
Another represents "How much humans like it."
Another represents "Does it match the recipe description?"
Another represents "Is it scientifically accurate?"

The robot chef learns to cook while looking at these stickers. It learns that this specific combination of ingredients leads to a "High Beauty" sticker, while that combination leads to a "High Accuracy" sticker. It doesn’t discard the "bad" ingredients; it just learns that they have different sticker profiles.

How It Works: The "Dial" Metaphor

In the old system, the robot was locked into making food that pleased one specific critic. In MIRO, the robot learns the relationship between ingredients and all the different quality stickers simultaneously.

At the end of training, when you want the robot to cook for you, you don’t just say "Make a steak." You can turn dials on a control panel:

Turn the "Beauty" dial up.
Turn the "Accuracy" dial down.
Turn the "Human Preference" dial to the middle.

The robot uses these dials to steer its cooking process in real-time. It knows exactly how to adjust the recipe to hit your specific balance of qualities.

Why Is This Better? (The Key Benefits)

It’s Much Faster (The "Shortcut"):
Because the robot is learning from the "stickers" (reward signals) from day one, it learns what makes a good image much faster. The paper claims MIRO converges up to 19 times faster than the traditional method. It’s like giving the chef a cheat sheet instead of letting them guess.
It’s Smaller but Stronger (The "Underdog"):
MIRO uses a relatively small model (0.36 billion parameters). Despite its small size, it beats massive models like FLUX-dev (12 billion parameters) on several benchmarks. It achieves this while using 370 times less computing power to train. It’s the equivalent of a small, efficient sports car beating a massive, gas-guzzling truck in a race.
No "Reward Hacking" (The "Cheater Prevention"):
In the old system, if you only rewarded "Beauty," the robot might start making blurry, colorful blobs because they scored high on beauty but looked like nothing real. This is called "reward hacking."
Because MIRO looks at multiple stickers at once (Beauty, Accuracy, Preference), the robot can’t cheat. If it makes a blurry blob, it might get a high Beauty score, but it will get a terrible Accuracy score. The multi-sticker system forces the robot to balance all qualities, resulting in better, more honest images.
It Keeps Everything (The "No Waste" Policy):
Traditional methods throw away "low-quality" data. MIRO uses all the data. It learns from the bad images too, understanding what makes them bad. This preserves the diversity of the world, preventing the robot from becoming obsessed with just one style of image.

The Verdict

MIRO changes the game by moving the "quality control" step from the end of the process to the beginning. By teaching the model to understand multiple types of quality simultaneously during its initial training, it creates a system that is:

Faster to train.
Cheaper to run.
More controllable for the user (via the "dials").
Higher quality than much larger, more expensive models.

It’s less about forcing the model to obey one master, and more about teaching it to understand the full spectrum of what makes an image "good."

Technical Summary: MIRO: MultI-Reward cOnditioned Pretraining Improves T2I Quality and Efficiency

Problem Statement
Current state-of-the-art text-to-image (T2I) generators typically follow a three-stage training paradigm: (1) pretraining on noisy web data, (2) supervised fine-tuning on curated subsets, and (3) reinforcement learning from human feedback (RLHF) or similar alignment techniques. The authors argue that this pipeline incurs significant costs: pretraining decouples likelihood maximization from user preferences; fine-tuning discards informative "low-quality" data, sacrificing structural signals; and RLHF collapses the distribution onto a single scalar reward, harming diversity (mode collapse) and semantic fidelity while locking reward trade-offs at training time. Consequently, users are left with an operating point determined by trainers rather than having control over the balance of quality dimensions.

Methodology
The authors propose MIRO (MultI-Reward cOnditioned pretraining), a framework that integrates alignment directly into the pretraining phase by conditioning the generative model on a vector of multiple reward scores.

Dataset Augmentation: The pretraining dataset is augmented with annotations from $N$ complementary reward models (e.g., AestheticScore, HPSv2, ImageReward, PickScore, VQAScore, JINA CLIP, and SciScore). For each image-text pair $(x, c)$ , a reward vector $s$ is computed. Raw scores are normalized and binned into discrete levels to ensure balanced representation across the quality spectrum.
Multi-Reward Conditioned Flow Matching: The authors employ flow matching as the generative framework. The model learns a conditional velocity field $v_\theta(x_t, c, \hat{s})$ , where $\hat{s}$ is the binned reward vector. The training objective minimizes the difference between the predicted velocity and the true velocity $(\epsilon - x)$ , conditioned on both the text prompt and the desired reward levels. This allows the model to learn the mapping between reward levels and visual characteristics across the full data distribution, rather than filtering for high-quality samples only.
Reward-Guided Inference: At inference, MIRO enables fine-grained control via Multi-Reward Classifier-Free Guidance (CFG). By conditioning on a positive reward target $\hat{s}_+$ (high quality) and a negative target $\hat{s}_-$ (low quality), the guidance vector steers generation toward regions where the log-odds ratio of high-to-low rewards is maximized. The authors provide a theoretical interpretation showing this corresponds to sampling from a "reward-tilted" distribution, effectively performing implicit gradient ascent on the log-odds of the rewards.

Key Contributions

Unified Framework: MIRO eliminates the need for separate filtering or post-hoc RLHF stages by learning $p(x|c, s)$ directly during pretraining.
Efficiency and Performance: A 0.36B-parameter MIRO model achieves state-of-the-art results on compositional benchmarks (GenEval) and user-preference metrics (PickScore, ImageReward, HPSv2), surpassing much larger models like FLUX-dev (12B parameters) with significantly less compute.
Convergence Speed: MIRO converges up to 19× faster than baseline pretraining on aesthetic and preference metrics due to dense multi-dimensional quality supervision.
Controllability: The reward vector serves as a controllable input at inference, allowing users to adjust trade-offs between different quality dimensions (e.g., aesthetics vs. semantic alignment) without retraining.

Results

Training Efficiency: On a 16M-image dataset, MIRO reaches baseline performance levels dramatically faster: 19× speedup for AestheticScore, 6.2× for HPSv2, and 3.5× for PickScore.
Compositional Alignment: MIRO improves GenEval scores by 9.6% over the baseline (57 vs. 52), with significant gains in challenging tasks like Color Attribution (+31%) and Two Objects (+24%).
Comparison to SOTA: MIRO (0.36B) outperforms FLUX-dev (12B) on GenEval (68 vs. 67) and PartiPrompts metrics, while requiring 370× less training compute (4.16 TFLOPs vs. 1540 TFLOPs).
Inference Efficiency: When combined with test-time scaling (sample-and-select), MIRO demonstrates superior sample efficiency. For instance, MIRO with 8 samples matches the baseline’s performance with 128 samples on ImageReward (16× efficiency gain).
Mitigation of Reward Hacking: Unlike single-reward models that may over-optimize one metric at the expense of others (e.g., high aesthetics but poor semantic alignment), MIRO’s multi-reward conditioning maintains a balanced performance across all metrics.

Significance
The paper claims that MIRO represents a shift from post-hoc alignment to integrated, reward-aware pretraining. By preserving the full data distribution and learning the relationship between multiple quality dimensions and visual features, MIRO offers a more efficient, diverse, and controllable alternative to traditional RLHF pipelines. The authors highlight that this approach not only improves quality and speed but also provides users with principled control over reward trade-offs at inference time, addressing the rigidity of current alignment methods.

The Problem: The "Three-Stage Factory" Bottleneck

The Solution: MIRO (The "Smart Label" Approach)

How It Works: The "Dial" Metaphor

Why Is This Better? (The Key Benefits)

The Verdict

More like this