💻 computer science

EvoLMM: Self-Evolving Large Multimodal Models with Continuous Rewards

EvoLMM is a self-evolving framework that enhances Large Multimodal Models' reasoning capabilities in a purely unsupervised manner by utilizing two cooperative agents to generate image-grounded questions and solve them through internal consistency, achieving significant performance gains on multimodal math benchmarks without relying on annotated data or external reward models.

Original authors: Omkar Thawakar, Shravan Venkatraman, Ritesh Thawkar, Abdelrahman Shaker, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Fahad Khan

Published 2026-03-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Omkar Thawakar, Shravan Venkatraman, Ritesh Thawkar, Abdelrahman Shaker, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Fahad Khan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but shy student named Evo. Evo is incredibly smart at looking at pictures and reading text, but they are stuck in a classroom where the teacher (humans) has stopped giving them homework. The teacher says, "I'm too busy to grade your papers or tell you if your answers are right. You have to learn on your own."

Most students would just sit there and do nothing. But Evo has a secret superpower: Self-Play.

This paper introduces EvoLMM, a system that teaches a computer model how to get smarter without any human help, using a clever trick involving two "personalities" inside the same brain.

The Two Personalities: The Questioner and The Solver

Instead of one brain trying to do everything, EvoLMM splits the model into two cooperative roles that talk to each other:

The Proposer (The Questioner): This personality looks at a raw image (like a chart, a maze, or a geometric diagram) and invents a question about it.
- Analogy: Think of this as a Game Master in a role-playing game. They look at the map (the image) and say, "Okay, here is a puzzle for you to solve!"
The Solver (The Thinker): This personality tries to answer the question.
- Analogy: This is the Player. They look at the map and the puzzle, think hard, and give an answer.

The Magic Ingredient: Continuous Rewards

In the past, if a computer tried to learn this way, it would fail quickly. Here's why:

The Old Way (Discrete Reward): Imagine the Game Master asks a question, and the Player gives an answer. If the answer is perfectly right, the Player gets a gold star (Reward = 1). If it's even slightly wrong, they get a zero (Reward = 0).
- The Problem: In the beginning, the Player is bad at math and logic. They get a "zero" almost every time. It's like trying to learn to ride a bike but getting a shock every time you wobble. You give up because you never get a "good job."
The New Way (Continuous Reward): EvoLMM uses a Continuous Reward.
- The Analogy: Instead of a pass/fail grade, imagine the Game Master gives a smooth, sliding scale of encouragement.
- If the Player gives 5 different answers and 3 of them are similar, the system says, "Hey, you're getting warmer! That's a 0.6 out of 1.0."
- If the answers are all over the place, it says, "That's a 0.2. Keep trying."
- Why it works: Even when the model is wrong, it gets a tiny "nudge" (a gradient) telling it how to move closer to the right answer. It's like a coach saying, "You're off by a few degrees, adjust your aim slightly," rather than just yelling "Wrong!"

The "Self-Evolving" Loop

Here is how the system grows over time, step-by-step:

The Proposer looks at a picture of a maze and asks, "Which path leads to the exit?"
The Solver tries to answer. It might guess 5 different paths.
The Score: The system checks if the 5 guesses agree. If they mostly agree on the same path, the Proposer gets a reward for asking a "good" question. If the Solver's answers are all different, the system knows the question was too hard or confusing, and it adjusts.
The Curriculum: Over time, the Proposer learns to ask questions that are just right—not too easy (where the Solver guesses randomly) and not too hard (where the Solver is confused). It creates its own "school curriculum" automatically.
The Result: The Solver gets better at reasoning, and the Proposer gets better at finding interesting puzzles. They evolve together, getting smarter every day without a single human teacher.

What Did They Discover?

The researchers tested this on Qwen2.5-VL, a powerful AI model. They fed it thousands of raw images (charts, diagrams, math problems) but zero correct answers or labels.

The Outcome: The model improved its math and reasoning skills by about 2% to 3% across many difficult tests.
The Surprise: The model didn't just memorize answers. It learned to think. It started generating more complex questions and solving them with more structured logic, just like a human student who practices on their own.

Why This Matters

Usually, to make AI smarter, we need armies of humans to write questions and grade answers. This is expensive and slow.

EvoLMM is like giving the AI a self-driving learning engine. It can look at the world (images), ask itself questions, check its own work, and get better on its own. It's a giant step toward AI that can learn continuously, just like humans do, without needing a teacher to hold its hand.

In short: EvoLMM is a computer that teaches itself by playing a game of "Question and Answer" with its own brain, using a gentle, continuous feedback loop to ensure it never gets stuck or gives up.

1. Problem Statement

Current Large Multimodal Models (LMMs) have achieved impressive reasoning capabilities but rely heavily on human-curated annotations or externally defined reward models (e.g., human-labeled correctness, knowledge-distilled reward networks) for training. These dependencies create significant bottlenecks:

Scalability: Annotating vast amounts of multimodal data is expensive and time-consuming.
Autonomy: Models cannot improve in domains where ground-truth labels or external evaluators are unavailable.
Generalization: Reliance on specific datasets limits the model's ability to adapt to new, unstructured visual environments.

While recent work in Large Language Models (LLMs) has explored self-improvement via self-questioning and majority-vote rewards, these approaches often fail in the multimodal domain. Multimodal reasoning involves visual uncertainty and complex grounding, making discrete "all-or-nothing" rewards (like exact match voting) too sparse and unstable for effective learning.

2. Methodology: EvoLMM Framework

The authors propose EvoLMM, a purely unsupervised, self-evolving framework that enables an LMM to improve its reasoning capabilities using only raw, unlabeled images. The framework operates without any human labels, metadata, or external reward models.

Core Architecture

The framework instantiates two cooperative agents from a single backbone model (e.g., Qwen2.5-VL):

Proposer ( $\pi_\phi$ ): Generates diverse, visually grounded mathematical or reasoning questions based on the input image.
Solver ( $\pi_\theta$ ): Attempts to answer the generated questions.

The Continuous Self-Reward Mechanism

The key innovation is replacing discrete majority-vote rewards with a continuous self-consistency signal. The training loop proceeds as follows:

Question Generation: The Proposer generates a question $q$ from an image $x$ .
Multi-Sample Solving: The Solver generates $N$ independent answer samples $\{y_1, ..., y_N\}$ for the same question.
Empirical Distribution: An empirical answer distribution $p(a|x, q)$ is computed based on the frequency of answers.
Solver Reward ( $r_{sol}$ ):
- Instead of a binary reward (correct/incorrect), the reward is a continuous function of the agreement score (likelihood) of the specific answer generated.
- It includes a length penalty to encourage conciseness.
- Formula: $r_{sol}^i = (p(y_i|x, q))^\gamma \cdot (1 - \lambda_{len} \cdot \max\{0, (w_i - \tau)/\tau\})$ .
- This provides dense gradients even when the model is only partially correct, stabilizing early-stage learning.
Proposer Reward ( $r_{prop}$ ):
- The Proposer is rewarded based on the entropy ( $H$ ) of the Solver's answer distribution.
- It uses a band-pass function (Gaussian) that peaks at moderate entropy.
- Goal: Encourage questions that are neither too easy (low entropy, trivial) nor too hard/ambiguous (high entropy, unsolvable). This creates an automatic curriculum where the Proposer learns to generate questions that challenge the Solver just enough to stimulate reasoning without causing collapse.

Optimization

Both agents are trained jointly using REINFORCE policy gradients.
KL-Regularization: A token-level KL divergence term is applied to keep the evolving policies close to the pre-trained backbone, preventing catastrophic forgetting or drift.
Moving-Average Baselines: Used to reduce variance in the gradient updates.

3. Key Contributions

Fully Unsupervised Self-Evolution: Introduces a framework where LMMs improve reasoning using only raw images, eliminating the need for annotated Q&A pairs or external reward models.
Continuous Reward Design: Proposes a continuous self-consistency reward mechanism that overcomes the sparsity and instability of discrete majority-vote rewards in multimodal settings. This allows for smooth gradient flow and stable optimization.
Automatic Curriculum Emergence: Demonstrates that the entropy-based Proposer reward naturally induces a curriculum, where question difficulty evolves from simple to complex as the Solver's capabilities improve.
Model-Agnostic Scalability: Shows that the framework works effectively across different model families (Qwen, InternVL, Gemma, Llama) and scales from 7B to 72B parameters.

4. Experimental Results

The framework was evaluated on the Qwen2.5-VL-7B backbone and compared against the baseline and a discrete-reward adaptation.

Performance Gains: EvoLMM achieved consistent improvements of ~2–3% across multiple multimodal math-reasoning benchmarks:
- ChartQA: 84.00% $\to$ 86.70% (+2.7%)
- MathVista: 68.46% $\to$ 70.52% (+2.06%)
- ScienceQA: 88.30% $\to$ 89.50% (+1.2%)
- MathVision: 23.91% $\to$ 24.81% (+0.9%)
Comparison with Discrete Rewards: The discrete majority-vote baseline showed negligible gains or performance degradation, confirming the instability of discrete rewards in early training stages.
Generalization: The method generalized to other backbones (InternVL3-8B, Gemma3-12B, Llama-3.2-11B) and larger scales (Qwen2.5-VL-72B), with larger models showing even stronger absolute gains.
Zero-Shot Transfer: The self-evolved model showed improvements on open-ended benchmarks (RefCOCOg, DocVQA, OCRBench) despite being trained only on chart/math images, indicating enhanced general reasoning capabilities.

5. Significance and Impact

Autonomy in AI: EvoLMM represents a significant step toward autonomous AI, demonstrating that models can self-improve in complex visual domains without human intervention.
Scalability: By removing the dependency on expensive human annotations, this approach offers a scalable path for training LMMs on the vast amounts of unlabeled visual data available on the internet.
Stability in Reasoning: The continuous reward mechanism solves a critical bottleneck in self-play RL for multimodal tasks, providing a stable learning signal where traditional discrete rewards fail.
Future Directions: The work suggests that "self-evolving" agents can emerge with structured reasoning behaviors purely through internal consistency, paving the way for open-ended, long-horizon reasoning systems.

In summary, EvoLMM proves that large multimodal models can autonomously refine their reasoning capabilities through a cooperative Proposer-Solver dynamic driven by continuous self-consistency, achieving state-of-the-art unsupervised improvements on complex visual reasoning tasks.