Empowering Small VLMs to Think with Dynamic Memorization and Exploration

This paper introduces DyME, a novel training paradigm that dynamically balances Supervised Fine-Tuning and Reinforcement Learning with Verifiable Reward, complemented by a synergistic Visual Supervision mechanism, to effectively empower Small Vision-Language Models with reliable thinking capabilities while overcoming the limitations of existing training methods.

The Gist

Imagine you have a brilliant but very small student (a Small Vision-Language Model, or SVLM). This student is great at looking at pictures and answering simple questions, but they struggle with complex reasoning. They are like a smart kid who can read a map but gets lost if you ask them to plan a multi-step road trip.

The paper introduces a new training method called DyME (Dynamic Memorization and Exploration) to teach this small student how to "think" before answering.

Here is the breakdown using simple analogies:

1. The Problem: Two Bad Teachers

Previously, researchers tried to teach these small models using two main methods, but both failed for the "small student":

• Teacher A (SFT - Supervised Fine-Tuning): This teacher forces the student to memorize long, perfect essays written by geniuses.
  • The Result: The small student tries to memorize the essay word-for-word but gets so overwhelmed by the text that they forget to look at the picture. They start "hallucinating" (making things up) because they are just reciting a script without understanding the image.
• Teacher B (RLVR - Reinforcement Learning): This teacher says, "Go figure it out yourself! Try different ways to solve the problem, and I'll give you a gold star if you get the right answer."
  • The Result: The small student gets confused. They try random guesses, get lost, and eventually stop trying because they can't figure out how to get the gold star. They collapse under the pressure of "exploring" without a guide.

2. The Solution: The "Smart Switch" (DyME)

The authors realized that the small student needs both teachers, but not at the same time in a fixed way. They need a Dynamic Switch that changes the teaching style based on how the student is doing right now.

• When the student is stuck or confused: The system switches to Memorization Mode (SFT). It gives the student a clear, simple example to copy. This stabilizes them and prevents them from panicking.
• When the student gets it right: The system switches to Exploration Mode (RL). It says, "Great job! Now try to solve a similar problem on your own to get even better." This encourages them to think creatively without getting lost.

The Analogy: Think of it like a video game with a dynamic difficulty setting.

• If you are failing a level, the game gives you a hint or a power-up (Memorization) so you don't quit.
• If you are winning, the game removes the hints and makes the level slightly harder (Exploration) so you keep improving.
• DyME does this automatically, step-by-step, ensuring the small model never gets too overwhelmed or too bored.

3. The Secret Weapon: The "Visual Fact-Checker"

The paper adds a special helper called Visual Supervision.

Imagine the student is trying to describe a picture of a cat.

• Without the helper: The student might say, "The cat is happy." (Vague, maybe wrong).
• With the helper: The helper looks at the picture and says, "Wait, look at the ears! They are pointed up. Look at the tail! It's twitching. Based on these specific facts, the cat is alert."

The system forces the model to extract specific "Visual Facts" (like the color of the shirt, the number on a chart, the angle of a line) before it is allowed to write its answer. This stops the model from making things up and forces it to ground its thoughts in reality.

4. The Result: Small Models, Big Brains

By using this "Smart Switch" and the "Fact-Checker," the paper shows that tiny, efficient models (which can run on a laptop or phone) can suddenly solve complex problems like:

• Reading medical X-rays.
• Interpreting complex business charts.
• Solving geometry problems.

In a nutshell:
DyME is like a personal trainer for a small robot. Instead of forcing it to lift heavy weights (memorizing huge datasets) or throwing it into the deep end (random exploration), the trainer watches the robot's form. If the robot struggles, the trainer gives a spotter's help. If the robot is strong, the trainer encourages a new challenge. The result is a small, efficient robot that can think clearly and reliably.

Technical Summary

1. Problem Statement

Small-scale Vision-Language Models (SVLMs), typically with fewer than 1 billion parameters, are crucial for resource-constrained edge devices but struggle to develop "thinking" (reasoning) capabilities under existing training paradigms. The paper identifies two dominant paradigms that fail when applied directly to SVLMs:

• Supervised Fine-Tuning (SFT) on Chain-of-Thought (CoT): SVLMs lack the capacity to memorize verbose, long-text CoT traces without compromising their visual grounding. This leads to "pseudo thinking traces" where the model mimics the format but hallucinates intermediate values or ignores the image.
• Reinforcement Learning with Verifiable Reward (RLVR): While RL encourages exploration, SVLMs often fail to adhere to strict output formats (e.g., specific tags for answers). This results in "advantage collapse," where the reward signal becomes noisy or zero, causing training instability.

Core Challenge: Existing hybrid approaches (e.g., two-stage training: SFT followed by RL) rely on static trade-offs governed by hyperparameters. Due to the extremely limited capacity of SVLMs, the window for a successful static balance is too narrow, making these methods brittle and prone to performance degradation below the baseline.

2. Methodology: DyME (Dynamic Memorization and Exploration)

The authors propose DyME, a novel training paradigm that dynamically switches between Memorization (SFT) and Exploration (RLVR) at every optimization step based on the model's current generation quality.

A. Dynamic Switching Mechanism

The core of DyME is a state-driven decision rule applied at each training step:

1. Generation: The SVLM generates $K$ responses for an input (image + instruction).
2. Verification: Each response is parsed into a thinking trace and a final answer. The answers are verified against ground truth using rule-based checks.
3. Mode Selection:
   • Exploration Mode (RLVR/GRPO): If at least one response is correct, the model enters RL mode. It uses Group Relative Policy Optimization (GRPO) to explore diverse, grounded thinking patterns, leveraging the relative advantage of correct generations.
   • Memorization Mode (SFT): If all responses are incorrect (or fail to parse), the model switches to SFT mode. It is trained to memorize the ground-truth CoT trace. This provides a low-variance, stable gradient to prevent the model from drifting into hallucination.

This binary switching ensures that the model never explores when it is completely lost (preventing collapse) and never just memorizes when it has found a solution (preventing rigidity).

B. Synergistic Visual Supervision

To further enhance performance, DyME introduces a Visual Supervision module comprising a Visual Checker and a Visual Refiner:

• Visual Facts ($I_c$): Fine-grained visual elements (objects, attributes, states) are automatically extracted from images using tools or LLMs.
• Visual Refiner: During SFT mode, this module reconstructs the ground-truth CoT. It takes the raw answer and injects the extracted visual facts, forcing the ground truth to be "image-grounded" rather than just text-based. This reduces the learning burden on the SVLM.
• Visual Checker: During RL mode, this module scores the model's generated thinking traces. It evaluates whether the trace correctly utilizes the visual facts ($I_c$) and follows the required structure. High-scoring traces receive higher rewards, encouraging the model to generate reasoning that is explicitly tied to visual evidence.

3. Key Contributions

1. First SVLM Thinking Paradigm: DyME is the first training framework specifically designed to equip SVLMs with reliable reasoning capabilities, significantly reducing reliance on the base model's initial capacity.
2. Dynamic Switching: It solves the SFT vs. RL trade-off by replacing static hyperparameter tuning with a dynamic, output-driven switching mechanism, preventing both pseudo-thinking and advantage collapse.
3. Visual Grounding: The synergistic visual supervision mechanism ensures that reasoning traces are not just syntactically correct but semantically grounded in the image, effectively filtering out hallucinations.
4. Data Efficiency: The method achieves substantial gains using only a few thousand training samples, making it practical for specialized tasks without requiring massive datasets.

4. Experimental Results

The authors evaluated DyME across three diverse domains: Medical VQA (SLAKE), Chart Understanding (ChartQA), and Geometric Problem Solving (MathVerse/Geo170K).

• Performance Gains:
  • SmolVLM (0.5B): Improved from a baseline of 49.9% to 55.6% (+5.7%) on average. In contrast, standard SFT dropped performance to 44.1%, and Two-stage training failed to improve significantly.
  • LLaVA-OV-S (0.5B): Improved from 50.7% to 55.4% (+4.7%).
  • InternVL2-S: Improved from 56.3% to 58.1% (+1.8%).
• Comparison to LVLMs: DyME-trained SVLMs (e.g., SmolVLM at 55.6%) achieved performance comparable to or surpassing larger Large-scale VLMs (LVLMs) like MoVA (54.2%) on these specialized tasks.
• Data Quality Robustness: DyME trained on "Medium" quality (semi-structured) data outperformed SFT trained on "High" quality (GPT-4o generated) data, demonstrating superior data efficiency.
• Cost-Effectiveness: The "Full DyME" pipeline (using open-source Qwen2.5-14B for supervision) achieved results comparable to using expensive GPT-4o for data construction, eliminating the need for proprietary data annotation.

5. Significance

• Democratizing Reasoning: DyME makes advanced reasoning capabilities accessible for small, efficient models, enabling their deployment on edge devices where large models cannot run.
• Stability in Training: By dynamically adapting to the model's learning state, DyME provides a robust training signal that prevents the common failure modes of SVLMs (hallucination and training collapse).
• Practical Application: The framework is particularly effective for structured, semi-structured tasks (charts, medical reports, geometry) where visual facts can be explicitly defined, offering a practical solution for industry applications requiring reliable, low-cost AI.

In conclusion, DyME represents a paradigm shift from static, capacity-heavy training to a dynamic, adaptive approach that allows small models to "think" reliably by balancing the safety of memorization with the potential of exploration.