🤖 machine learning

Do Not Imitate, Reinforce: Iterative Classification via Belief Refinement

The paper proposes Reinforced Iterative Classification (RIC), a method that replaces standard supervised imitation with a reinforcement learning approach to enable an anytime classifier that adaptively allocates computation and achieves better calibration by iteratively refining its predictions.

Original authors: Mahdi Kallel, Johannes Tölle, Ahmed Hendawy, Carlo D'Eramo

Published 2026-04-27

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Mahdi Kallel, Johannes Tölle, Ahmed Hendawy, Carlo D'Eramo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are taking a multiple-choice exam.

The "Standard" Way (Supervised Learning):
Most AI models are like a student who is forced to circle an answer the very millisecond they finish reading the question. They don't have time to pause, look back, or rethink. Because they are trained to "mimic" the perfect answer key, they often become "know-it-alls." Even if they are guessing, they circle the answer with absolute, unshakable confidence. This makes them prone to "overconfidence"—they might be dead wrong, but they are 100% sure they are right.

The "RIC" Way (Reinforced Iterative Classification):
The researchers propose a new way called RIC. Instead of a single, frantic guess, RIC turns the AI into a "thinker."

Think of RIC as a student who is allowed to use a scratchpad. They look at the question, make an initial guess, and then—instead of stopping—they look at their own guess and ask, "Wait, does this actually make sense? Can I do better?" They refine their answer step-by-step, getting better and better with each "thought" until they feel they’ve reached the best possible conclusion.

The Three Big Breakthroughs

1. The "Anytime" Expert (The Progressive Sketch)
Imagine an artist drawing a portrait. In the standard way, you only see the finished painting. If you stop the artist halfway, you just have a mess.
RIC is like an "anytime" artist. Even if you stop them after 30 seconds, you have a rough sketch. After a minute, you have a clear outline. After five minutes, you have a masterpiece. Because the AI is trained to improve step-by-step, it provides useful answers at any point in its thinking process.

2. The "Healthy Skeptic" (Better Calibration)
Standard AI models suffer from "The Dunning-Kruger Effect"—they are often the most confident when they are the most wrong.
Because RIC is trained through Reinforcement Learning (learning from rewards), it learns the value of uncertainty. If an image is blurry or confusing, the AI doesn't just shout a random answer; its "internal compass" tells it, "I'm not quite sure yet." This makes the AI much more "calibrated"—meaning when it says it is 80% sure, it is actually right about 80% of the time.

3. The "Smart Worker" (Adaptive Computation)
Standard AI is like a factory machine that runs at the exact same speed for every item on the belt, whether it's a simple bolt or a complex engine. This wastes a lot of energy on the easy stuff.
RIC is like a smart worker. If the task is easy (like identifying a bright red apple), the worker finishes in one second and moves on. If the task is hard (like distinguishing between two very similar species of birds), the worker slows down and spends more time "thinking" and refining the answer. It knows exactly when to stop because it can sense when further thinking won't actually help.

Summary

In short, instead of teaching AI to imitate a perfect answer in one shot, this paper teaches AI to reason through an answer. This makes the AI smarter, more honest about what it doesn't know, and much more efficient with its "brainpower."

Technical Summary: Reinforced Iterative Classification (RIC)

1. Problem Statement

Standard supervised classification is fundamentally limited by its "imitation" nature. It treats classification as a single-step behavior cloning task where a model is trained to mimic an oracle label in a single forward pass. This paradigm suffers from two primary pathologies:

Fixed Compute Budget: The model applies the same amount of computation to every input, regardless of whether the image is simple (easily resolvable) or highly complex/ambiguous.
Pathological Overconfidence: The standard cross-entropy objective encourages the model to push logit norms toward infinity to minimize loss on separable data. This leads to poorly calibrated models that express extreme certainty even when they are incorrect, a problem that persists during evaluation.

2. Methodology

The authors propose Reinforced Iterative Classification (RIC), which reframes classification from a single-step imitation task into a sequential decision-making process (a "Thought MDP").

MDP Formulation: Instead of a single prediction, a recurrent agent maintains an internal "thought state" ( $\tau_t$ ) and iteratively refines a continuous predictive distribution (a belief) over the class simplex ( $\Delta_{K-1}$ ).
Reward Mechanism: Rather than using a sparse label at the end of a sequence, RIC uses an incremental reward based on the stepwise improvement in the log-score (the log-probability assigned to the true class $y$ ). The reward at step $t$ is defined as $r_t = \log a_{t,y} - \log a_{t-1,y}$ .
Architecture: The framework employs an Actor-Critic architecture. A recurrent backbone (e.g., GRU) updates the thought state; the Policy Head outputs a Dirichlet distribution over class probabilities, and the Value Head estimates the expected future improvement (the value function).
Adaptive Computation: During inference, the model uses the learned value function as a natural halting signal. If the predicted value $V(s_t)$ falls below zero (indicating that further computation is unlikely to improve the prediction), the model terminates and returns the current belief.

3. Key Contributions

Theoretical Framework: The authors prove that the discounted RL objective is mathematically equivalent to an anytime classifier (maximizing the expected log-score at a geometrically distributed random stopping time).
Optimization Dynamics:
- Proposition 1: They show that under realizability, the optimal policy still recovers the true class probabilities, matching the target of standard cross-entropy.
- Proposition 2: They demonstrate that because the shared classifier must also evaluate early, unrefined (non-separable) states, the objective inherently prevents unbounded logit growth, leading to a finite logit-scale maximizer.
Principled Halting: Unlike prior adaptive methods (like PonderNet) that require explicit penalties or auxiliary networks, RIC’s halting mechanism emerges naturally from the value function.

4. Results

The model was evaluated on CIFAR-10, SVHN, and ImageWoof.

Accuracy: RIC achieves competitive accuracy compared to standard supervised learning (SL). While SL may learn faster, RIC shows better generalization (higher validation accuracy for a given level of training accuracy).
Calibration: RIC significantly outperforms SL and other adaptive baselines (ACT, PonderNet) in calibration. It produces a more dispersed confidence distribution and is much more robust to label noise (tested on CIFAR-10N).
Adaptive Behavior: Empirical analysis on ImageWoof confirms that the model allocates more refinement steps to "intermediate" difficulty images and terminates early on "easy" ones. The value function successfully concentrates computation on resolvable inputs.

5. Significance

This paper shifts the paradigm of classification from imitation (mimicking a label) to refinement (improving a belief). By treating classification as a sequential process, the authors provide a structural solution to overconfidence and a principled method for adaptive computation. This approach suggests that "thinking" (iterative refinement) can be optimized via reinforcement learning to balance accuracy, calibration, and computational efficiency.