🤖 AI

MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop

The paper introduces MulFeRL, a multi-turn reinforcement learning framework that enhances reasoning by converting rich verbal feedback on failed samples into trainable learning signals, thereby outperforming existing supervised and RLVR baselines in both in-domain and out-of-domain tasks.

Original authors: Xuancheng Li, Haitao Li, Yujia Zhou, YiqunLiu, Qingyao Ai

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xuancheng Li, Haitao Li, Yujia Zhou, YiqunLiu, Qingyao Ai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Silent Failure" of AI

Imagine you are teaching a student to solve complex math problems. You give them a test, and they get an answer wrong.

In standard AI training (called RLVR), the teacher just says, "Wrong." That's it. No explanation, no hint, just a zero score.

The Issue: If the student gets 100 problems wrong, they get 100 "Wrong" scores. They have no idea why they failed. Did they add the numbers wrong? Did they misunderstand the question? Did they skip a step?
The Result: The AI gets stuck. It keeps guessing randomly because the "learning signal" (the feedback) is too sparse and unhelpful. It's like trying to learn to drive by only being told "Crash" when you hit a wall, without ever being told you forgot to look in the rearview mirror.

The Solution: MulFeRL (The "Coach with a Clipboard")

The researchers propose MulFeRL (Multi-turn Feedback-guided Reinforcement Learning). Instead of just saying "Wrong," this system acts like a tough but helpful coach who gives specific, verbal advice.

Here is how it works, step-by-step:

1. The "All-Failed" Trap

Usually, if an AI tries to solve a hard problem and fails, it stops. The training loop breaks because there is no "good" answer to compare against the "bad" one.

MulFeRL's Move: When the AI fails completely, the system doesn't give up. It pauses and asks a "Feedback Provider" (like a smart human or a stronger AI) to look at the mess.

2. The "Verbal Feedback" (The Coach's Notes)

The Feedback Provider doesn't just say "Wrong." It writes a note:

"You tried to use the Pythagorean theorem, but you forgot to square the numbers first."
"You missed a negative sign in step 3."
This is the Verbal Feedback. It turns a vague failure into a specific lesson.

3. The "Regeneration" (The Do-Over)

Now, the AI gets a second chance. It takes the original problem plus the coach's notes and tries to solve it again.

The Magic: If the AI uses the notes correctly, it might finally get the right answer.
The Credit: The system now knows: "Ah! When we gave the AI the specific note about squaring numbers, it succeeded." This turns the "failure" into a "success" that can be learned from.

4. Two Ways to Learn (The Scorecard)

The paper introduces two specific ways to grade this new process:

Scenario A: The "Mixed Bag" (GRPO)
Sometimes, after getting the notes, the AI tries 8 times. Some attempts are still wrong, but some are right.
- Analogy: It's like a sports team where some players messed up but others played great. The coach can say, "Look at the players who succeeded; copy their moves." The AI learns by comparing its own good attempts against its own bad attempts.
Scenario B: The "Total Turnaround" (FCO)
Sometimes, the notes are so good that all 8 attempts become correct.
- Analogy: The whole team suddenly plays perfectly. In standard training, this is confusing because there are no "bad" examples to compare against.
- MulFeRL's Trick: It looks at the before and after. It compares the "Before" (where everyone failed) with the "After" (where everyone succeeded). It tells the AI: "Remember how you failed before? Remember how you succeeded after the notes? Do more of the 'After' version." This is called Feedback-Contrastive Optimization (FCO).

5. The "Fixed Slot" (The Sticky Note)

To make sure the AI actually reads the notes, MulFeRL doesn't just paste the feedback at the bottom of the page. It puts the feedback in a fixed, special box (like a <feedback> tag) right in the middle of the thinking process.

Analogy: It's like a student having a sticky note on their calculator that says "Check your signs!" rather than just reading a report card at the end of the week. It forces the AI to look at the advice while it's working.

Why This Matters

The paper shows that by using this "Coach with a Clipboard" approach:

It fixes the "Silent Failure" problem: It finds a way to learn even when the AI gets everything wrong initially.
It learns faster: The AI improves more quickly because it gets specific instructions on how to fix its mistakes, not just a score.
It works on new things: Even though it was trained on math problems, it got better at science and general reasoning tasks too, proving it learned a better way of thinking, not just memorized math answers.

Summary

MulFeRL is a method that stops AI from getting stuck when it fails. Instead of ignoring failed attempts, it uses verbal feedback to guide the AI to try again. It then rewards the AI not just for getting the answer right, but for improving based on the feedback. It turns "I failed" into "Here is how I fixed it," making the learning process much richer and more effective.

Technical Summary: MulFeRL

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard approach for enhancing reasoning in Large Language Models (LLMs), particularly in mathematics and code generation. However, standard RLVR relies on outcome-only scalar rewards (e.g., binary correct/incorrect signals). The authors argue that as model performance increases, this approach hits a bottleneck due to the sparsity and uninformative nature of scalar rewards, especially for failed samples.

When a model generates a large fraction of entirely incorrect solutions, standard RLVR assigns near-zero or negative rewards that merely indicate failure without explaining why the reasoning broke down or how to improve. This leads to:

Poor credit assignment: Gradient estimates become noisy.
Training plateaus: The learning signal collapses for "all-failed" rollout groups where no within-group contrast exists.
Sample inefficiency: Training struggles to progress on difficult problems where scalar rewards provide no usable contrast.

While humans leverage verbal feedback to localize errors and close gaps to target solutions, existing RL-style feedback methods often allocate feedback broadly across samples rather than targeting the specific "all-failed" cases where outcome rewards are least informative.

Methodology: MulFeRL

The paper proposes MulFeRL (Multi-turn Feedback-guided Reinforcement Learning), an event-triggered, multi-turn RLVR framework designed to convert verbal feedback into stable, trainable learning signals for failed samples. The framework operates through two primary stages: Progress Induction and Progress Credit Assignment.

1. Event-Triggered Progress Induction

MulFeRL targets rollout groups where all $K$ samples fail verification (the FAILED state). In this state, standard RLVR provides no learning signal.

Feedback as Conditioning: Instead of optimizing feedback directly, MulFeRL uses verbal feedback as a conditioning intervention. When a group is in the FAILED state, a feedback provider (e.g., an LLM simulator) generates actionable feedback identifying dominant failure modes and suggesting revisions.
Regeneration: This feedback is injected into the context, and the model regenerates a new group of $K$ rollouts.
Goal: The objective is to induce a transition from a FAILED state to either a CONTRASTIVE state (mixed success/failure) or a SOLVED state (all successful). This transition constitutes "verifier-certified progress."

2. Progress Credit Assignment

Once progress is induced, MulFeRL assigns credit based on the nature of the transition using two complementary objectives:

Within-State Credit (GRPO): If the regenerated group is CONTRASTIVE (contains both correct and incorrect answers), MulFeRL applies standard Group Relative Policy Optimization (GRPO). This exploits the relative advantage between successful and failed rollouts within the same feedback-conditioned context.
Cross-State Credit (FCO): If the regenerated group becomes SOLVED (all correct), within-group contrast vanishes (all rewards are 1), rendering standard GRPO ineffective. To recover the learning signal, MulFeRL introduces Feedback-Contrastive Optimization (FCO). FCO constructs pairs between the current SOLVED group and the immediately preceding FAILED group. It optimizes the policy to increase the relative score of the successful trajectories (conditioned on feedback) compared to the failed trajectories (conditioned on the original prompt), effectively treating the transition as a preference learning task.

3. Structured Feedback Injection

To ensure the model effectively utilizes feedback during regeneration, MulFeRL employs a fixed-slot injection mechanism. Verbal feedback is inserted into a dedicated <feedback> tag within the reasoning template (e.g., inside a <thinking> block) rather than appended to the prompt. This design:

Treats feedback as an explicit conditioning signal.
Embeds feedback tightly into the reasoning context.
Allows for precise token masking during training, ensuring the model learns to reason given the feedback without optimizing the feedback tokens themselves.

Key Contributions

Novel Framework for Low-Signal Regimes: MulFeRL addresses the specific limitation of RLVR on "all-failed" samples by introducing a multi-turn loop that uses verbal feedback to induce progress, converting previously uninformative failures into verifier-certified learning opportunities.
Feedback-Contrastive Optimization (FCO): The paper introduces a new objective to handle "reward-saturated" progress (FAILED $\to$ SOLVED transitions). This allows the model to learn from cases where feedback successfully resolves all failures, a scenario where standard within-group contrast methods fail.
Structured Feedback Integration: The authors demonstrate that injecting feedback into a fixed reasoning slot, rather than simple prompting, significantly improves the model's ability to condition on and utilize feedback for regeneration.
Event-Triggered Routing: The framework dynamically routes training signals based on the verifier's observation of the group state (CONTRASTIVE vs. SOLVED vs. FAILED), ensuring updates are only applied when a usable discrimination signal exists.

Experimental Results

The authors evaluated MulFeRL on a sampled subset of OpenR1-Math and tested performance on five mathematical reasoning benchmarks (AMC 2023, AIME 2024, OlympiadBench, MATH-500, Minerva-Math) and three out-of-domain (OOD) science/general reasoning tasks (MMLU-Pro, GPQA-Diamond, TheoremQA).

Performance Gains: MulFeRL consistently outperformed strong baselines, including Supervised Fine-Tuning (SFT), Self-Distillation (SDPO), and various RL-based methods (GRPO, Dr.GRPO, Critique-GRPO, RLTF-FM).
- On the Qwen2.5-7B-Base backbone, MulFeRL achieved an average Pass@1 of 46.26%, significantly surpassing the best RL baseline (Critique-GRPO at 42.83%).
- On the Qwen3-4B-Inst backbone, MulFeRL reached 68.36% average Pass@1, outperforming Critique-GRPO (63.65%) and other baselines.
Generalization: The model showed strong out-of-domain generalization, improving performance on science and general reasoning tasks despite being trained primarily on math data.
Ablation Studies: Removing core components led to performance degradation:
- Removing feedback-guided regeneration caused the largest drop, confirming the necessity of inducing progress from failed states.
- Removing FCO substantially hurt performance, validating its role in capturing FAILED $\to$ SOLVED transitions.
- Removing structured feedback injection led to consistent declines, highlighting the importance of the fixed-slot design.
Simulator Robustness: The method remained effective even when using weaker feedback simulators (e.g., the base model itself or GPT-4o-mini), though stronger simulators (GPT-4o) yielded further gains.

Significance and Claims

The paper claims that MulFeRL provides a robust solution to the "sparse reward" bottleneck in RLVR by leveraging the richness of verbal feedback without falling into the pitfalls of direct feedback optimization or simple imitation.

Mechanism: The authors emphasize that the improvement stems not merely from exposure to feedback, but from the specific mechanism of progress induction (turning failures into successes via feedback) and feedback-aware credit assignment (using FCO to learn from these transitions).
Efficiency vs. Cost: The authors acknowledge that MulFeRL incurs higher training costs due to multi-turn regeneration and verification overhead. However, they argue that the method is more sample-efficient, reaching strong performance in fewer optimization steps because it extracts denser learning signals from failed samples that would otherwise be discarded.
Limitations: The paper modestly notes that the approach depends on reliable outcome verification and useful feedback. It also highlights that the current design focuses on rollout-level feedback for all-failed groups, potentially missing finer-grained improvements in partially correct solutions, suggesting future work could extend to step-level feedback.

In conclusion, MulFeRL demonstrates that transforming verbal feedback into verifier-gated learning signals allows RL training to continue progressing on difficult, failed samples, leading to superior reasoning capabilities across both in-domain and out-of-domain tasks.