🤖 machine learning

LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Models

The paper introduces LEAD, a novel method that employs online, self-adaptive mechanisms to dynamically balance correctness and efficiency during reinforcement learning, thereby enabling large reasoning models to generate significantly shorter Chain-of-Thought outputs without compromising accuracy across diverse mathematical benchmarks.

Original authors: Songtao Wei, Yi Li, Zhikai Li, Xu Hu, Yuede Ji, Guanpeng Li, Feng Chen, Carl Yang, Zhichun Guo, Bingzhe Li

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Songtao Wei, Yi Li, Zhikai Li, Xu Hu, Yuede Ji, Guanpeng Li, Feng Chen, Carl Yang, Zhichun Guo, Bingzhe Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but overly chatty student taking a math test. This student, representing modern Large Language Models (LLMs), is incredibly smart and can solve hard problems. However, they have a bad habit: they tend to "overthink." Even for a simple question like "What is 2+2?", they might write a 10-page essay explaining the history of numbers, the philosophy of addition, and the life story of the number 2, before finally giving the answer "4."

This "overthinking" wastes time, energy, and computer resources. The paper introduces a new training method called LEAD (Length-Efficient Adaptive and Dynamic Reasoning) to teach this student how to be concise without losing their smarts.

Here is how LEAD works, broken down into simple concepts:

The Problem: The "One-Size-Fits-All" Trap

Previous attempts to fix this chatty behavior were like a strict teacher who says, "No matter the question, your answer must be exactly 50 words long."

The Issue: This is unfair. A simple question should be short, but a complex Olympiad-level math problem needs a long explanation. If you force a short answer on a hard problem, the student gets it wrong. If you force a long answer on an easy problem, they waste time.
The Result: Old methods either made the student too short (and wrong) or didn't shorten them enough.

The Solution: LEAD's Two Smart Tricks

LEAD acts like a wise coach who adjusts their teaching style in real-time. It uses two main tricks:

1. The "Dynamic Volume Knob" (Adaptive Rewards)

Imagine the student is being graded on two things: Correctness (getting the right answer) and Brevity (keeping it short).

Old Way: The teacher sets a fixed rule: "50% of your grade is for being right, 50% is for being short." This doesn't work well because early in training, the student needs to focus entirely on learning how to solve the problem. If you pressure them to be short too soon, they stop thinking and start guessing. Later, once they know how to solve it, you want to push them to be shorter.
LEAD's Way: LEAD uses a smart volume knob.
- Early Training: The knob is turned up for "Correctness." The student is allowed to ramble as much as they need to figure out the solution.
- Later Training: Once the student starts getting answers right, the knob automatically shifts. The "Brevity" volume goes up, and the "Correctness" volume goes down (since they already know how to solve it).
- The Magic: The system constantly checks: "Is the student still learning how to be shorter?" If yes, it pushes them. If the student is already short, it stops pushing and focuses on making sure the answer is still correct.

2. The "Personalized Target" (Per-Problem Budget)

Instead of giving every question a fixed word limit, LEAD gives each question its own personalized target length.

How it works: The system looks at the student's successful attempts.
- If the student solved a hard problem correctly in 2,000 words, LEAD says, "Okay, for this specific hard problem, the target is 2,000 words. Don't go under that, or you might be skipping steps."
- If the student solved an easy problem in 200 words, LEAD says, "Great, the target for this one is 200 words."
The Symmetric Reward: LEAD is fair. It doesn't just punish long answers. It also punishes too-short answers. If the student gives a 10-word answer to a hard problem, LEAD says, "That's too short; you probably cheated or guessed." This prevents the student from taking lazy shortcuts.

The Result: The "Goldilocks" Student

When tested on five different math benchmarks, LEAD produced a student who:

Got more answers right than other methods that tried to shorten responses.
Spoke significantly less than the original, chatty model.
Achieved the best balance (called the "Accuracy-Efficiency Score") between being smart and being concise.

The Analogy in a Nutshell

Think of the original model as a tour guide who talks non-stop, even when the tourists just want to see a landmark quickly.

Old methods tried to put a timer on the guide: "Stop talking after 5 minutes." This meant the guide would rush through the complex museums (getting it wrong) or drag out the simple park visits (wasting time).
LEAD is like a smart manager who watches the guide.
- If the guide is struggling to explain a complex painting, the manager says, "Take your time, explain it fully."
- If the guide is explaining a simple statue, the manager says, "You've got the point, wrap it up in two sentences."
- The manager adjusts the rules while the tour is happening, ensuring the guide is always just the right amount of talkative.

In short, LEAD teaches AI models to know when to think deeply and when to be brief, adapting to the difficulty of the task and their own progress, rather than following a rigid, one-size-fits-all rule.

Technical Summary: LEAD (Length-Efficient Adaptive and Dynamic Reasoning)

1. Problem Statement

Large Reasoning Models (LRMs), such as OpenAI o1 and DeepSeek-R1, have demonstrated that explicit Chain-of-Thought (CoT) reasoning significantly improves performance on complex tasks. However, this capability comes with a critical inefficiency: as reasoning capabilities improve, models tend to become increasingly verbose, generating reasoning traces that exceed what is necessary for the underlying problem. This "overthinking" wastes computational resources, increases latency, and consumes context budgets.

Existing approaches to enforce efficiency typically augment Reinforcement Learning (RL) training with length-based efficiency signals alongside correctness rewards. However, the paper identifies two fundamental limitations in current static methods:

Non-Stationary Reward Balance: The optimal balance between correctness and efficiency changes throughout training. Early in training, correctness-oriented exploration is vital, and excessive length pressure can suppress the discovery of valid solutions. Later, as models learn to solve problems, efficiency signals become more useful for removing redundancy. Static reward weights (e.g., a fixed $\lambda_c, \lambda_\ell$ ) cannot adapt to this shifting landscape, leading to a compromise between degraded accuracy or unrealized compression.
Inflexible Global Budgets: Different problems require vastly different amounts of reasoning (e.g., simple arithmetic vs. Olympiad-level math). Applying a single global length budget ( $B$ ) forces a trade-off: setting $B$ aggressively to compress easy problems causes hard problems to be truncated (hurting accuracy), while setting $B$ loosely to preserve hard problems fails to compress easy ones (wasting tokens).

2. Methodology: LEAD

The authors propose LEAD (Length-Efficient Adaptive and Dynamic reasoning), a framework that replaces static heuristics with online, self-adaptive mechanisms. LEAD addresses the two challenges above through two core components:

A. Dynamic Reward Weighting via Potential-Scaled Instability (PSI)

Instead of combining rewards before normalization (which leads to reward collapse) or using static weights, LEAD employs decoupled group normalization followed by online dynamic weighting.

Decoupled Normalization: Following GDPO, LEAD normalizes the correctness reward ( $r_c$ ) and length reward ( $r_\ell$ ) separately within their groups before aggregation. This prevents the higher-variance signal from dominating the advantage calculation.
PSI Controller: The weights $\lambda_c$ $λ_{c}$ and $\lambda_\ell$ $λ_{ℓ}$ are updated online at each training step based on the Potential-Scaled Instability (PSI) of each reward signal.
- Instability: Measured by the coefficient of variation ( $CV_k$ ), indicating how much the reward is changing.
- Potential (Headroom): Measured by $P_k$ , indicating how far the reward is from its theoretical ceiling (e.g., 1.0 for correctness).
- Mechanism: The PSI score $\Psi_k = \widehat{CV}_k \cdot P_k$ is high when a reward is both noisy (informative) and has room to improve. The weights are derived from these scores and smoothed via Exponential Moving Average (EMA).
- Effect: This creates a transient curriculum where length efficiency guides early compression. As the efficiency signal saturates (low headroom), the controller automatically shifts optimization capacity toward correctness, which remains informative on hard prompts.

B. Per-Problem Online Target-Length Calibration

LEAD replaces the global length budget with a per-prompt target length $L^*_q$ , estimated dynamically from the model's own correct rollouts.

Target Estimation: For a prompt $q$ , $L^*_q$ is calculated as the mean length of all correct rollouts in the current batch. If no correct rollouts exist, the target defaults to the maximum budget ( $B_{max}$ ), encouraging exploration rather than premature compression.
Symmetric Efficiency Reward: The length reward $r_\ell$ $r_{ℓ}$ is symmetric around $L^*_q$ $L_{q}^{*}$ . It peaks at 1 when the rollout length equals $L^*_q$ $L_{q}^{*}$ and decreases linearly to -1 for deviations in either direction.
- Penalizing Over-Compression: Crucially, the reward penalizes solutions that are too short (under $L^*_q$ ). This prevents the model from learning "shortcuts" or pattern-matched answers that bypass necessary reasoning steps.
- Adaptivity: As the model learns to solve a problem more concisely, the mean length of correct rollouts decreases, automatically tightening $L^*_q$ without manual intervention.

3. Key Contributions

The paper claims three primary contributions:

Identification of Challenges: It formally identifies two algorithm-agnostic challenges in efficient-reasoning RL: the need for dynamic reward balancing over training and adaptive efficiency across varying problem difficulties, demonstrating that static coefficients fail to resolve these reliably without task-specific tuning.
The LEAD Framework: It proposes a method combining online instability-driven reward weighting (PSI) with per-problem target-length calibration. This approach requires no manual coefficient scheduling or global budget tuning.
Empirical Validation: The method is validated across five mathematical reasoning benchmarks on 1.5B and 7B parameter models, demonstrating consistent improvements in the accuracy-efficiency trade-off over state-of-the-art baselines.

4. Experimental Results

The authors evaluated LEAD on five benchmarks (AIME 2024/25, AMC 2023, MATH-500, OlympiadBench) using DeepSeek-R1-Distill-Qwen-1.5B and 7B models.

Performance: LEAD achieved the highest Accuracy-Efficiency Score (AES) among all tested RL-trained methods.
- On the 1.5B model, LEAD improved accuracy by 2.62 points and AES by 0.18 over the strongest baseline (DRPO), while significantly reducing output length.
- On the 7B model, while all RL methods saw some accuracy regression compared to the base model, LEAD exhibited the smallest drop and the best AES.
Ablation Studies:
- Dynamic vs. Static Weighting: Dynamic weighting significantly outperformed static ratios, confirming that the non-stationary nature of the training process requires adaptive control.
- Target Aggregator: Using the mean of correct rollouts (LEAD) outperformed using the minimum (ShorterBetter's SOL), median, or mean of all rollouts. The "minimum" approach caused accuracy drops by setting unrealistically aggressive targets based on outliers, while the "mean of all" was diluted by incorrect trajectories.
Token Allocation: Analysis showed that LEAD allocates tokens non-uniformly based on difficulty. It preserves more reasoning tokens for harder problems (tracking the base model's difficulty-length correlation with $\rho=0.67$ ) while compressing easier problems, unlike baselines that apply uniform compression.

5. Significance and Claims

The paper positions LEAD as a step toward reasoning models that can adapt their computational footprint online to both problem difficulty and optimization progress.

Efficiency without Sacrifice: The authors claim LEAD achieves a superior trade-off, producing substantially shorter outputs than the base model while maintaining or improving accuracy compared to other efficiency-focused RL methods.
Self-Calibration: A key significance is the elimination of manual hyperparameter scheduling (e.g., tuning $\lambda$ or $B$ ). The system self-calibrates the correctness-efficiency trade-off and the target length for each prompt.
Limitations: The authors modestly note that LEAD is designed for settings where correctness can be reliably verified (e.g., math). Extending it to open-ended generation or subjective tasks may require task-specific reward models. Furthermore, LEAD optimizes the policy during training but does not enforce a hard inference-time token budget; it learns efficient behavior rather than acting as a deployment-time controller.

In summary, LEAD addresses the verbosity of large reasoning models by dynamically balancing learning signals and adapting reasoning budgets per problem, resulting in more efficient reasoning trajectories that do not compromise on correctness.