💬 NLP

DND: Boosting Large Language Models with Dynamic Nested Depth

This paper introduces Dynamic Nested Depth (DND), a novel post-training method that enhances off-the-shelf LLMs by dynamically identifying and reprocessing critical tokens through a nested depth mechanism, achieving significant performance gains on diverse benchmarks with minimal computational overhead.

Original authors: Tieyuan Chen, Xiaodong Chen, Haoxing Chen, Zhenzhong Lan, Weiyao Lin, Jianguo Li

Published 2026-01-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tieyuan Chen, Xiaodong Chen, Haoxing Chen, Zhenzhong Lan, Weiyao Lin, Jianguo Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a student taking a very difficult exam. You have a limited amount of time and mental energy.

The Old Way (Standard AI):
Most Large Language Models (LLMs) today work like a student who treats every single question on the test exactly the same way. Whether the question is "What is 2+2?" or "Explain the quantum mechanics of a black hole," the student spends the exact same amount of time and brainpower on both. They read the easy question, think for a second, write the answer, and move on. They do the same for the hard question, but because they are using the same "one-size-fits-all" effort, they might rush through the hard part and get it wrong, or waste time overthinking the easy part.

The New Idea (DND):
The paper introduces a method called Dynamic Nested Depth (DND). Think of this as a smart tutor who watches the student take the test and intervenes only when necessary.

Here is how it works, broken down into simple steps:

1. The "Traffic Cop" (The Router)

As the model processes a sentence, it hits a "traffic cop" at the end of every layer (a step in its thinking process). This cop looks at each word (token) individually.

Easy words: If the word is simple (like "the" or "and"), the cop says, "You're good, move along." The model processes it normally and moves to the next step.
Hard words: If the word is tricky (like a complex math symbol or a logical connector in a riddle), the cop says, "Wait! This one needs more attention."

2. The "Detour" (Nested Depth)

When the cop flags a hard word, it doesn't just let it pass. Instead, it sends that specific word on a detour.

Imagine the word is sent back into the classroom for a second round of study. It gets re-read, re-analyzed, and "reviewed" by the model's internal logic.
This is the "Nested Depth." The model digs deeper into just those difficult words without wasting time on the easy ones. It's like a student re-reading a confusing paragraph in a book while skimming the easy parts.

3. The "Merging" (Fusion)

After the hard words have had their extra "review," they are brought back to the main line. The model then combines the original understanding with this new, deeper understanding to create the final answer.

Why is this special?

The paper claims this method is a "plug-and-play" upgrade. You don't need to rebuild the whole school (the model) from scratch. You just add this smart traffic cop system to existing models (like Qwen, Llama, or Gemma) and train it for a short while.

The Results (The Report Card):
The authors tested this on several different models:

Small Models: They took small models (like 1 billion parameters) and made them significantly smarter. For example, one model improved its reasoning and coding scores by about 2.5% to 2.6%.
Big Models: They even tested it on a massive, complex model (30 billion parameters). It improved that model's performance by nearly 1%.
Efficiency: The best part is that this didn't make the model much slower or bigger. It only added a tiny amount of extra "brainpower" (parameters) and computation. It's like getting a better grade without needing to study for twice as long.

The Secret Sauce (Training Strategy)

The paper also explains that teaching the "traffic cop" to be accurate is hard. If the cop is too picky, it stops everything; if it's too lazy, it stops nothing.

The Solution: They created a special training rule. They taught the cop to be very good at telling the difference between "easy" and "hard" words (so it doesn't get confused) and gave it a dynamic way to adjust its own strictness (the threshold) so it always picks the right amount of words to review.

In Summary:
DND is a way to make AI smarter by letting it spend extra time only on the hard parts of a sentence, while breezing through the easy parts. It's a smarter, more efficient way to think, rather than just thinking harder at everything.

Technical Summary: Dynamic Nested Depth (DND) for Large Language Models

1. Problem Statement

The prevailing paradigm for improving Large Language Models (LLMs) relies on scaling parameters, data, and computation. However, this approach incurs exponential computational overhead for both training and inference. Recent insights suggest that prediction difficulty varies significantly across tokens; while many tokens are trivial to predict, others require deep computational processing. Existing methods like token pruning reduce overhead by discarding uninformative tokens but operate on a binary "keep or discard" basis. Furthermore, while test-time scaling strategies (e.g., Latent Strategies) show that recurrent computation in hidden states can improve reasoning, they often lack mechanisms to dynamically allocate this extra depth only to the most critical tokens without incurring redundant costs for simpler inputs. There is a need for an efficient approach that adaptively allocates additional computation to difficult tokens while preserving the efficiency of standard processing for others, ideally as a post-training method compatible with existing architectures.

2. Methodology: Dynamic Nested Depth (DND)

DND is a post-training method that integrates a dynamic nested depth mechanism into existing Transformer layers. It operates by identifying critical tokens and subjecting them to an additional round of processing ("review") within the same layer, while allowing easier tokens to pass through normally.

2.1 Architecture Design

The DND strategy is applied to intermediate layers of the model (from layer $L_s$ to $L_e$ ), leaving the initial and final layers unchanged to preserve pre-trained reasoning patterns. The architecture consists of three main components:

Token-Choice Routing: Unlike "expert-choice" routing (Top-k over the whole sequence) which risks information leakage in auto-regressive models, DND employs a token-choice strategy. A linear router $R$ processes the hidden state $x_i^v$ of each token from the vanilla forward pass to generate a preference score. This score is normalized via a sigmoid function to produce a probability $p_i \in (0, 1)$ . A token is selected for reprocessing if and only if $p_i > \tau$ , where $\tau$ is a predefined threshold.
Nested Depth Processing: Selected tokens are masked and packed into a compact sequence. This sequence undergoes a second forward pass through the same transformer layer $L_i$ , effectively creating a "nested" depth. The output of this pass, $X_d$ , represents a refined hidden state for the critical tokens.
Normalized Fusion: The final output for a token is a fusion of the vanilla output ( $X_v$ ) and the nested output ( $X_d$ ). For selected tokens, the fusion is governed by a gating mechanism: $x_i = (\beta \cdot p_i) \cdot x_i^d + (1 - \beta \cdot p_i) \cdot x_i^v$ , where $\beta$ is a learnable balancing parameter. Unselected tokens retain their vanilla output.

2.2 Training Strategies

To ensure the router learns to distinguish tokens effectively and the selection remains stable, the authors propose two novel training strategies:

Router Controlling Loss: To prevent router scores from clustering (which leads to unstable selection) or saturating (causing gradient vanishing), a dual-objective loss is introduced:
- Score Dispersion Loss ( $L_{sd}$ ): Based on information entropy, this encourages scores to spread across a wide range, ensuring clear distinctions between tokens.
- Distribution Preservation Loss ( $L_{dp}$ ): A Mean Squared Error penalty that pulls scores toward the center of the sigmoid's dynamic range (0.5), preventing them from collapsing into saturation regions (0 or 1) and maintaining gradient flow.
Threshold Control Scheme: To maintain a stable selection ratio without relying on top-k constraints, a dynamic threshold control is used:
- Buffer Proportional Control: The threshold $\tau$ is updated in real-time based on the error between the actual selection ratio and a target ratio $k_{target}$ within a training buffer.
- EMA Synchronization: To prevent misalignment between the router's optimization and the threshold, an Exponential Moving Average (EMA) of ideal thresholds (derived from a buffer of samples) is used to gently nudge the operational threshold, ensuring long-term stability.

3. Key Contributions

Dynamic Nested Depth (DND): A novel paradigm that adaptively identifies critical tokens and selectively deepens their computation via nested re-processing, moving beyond binary token pruning.
Tailored Training Strategy: A comprehensive training framework featuring a router controlling loss (for selection precision) and an adaptive threshold control scheme (for selection stability), enabling robust token-level adaptive computation.
Post-Training Integration: A plug-and-play method that can be directly integrated into both dense and Mixture-of-Experts (MoE) architectures with minimal parameter and computational overhead.

4. Experimental Results

The authors evaluated DND on diverse benchmarks across general knowledge, mathematics, reasoning, coding, and agent tasks.

Dense Models: On small-scale dense models (Qwen3-1.7B, Llama3.2-1B, Gemma3-1B), DND achieved average performance improvements of 1.88%, 2.61%, and 2.50% respectively over standard Supervised Fine-Tuning (SFT). Notable gains were observed in complex reasoning tasks (e.g., BBH, GPQA).
MoE Models: On the large-scale sparse MoE model Qwen3-30B-A3B, DND yielded an average improvement of 0.87% across 17 benchmarks. The most significant gains were in Coding and Agent tasks (e.g., +2.05 on BFCL v3).
Efficiency: The method introduces a negligible parameter increase (< 0.1M). In terms of computation, reviewing 20% of tokens adds approximately 6% extra FLOPs. Inference throughput measurements indicate the model retains 91.6%–93.1% of the vanilla model's speed under various sequence lengths.
Ablation Studies: Experiments confirmed that both the router controlling loss and the threshold control scheme are essential; removing them significantly reduces performance gains. The optimal token selection ratio was found to be around 20–30%.

5. Significance and Claims

The paper claims that DND offers a highly parameter-efficient approach to unlocking significant capabilities in LLMs without the massive costs associated with scaling or training from scratch. By dynamically allocating computational resources to tokens that exhibit high uncertainty (as evidenced by higher logit entropy in the vanilla pass), DND allows models to "review" difficult concepts internally. This approach validates the hypothesis that reasoning and complex planning rely on a sparse subset of critical tokens, and that providing these tokens with additional depth significantly enhances model performance. The method is presented as a practical, post-training solution that can be applied to existing state-of-the-art models to boost their reasoning and generation capabilities with minimal overhead.