🤖 AI

When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction

This paper introduces ESTP, a lightweight framework that improves LLM output length prediction by combining token entropy with attention-based semantic importance scores to enable more efficient, length-aware scheduling and reduce computational overhead.

Original authors: Feiyang Ren, Shengtao Wen, Lingbing Guo, Yu Tian, Yuanning Cui, Xiang Chen

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Feiyang Ren, Shengtao Wen, Lingbing Guo, Yu Tian, Yuanning Cui, Xiang Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, large language models act as powerful engines that generate text, answering questions and solving problems by predicting the next word in a sentence. To run these engines efficiently on computer hardware, engineers often group many requests together, much like a bus carrying multiple passengers. However, a persistent inefficiency arises because the hardware must treat every request in a group as if it were the longest one, padding shorter responses with empty space to make them fit. This wasted space slows down the entire system, particularly when the model is asked to perform complex reasoning or generate long, detailed answers. To fix this, researchers have long sought a way to predict exactly how long an answer will be before it is fully written, allowing the system to allocate resources precisely. For some time, the leading method for making these predictions relied on measuring the model's uncertainty, a concept known as entropy, which essentially gauges how unsure the model is about its next word.

A new study challenges the assumption that uncertainty alone is the best guide for this task. The researchers found that while uncertainty is useful, it misses a crucial layer of meaning. In the process of generating text, the model produces a vast array of signals, some of which indicate confusion or hesitation, while others highlight the most important facts, numbers, or instructions. The previous methods, which focused heavily on uncertainty, often treated the uncertain, transitional words as the most critical, while inadvertently downplaying the solid, factual tokens that actually determine the length and structure of the response. This is like trying to navigate a city by only paying attention to the foggy intersections and ignoring the clear street signs; the result is a prediction that is often wrong because it misses the core content.

To solve this, the team introduced a new framework called ESTP, which combines the measure of uncertainty with a direct look at the importance of each word. Instead of just asking how unsure the model is, the new method also asks how much attention the model is paying to a specific word. In the architecture of these models, attention acts as a spotlight, showing which words are currently driving the thinking process. The researchers discovered that words carrying the most semantic weight—such as key entities, specific numbers, and core instructions—often receive high attention even when the model is very confident about them. By merging this attention-based importance with the traditional uncertainty signal, the new system creates a balanced view of the text. It learns to value the critical information that defines the answer's length while still acknowledging the uncertain parts that signal the need for more elaboration.

The researchers tested this approach on a variety of challenging tasks, including complex mathematical reasoning and dynamic sampling scenarios, using several different large language models. They found that their combined method consistently predicted the length of the output more accurately than the previous best techniques. In many cases, the error rate dropped significantly, particularly in the complex reasoning tasks where the previous methods struggled the most. The study showed that a substantial portion of the tokens that were previously overvalued by the old system were actually low-value filler words, while many of the undervalued tokens were the very facts that dictated the final length. By correcting this misalignment, the new system provided a much clearer signal to the computer hardware.

When integrated into a full system designed to manage these AI requests, the improvement translated into tangible real-world benefits. The system was able to schedule tasks more efficiently, reducing the amount of wasted empty space that had to be filled with padding. This led to a noticeable increase in the speed at which the hardware could process requests and a decrease in the time it took to complete jobs. The method achieved these gains without requiring extra memory or slowing down the model, because it reused the internal signals the model was already generating. The results suggest that for AI systems to become truly efficient, they must look beyond simple measures of uncertainty and learn to recognize the semantic importance of the words they are processing. This shift allows the technology to handle complex, long-form reasoning with a level of precision that was previously out of reach, paving the way for faster and more reliable AI services.

Technical Summary: When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction

Problem Statement

Efficient Large Language Model (LLM) serving is frequently bottlenecked by the "barrel effect," where shorter sequences in a batch are padded to match the longest sequence, wasting computational resources and degrading throughput. While length-aware scheduling can mitigate this by predicting output lengths in advance, existing prediction methods face significant limitations.

Current state-of-the-art approaches, such as Entropy-Guided Token Pooling (EGTP), rely primarily on token-wise entropy to determine importance. However, this paper identifies a systematic blind spot: entropy measures generation uncertainty rather than semantic importance. Consequently, tokens with high semantic value (e.g., key entities, constraints, and instructions) often exhibit low entropy (high confidence) and are underweighted. Conversely, transitional or filler words with high entropy are over-amplified. This misalignment leads to the loss of critical semantic signals, reducing the reliability of length predictions, particularly in complex reasoning and dynamic Reinforcement Learning (RL) scenarios.

Methodology: ESTP Framework

The authors propose ESTP (Entropy-and-Semantic Token Pooling), a lightweight framework designed to recover lost semantics without introducing significant computational overhead. The method operates in three main stages:

Attention-Based Semantic Importance Modeling:
Instead of relying solely on entropy, ESTP computes a semantic weight ( $S_j$ ) for each token $j$ by averaging the attention values received from all other valid tokens in the final layer of the LLM. This leverages the observation that high-attention tokens often correlate with semantic importance.
$S_j = \frac{1}{N} \sum_{i=1}^{N} A_{ij}$
where $A_{ij}$ is the attention weight from token $i$ to token $j$ .
Semantics-Entropy Joint Weight Fusion:
The framework combines the semantic score ( $S_i$ ) with the token's entropy ( $H_i$ ) to produce a unified importance score ( $T_i$ ):
$T_i = \lambda_1 S_i + \lambda_2 H_i$
These scores are normalized via a softmax function to generate the final pooling weights ( $e_i$ ), where a temperature coefficient $\alpha$ is introduced to adjust the concentration degree of the distribution:
$e_i = \frac{\exp(T_i)}{\sum_{j=1}^{n}\exp(T_j)}$
This fusion allows the model to balance uncertainty signals with semantic content.
Soft-Label Distribution Regression:
To handle the heavy-tailed nature of output length distributions, ESTP employs a prediction head that treats length prediction as a distribution problem. Target lengths are discretized into bins, and a soft probability distribution is generated. The model is optimized using a combined loss function of Cross-Entropy (for classification stability) and Mean Squared Error (for precise regression), allowing it to learn both the distribution shape and the exact length expectation.

Crucially, ESTP reuses hidden states and attention weights generated during the standard LLM prefill phase, adding negligible VRAM overhead and inference latency.

Key Contributions

Empirical Verification of Misalignment: The paper provides quantitative evidence that entropy and semantic importance are often misaligned. In a diagnostic study, over 20% of tokens in the top 40% high-entropy group were found to have low semantic importance, while a similar proportion of low-entropy tokens carried high semantic value.
The ESTP Framework: A novel, lightweight pooling mechanism that fuses entropy with attention-based semantic scores. It recovers semantic signals that entropy-only methods miss, achieving this with minimal additional cost by reusing internal LLM activations.
Comprehensive Evaluation: Extensive experiments across four LLM backbones (Qwen2.5-3B/7B, Llama3.2-1B/3B) and three challenging scenarios (Long-Sequence, Complex Reasoning, Dynamic RL Sampling) on the ForeLen benchmark.

Experimental Results

Prediction Accuracy: On the ForeLen benchmark, ESTP consistently outperforms baselines, including EGTP, TRAIL, and external auxiliary models (SSJF-Reg/MC). It achieves the lowest Mean Absolute Error (MAE) across all models, reducing the average MAE by over 9 points compared to EGTP, with improvements exceeding 20 points in RL scenarios.
Robustness: ESTP demonstrates lower Root Mean Square Error (RMSE), indicating better handling of large prediction deviations.
System Efficiency: When integrated with a length-aware scheduler in end-to-end system tests (using vLLM), ESTP significantly improves throughput and reduces the padding ratio. For instance, in Reasoning scenarios, the padding ratio dropped from 1.18 (EGTP) to 0.37 (ESTP), and average Job Completion Time (JCT) was reduced.
Ablation Studies: Experiments confirm that both entropy and semantic features are essential; using either alone yields inferior results compared to the joint approach.

Significance and Limitations

The paper positions ESTP as a practical and effective building block for length-aware LLM serving systems. By addressing the semantic blind spot of entropy-only methods, it enables more accurate resource allocation and scheduling, directly improving hardware utilization and throughput.

The authors acknowledge specific limitations:

The attention-based proxy for semantic importance is correlational rather than causal.
The method relies on internal activations, limiting its immediate applicability to closed-source models where these states are inaccessible.
Future work is suggested to explore causal attribution, black-box adaptation, and multi-feature fusion for broader scenario robustness.