💬 NLP

HE-SNR: Uncovering Latent Logic via Entropy for Guiding Mid-Training on SWE-bench

This paper addresses the lack of effective mid-training metrics for SWE-bench by introducing a data filtering strategy and the HE-SNR metric, which is grounded in the Entropy Compression Hypothesis to better guide Large Language Model optimization for complex software engineering tasks.

Original authors: Yueyang Wang, Jiawei Fu, Baolong Bi, Xili Wang, Xiaoqing Liu

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yueyang Wang, Jiawei Fu, Baolong Bi, Xili Wang, Xiaoqing Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a brilliant but chaotic apprentice to become a master software engineer. You have a massive library of code (the training data), and you want to know if the apprentice is actually learning to solve complex problems or just memorizing the look of the answers.

This paper introduces a new way to check on the apprentice's progress while they are still learning, before you put them through the final, expensive "final exam" (which involves a lot of human-like instruction tuning).

Here is the breakdown of their discovery, using simple analogies:

1. The Problem: The "Fake Confidence" Trap

Usually, when we check if a computer is learning, we use a metric called Perplexity (PPL). Think of PPL as a "surprise meter." If the computer is less surprised by the next word in a sentence, it thinks the computer is doing well.

However, the authors found two big problems with this meter:

The "Long-Context Tax": When you ask the computer to read a very long document (like a whole book instead of a page), the "surprise meter" goes haywire. It spikes up, making it look like the computer is getting worse, even if it's actually getting smarter. It's like a runner tripping over their shoelaces at the start of a marathon; the stumble doesn't mean they can't run the race, but the stopwatch looks bad.
The "Rote Memorizer" vs. "Thinker": The old meter mostly rewards computers that are good at guessing the exact next word (like a parrot repeating a phrase). But solving real software bugs isn't about guessing the exact next word; it's about having a few good options in mind and picking the right one. The old meter ignores this "thinking process."

2. The New Idea: Measuring "Reasonable Hesitation"

The authors propose a new theory: True intelligence isn't about having zero uncertainty; it's about having organized uncertainty.

Imagine a detective solving a crime:

A bad detective is either 100% sure they know the killer (low uncertainty) or completely lost and guessing randomly among 1,000 suspects (high, chaotic uncertainty).
A smart detective narrows it down to just 3 likely suspects. They are "hesitating" between these three, but they know it's one of them. This is called a "Reasonable Hesitation."

The paper calls this an "Entropy-Compressed State." Instead of the computer being confused about 100 things, a smart computer is confidently confused about only 2 or 3 things.

3. The Discovery: The "Shift to ln 3"

The authors looked at the computer's brain during training and found a magic number: ln 3 (which is roughly 1.1).

Early Training: The computer is confused about many options (entropy is high and messy).
Smart Training: As the computer gets better at logic, its confusion "compresses." Even when it doesn't know the exact right answer immediately, it narrows its choices down to a tight group of about 3 options.
The Shift: The paper noticed that the best models consistently show a "peak" in their confusion levels right around this number (ln 3). It's like the computer saying, "I'm not 100% sure, but I'm 99% sure it's one of these three."

4. The New Tool: HE-SNR (The "Signal-to-Noise" Compass)

To fix the broken "surprise meter," the authors built a new tool called HE-SNR.

How it works: Instead of asking, "How surprised is the computer by the exact right answer?" (which is the old way), HE-SNR asks: "When the computer is genuinely stuck and thinking hard, how well does it narrow down its choices?"
Filtering the Noise: They realized that computers often get distracted by "style" (like using fancy words or formatting) rather than "logic" (the actual code). So, they built a filter to ignore the style and only look at the hard, logical parts of the code.
The Result: This new compass ignores the "shoelace trips" (the Long-Context Tax) and the "parrot tricks" (memorization). It only measures the "Reasonable Hesitation."

5. The Big Surprise: The "Alignment Tax"

The paper also found something surprising about the final "instruction tuning" phase (where humans teach the computer to follow orders).

The Trade-off: When they taught the computer to follow instructions perfectly, the computer became better at guessing the exact right answer (Top-1 accuracy went up).
The Cost: However, this training actually worsened the "Reasonable Hesitation." The computer stopped considering the other 2 or 3 good options and just blindly picked one.
The Conclusion: The authors suggest that by forcing the computer to be too confident too early, we might be accidentally crushing its ability to think deeply about complex problems. The computer becomes a better "yes-man" but a worse "detective."

Summary

The paper argues that to build truly smart software engineers, we shouldn't just measure how well a computer guesses the next word. Instead, we should measure how well it narrows down its confusion to a small, manageable group of options.

Their new tool, HE-SNR, acts like a spotlight that ignores the computer's "style" and "shoelace trips," focusing only on its ability to think logically and handle uncertainty. This allows developers to train better models faster and avoid the trap of making models that are confident but not actually smart.

Technical Summary: HE-SNR: Uncovering Latent Logic via Entropy for Guiding Mid-Training on SWE-BENCH

Problem Statement

The evaluation of Large Language Models (LLMs) on complex software engineering tasks, particularly via the SWE-BENCH benchmark, currently relies heavily on Supervised Fine-Tuning (SFT) to elicit agentic capabilities. However, the fundamental reasoning abilities required for these tasks are acquired during the mid-training phase. A critical deficit exists in metrics capable of guiding this mid-training process effectively.

Standard metrics like Perplexity (PPL) are insufficient for two primary reasons:

Weak Correlation with Agentic Potential: PPL correlates strongly with Top-1 accuracy, which often exceeds 90% on curated datasets, measuring replication precision rather than the latent reasoning potential required for complex, multi-turn agent tasks.
The "Long-Context Tax": When extending context windows (e.g., from 32K to 128K) via frequency scaling (e.g., RoPE, YaRN), models undergo a transient distributional shift in positional embeddings. This causes a temporary spike in predictive entropy and a degradation in PPL, obscuring genuine capability growth and creating a misleading signal for training optimization.

Consequently, there is an urgent need for a robust metric that isolates true reasoning signals, is immune to the Long-Context Tax, and can predict downstream SWE performance without the computational cost of post-SFT evaluation.

Methodology

1. Data Curation and Filtering Strategy

To isolate signals predictive of latent capability, the authors introduce a rigorous token-granular filtering strategy.

Trajectory Selection: 500 successful trajectories from SWE-BENCH-VERIFIED are curated, ensuring strict distributional alignment with SFT data.
Component Filtering: Metrics are computed exclusively on Action components (executable code, file paths, commands), masking Observations (environment feedback) and excluding Thoughts (reasoning traces often dominated by stylistic artifacts).
Noise Reduction: Structural markers (XML tags), markdown, comments (via AST parsing), and redundant whitespace are stripped to focus purely on functional correctness.

2. Theoretical Insight: Entropy Compression Hypothesis

The paper proposes the Entropy Compression Hypothesis, redefining intelligence not as scalar information minimization (low loss), but as the capacity to structure uncertainty into Entropy-Compressed States.

Entropy-Compressed States: These are states where probability mass is uniformly distributed over a small set of top- $k$ candidates, manifesting as distinct entropy peaks at $\ln k$ .
The "Shift to $\ln 3$ ": Empirical analysis reveals that as models mature, the probability mass of high-entropy tokens (specifically Non-Top-2 tokens) shifts from a dispersed state (centered around $\ln 4$ ) to a compressed state at $\ln 3$ . This indicates the model narrows its uncertainty to a refined set of 2–3 plausible candidates, termed "reasonable hesitation."
Differentiation: This shift is distinct from stochastic uncertainty (e.g., in Observation tokens involving random IDs), which exhibits peaks at $\ln 10$ and lacks the structured logic of the $\ln 3$ peak.

3. Metric Formulation: HE-SNR

Based on the hypothesis, the authors formulate HE-SNR (High-Entropy Signal-to-Noise Ratio).

High-Entropy Decision Set ( $H$ ): Tokens where the Top-10 entropy exceeds a threshold $\epsilon = (\ln 3 + \ln 4)/2$ and the ground truth is within the Top-10 candidates. This filters out tokens with severe distributional divergence while focusing on critical reasoning points.
Calculation: HE-SNR measures the relative strength of the target signal against the background noise:
$\text{HE-SNR} = \frac{1}{|H|} \sum_{t \in H} \frac{p(x_t)}{H_{\text{top10}}(x_t)}$
where $p(x_t)$ is the signal probability and $H_{\text{top10}}(x_t)$ is the Top-10 entropy (noise).
Robustness: By normalizing the signal against the entropy floor, HE-SNR mitigates the Long-Context Tax, which inflates absolute entropy but does not necessarily degrade the relative signal-to-noise ratio of valid reasoning paths.

Key Results

1. Correlation with Downstream Performance

Superiority over PPL: Standard PPL shows poor correlation with SWE-BENCH Pass@1 scores, particularly failing to distinguish capability growth during the Long-Context Tax (e.g., MoE-A26B at Step 200 of 128K training shows degraded PPL but improved SWE performance).
HE-SNR Linearity: HE-SNR demonstrates a strict linear relationship with downstream SWE-BENCH performance across models ranging from 68B to 560B parameters and context windows of 32K/128K.
Data Efficiency: The proposed metric achieves high correlation (Pearson $r \approx 0.96$ , Kendall $\tau \approx 0.98$ ) using only 500 trajectories (~12.5M tokens), significantly reducing evaluation costs.

2. Ablation Studies

Filtering Impact: Evaluating "Action" tokens with full filtering (removing XML, comments, whitespace) significantly outperforms raw token evaluation, confirming that stylistic noise obscures latent reasoning signals.
Threshold Robustness: The chosen entropy threshold $\epsilon = (\ln 3 + \ln 4)/2$ is near-optimal, with the metric maintaining high correlation across a range of thresholds, indicating structural soundness rather than overfitting to a specific value.

3. Insights into SFT and Alignment Tax

SFT Effects: While SFT improves global PPL and Top-1 accuracy, it degrades HE-SNR and HE-PPL on the High-Entropy Decision Set.
Alignment Tax Mechanism: This suggests that SFT may prioritize superficial pattern matching and stylistic alignment at the expense of the model's ability to navigate complex, high-entropy reasoning paths, offering a novel explanation for the "Alignment Tax" in agentic tasks.

Significance and Claims

The paper claims to provide both a theoretical foundation and practical tools for optimizing the latent potential of LLMs in complex engineering domains.

Theoretical Contribution: It challenges the scalar compression view of intelligence, proposing that true reasoning is characterized by the structured compression of uncertainty into low-order Entropy-Compressed States (specifically the "Shift to $\ln 3$ ").
Practical Utility: HE-SNR serves as a robust compass for mid-training, enabling efficient iteration by decoupling genuine capability growth from transient artifacts like the Long-Context Tax.
Generalizability: The "Shift to $\ln 3$ " phenomenon is observed across different architectures (Dense and MoE) and domains (Software Engineering and Mathematical Reasoning), suggesting it is a fundamental signature of structured logical reasoning.

The authors conclude that by focusing on high-entropy decision points and utilizing HE-SNR, the community can better identify sub-optimal candidates early, potentially reducing the computational costs and carbon footprint associated with training large-scale models, while fostering the development of more robust reasoning capabilities over rote memorization.