Entropy Sentinel: Continuous LLM Accuracy Monitoring from Decoding Entropy Traces in STEM

1. Problem Statement

Deploying Large Language Models (LLMs) in production faces two coupled challenges:

1. Monitoring: Estimating where a model underperforms as traffic patterns and domains shift over time.
2. Improvement: Prioritizing data acquisition to close the largest performance gaps.

Current practices rely on manually curated benchmarks and periodic human-labeled evaluations. These methods are expensive, slow, and fail to provide continuous, granular insights into specific traffic slices (e.g., specific topics or customer segments). The paper proposes using inference-time signals already generated by the model to estimate accuracy without requiring repeated labeling or access to hidden model states.

2. Methodology

The authors propose a two-stage, lightweight methodology to estimate domain-level accuracy using only top-$k$ decoding log-probabilities (available via standard model-serving APIs).

A. Signal Extraction: Entropy Profiles

Instead of relying on a single uncertainty metric, the method constructs a compact entropy profile for each generated response:

• Approximation: Since APIs often only expose top-$k$ token probabilities (e.g., $k=20$), the authors approximate the Shannon entropy at each decoding step $t$ by summing over the top-$k$ tokens: $\tilde{H}(t) = -\sum_{i \in \text{Top-}k} p_i^{(t)} \log p_i^{(t)}$.
• Trajectory: This creates an entropy trajectory $\{\tilde{H}(t)\}_{t=1}^T$ for the entire generation.
• Feature Vector: The trajectory is summarized into a 17-dimensional feature vector containing:
  • Central tendency and dispersion (Mean, Std Dev, Max).
  • Distributional tails (Quantiles Q10–Q90).
  • Shape metrics (Skewness, Kurtosis).
  • Accumulation metrics (Sum of Entropy, Sum of NLL).
  • Traditional white-box uncertainty metrics (e.g., Perplexity, NLL).

B. Prediction and Aggregation

1. Instance-Level Prediction: A lightweight probabilistic classifier (e.g., Random Forest, Logistic Regression, MLP) is trained on the 17D feature vectors to predict the probability of correctness ($\hat{P}(x) \in [0,1]$) for a single instance.
2. Domain-Level Estimation: To estimate the accuracy of a specific domain or traffic slice $D$, the predicted correctness probabilities are averaged:
   $$\hat{A}(D) = \frac{1}{|X_D|} \sum_{x \in X_D} \hat{P}(x)$$
   This yields an accuracy estimate in interpretable units (0–100%), suitable for monitoring and prioritization.

3. Experimental Setup

• Benchmarks: 10 STEM reasoning benchmarks covering elementary math, advanced math, and science (e.g., GSM8K, MATH, GPQA, OlympiadBench).
• Models: 9 LLMs from 6 families, ranging from 3B to 20B parameters (including Phi-3.5, Ministral, Qwen, Gemma, Llama, and GPT-OSS).
• Robustness Protocol (Domain Shift): To stress-test generalization, the authors performed an exhaustive sweep:
  • For $k \in \{1, 2, 3, 4\}$, they trained the estimator on all possible combinations of $k$ benchmarks.
  • They evaluated accuracy estimation on the remaining $10-k$ unseen benchmarks.
  • This resulted in 385 training groups and over 160,000 total configurations (varying classifier types, calibration, balancing, and feature subsets).
• Ground Truth: Correctness labels were generated using an external validator LLM (Grok-4.1) to compare model outputs against reference answers.

4. Key Results

A. Accuracy Estimation Performance

• Strong Correlation: The entropy-based estimators often track held-out benchmark accuracy closely. For the best-performing model (Phi-3.5-Mini), the Spearman correlation ($\rho$) reached 1.00, indicating near-perfect ranking of domains by difficulty.
• Error Rates: With supervision from just two benchmarks (an "Extremes" set: one easy, one hard), the median Absolute Error (AEE) across models was low (0.03–0.12).
• Model Dependence: Reliability varies significantly by model. While Phi-3.5 and Ministral showed excellent calibration, models like Qwen-8B and GPT-OSS showed weaker agreement, highlighting the need for per-model validation before deployment.

B. Impact of Training Composition (Crucial Finding)

The most significant factor influencing performance was not the classifier architecture or feature dimensionality, but the diversity of the supervision set:

• Difficulty Balance: Training sets that span a wide difficulty range (mixing easy and hard tasks) consistently outperformed homogeneous sets.
• U-Shaped Relationship: Supervision sets with intermediate weighted accuracy (roughly 0.4–0.7) yielded the lowest error. Sets that were too easy or too hard failed to generalize because they underrepresented either high-entropy failure patterns or low-entropy success patterns.
• Data Quantity: Increasing the number of training benchmarks ($k$) from 1 to 4 significantly reduced error and variance, but a well-chosen pair of benchmarks (e.g., GSM8K + OlympiadBench) often matched the performance of larger sets.

C. Feature and Model Sensitivity

• Feature Robustness: While the full 17D profile was used, simpler subsets (e.g., just the 10 entropy distribution stats, or even a single metric like Sum of Entropy) performed nearly as well. This suggests that accumulation and extrema metrics capture the bulk of the signal.
• Classifier Choice: Simple classifiers (Random Forest, Logistic Regression) performed as well as or better than complex MLPs, suggesting the signal is linearly separable enough that deep learning is unnecessary.

5. Key Contributions

1. Practical Monitoring Primitive: Demonstrated that standard inference logs (top-$k$ logprobs) contain sufficient signal to estimate domain-level accuracy without access to hidden states or full vocabulary distributions.
2. Exhaustive Robustness Study: Provided the first large-scale evaluation of accuracy estimation under domain shift across 9 models and 10 benchmarks with exhaustive train/test compositions.
3. Insight on Supervision: Identified that difficulty diversity in the training data is the dominant factor for successful cross-domain generalization, outweighing classifier complexity or feature engineering.
4. Actionable Metric: Proposed a method that outputs accuracy in absolute units (0–1), enabling direct prioritization of data acquisition for low-performing slices.

6. Significance and Limitations

Significance:
This work offers a scalable, cost-effective solution for continuous LLM monitoring. By leveraging existing inference traces, teams can identify performance degradation in real-time and target data collection efforts precisely, moving away from expensive, static benchmarking cycles.

Limitations:

• Domain Scope: Evaluated strictly on STEM reasoning with verifiable answers. Performance in open-ended domains (creative writing, dialogue) where "correctness" is subjective remains unproven.
• Decoding Sensitivity: The entropy signal depends on decoding parameters (temperature, max length). Changes in these settings could shift the distribution without reflecting actual capability changes.
• Absolute Calibration: While the method excels at ranking domains (identifying which is harder), absolute accuracy estimates (AEE) can still have non-trivial errors for some models, suggesting it should be used for prioritization rather than as a sole source of truth for high-stakes decisions without further validation.