Inference-Time Decision Calibration for Temporal Classification

Based on the provided paper, here is a detailed technical summary of "Inference-Time Decision Calibration for Temporal Classification."

Problem Statement

Temporal classification is traditionally treated as a representation learning problem: when a model fails, the standard response is to build a stronger temporal encoder (e.g., deeper recurrent networks, Transformers, or multi-scale architectures). However, this paper argues that errors may not solely stem from missing temporal evidence (representation failure) but also from how available evidence is converted into final decisions (decision failure).

Standard approaches make it difficult to distinguish between these two failure modes because replacing a backbone or jointly training a larger model alters both representation learning and decision conversion simultaneously. The authors posit that a trained classifier makes decisions based on logits, thresholds, and calibration choices, not directly from representations. When the target metric (e.g., macro-F1) differs from the training loss, the final score-to-decision step can significantly impact performance without changing the underlying temporal representation.

Methodology: Representation–Calibration Decomposition

The paper proposes a framework that freezes a trained "native" temporal classifier and introduces two inference-time interventions to diagnose and address errors without retraining the backbone. This creates a diagnostic decomposition of temporal classification errors.

1. Native–Residual Decomposition

• Native Branch: The original backbone ($f_\psi$) remains frozen, producing native logits $\ell_{nat}(x)$.
• Residual Multi-Scale Branch: A conservative auxiliary branch is added to extract evidence from multiple temporal resolutions.
  • Input sequences are projected to different scales $s \in S$ (e.g., via downsampling or pooling).
  • Each scale is encoded by a scale-specific encoder $E_s$.
  • Scale representations are aggregated using trust weights $\alpha_s(x)$ to form an auxiliary representation $z_{aux}(x)$.
  • This is mapped to auxiliary logits $\ell_{aux}(x)$.
  • Fusion: The raw combined logits are computed additively: $\ell_{raw}(x) = \ell_{nat}(x) + \gamma_{res}\ell_{aux}(x)$. This additive structure ensures the auxiliary branch acts as a correction rather than overwriting the native model.
• Trust Allocation Variants: The paper evaluates three methods for determining scale trust weights $\alpha_s(x)$:
  • Residual-MS: Uniform fusion ($\alpha_s = 1/M$).
  • Residual-ZOnly: Learned trust based solely on scale representations via a gating network.
  • Residual-RCG: Learned trust based on scale representations augmented with explicit reliability descriptors (measuring local instability, cross-scale consistency, and representation energy).

2. Post-Hoc Decision Calibration
After the raw predictor (native + residual) is trained, all representation parameters are frozen. A lightweight calibrator $C_\phi$ is fitted on a validation set to recombine the evidence.

• Branch-Aware Calibration: Unlike standard methods that operate on raw logits, this approach takes a branch-wise evidence vector $v(x) = [\ell_{nat}(x); \ell_{aux}(x)]$ as input.
• Mechanism: The calibrator learns to reweight and transform the native and auxiliary logits (e.g., via temperature scaling, class-wise gains, and bias) to produce calibrated logits $\tilde{\ell}(x)$.
• Diagnostic Logic:
  • If the raw residual branch improves performance, the bottleneck is representational (missing evidence).
  • If raw gains are small but calibration improves performance, the bottleneck is decisional (underused evidence).
  • If neither helps, the system is in a near-saturated regime.

Key Contributions

1. Decomposition Framework: A novel separation of temporal classification errors into missing temporal evidence, underused decision-level evidence, and near-saturation regimes, achieved by freezing the backbone.
2. Conservative Residual Adapter: A multi-scale adapter that adds auxiliary logits as an additive correction, enabling the analysis of complementary evidence without backbone retraining.
3. Branch-Aware Calibration: A post-hoc decision layer that operates on separated native and auxiliary logits to test if residual evidence is useful even when raw residual gains are minimal.
4. Empirical Regime Analysis: A comprehensive evaluation across five benchmarks (FI-2010, PTB-XL, UCI-HAR, MHEALTH, HARTH) and multiple backbone families (CNN, RNN, ResNet, Transformer) to identify which bottleneck dominates in different settings.

Experimental Results

The authors evaluated the framework across five datasets with distinct temporal structures:

• FI-2010 (Financial Limit Order Book): Shows the strongest "residual-sensitive" regime. Residual multi-scale evidence consistently improved performance, particularly in short-horizon, high-noise settings where native decision boundaries were insufficient. Calibration provided smaller, horizon-dependent gains.
• PTB-XL (ECG Classification): Exhibited a "calibration-sensitive" regime, particularly for recurrent backbones. While raw residual gains were sometimes modest, branch-aware calibration significantly improved performance, suggesting that native and auxiliary logits contained complementary evidence not fully exploited by raw decision rules.
• HARTH & MHEALTH (Wearable Activity Recognition): Showed mixed results. Gains were often concentrated in weaker recurrent backbones (GRU/LSTM), while stronger convolutional or Transformer-based models showed limited headroom.
• UCI-HAR: Represented a "near-saturated" regime. Strong native performance left little room for either residual correction or post-hoc calibration to improve results.

Key Findings:

• Regime Dependence: Gains are not universal. Residual evidence is most valuable in noisy or representation-limited settings. Calibration is most useful when complementary evidence exists but is poorly combined.
• Trust Allocation: Learned trust allocation (ZOnly, RCG) was not uniformly superior to uniform fusion (MS). Uniform fusion often performed best in high-noise financial data, while learned trust helped in specific recurrent settings.
• Diagnostic Utility: The decomposition successfully identified whether a specific dataset-backbone pair was limited by representation, decision conversion, or saturation.

Significance and Claims

The paper claims that temporal classification should not be viewed solely as a representation learning problem. Instead, it argues that errors often arise from how available evidence is trusted, combined, and calibrated.

The significance of this work lies in providing a diagnostic framework rather than a universal "winning" module. By separating representation learning from decision conversion, the authors demonstrate that:

1. Many temporal classification errors are not due to weak encoders but to suboptimal decision rules.
2. A "one-size-fits-all" approach to improving temporal classifiers is ineffective; the appropriate intervention (adding evidence vs. calibrating decisions) depends heavily on the specific dataset regime and backbone architecture.
3. Inference-time interventions can provide a clearer understanding of model limitations without the computational cost and confounding factors of retraining large backbones.

The authors conclude that their decomposition offers a practical tool for identifying whether a model is representation-limited, decision-limited, or near-saturated, guiding future efforts toward the correct type of improvement.