Technical Summary: CRAFTER – Corrective Residual Agent with Feature-based Temporal Exploration and Reasoning

Problem Statement

Large-scale forecasting is increasingly served by pretrained foundation models deployed in a "frozen" state (zero-shot) to handle thousands of time series without per-series fine-tuning. While efficient, these frozen backbones fail in structured, recurring ways—such as missing promotions, lagging regime shifts, or underestimating event spikes. These errors are business-critical and require interpretability to act upon.

The core challenge is corrective feature discovery: identifying interpretable features of the residual (the error signal) of a specific frozen forecaster to drive a lightweight post-hoc corrector. Unlike traditional automated feature engineering, which models the data-generating process to train a new predictor, this task models the model-failure process. The goal is not to find general regularities but to discover features that explain the specific blind spots of a deployed model. This discovery is difficult because:

1. Semantic Necessity: The most valuable corrective features often require domain-specific naming (e.g., "renewable penetration") rather than simple statistical enumeration of raw channels.
2. Regime Dependence: A feature's value is highly dependent on the specific backbone-dataset pairing; a feature that helps one model may be inert or harmful to another.

Methodology: CRAFTER

The authors propose CRAFTER (Corrective Residual Agent with Feature-based Temporal Exploration and Reasoning), a framework that keeps the backbone frozen and mines its residual using two distinct generators, unified by a source-blind acceptance gate.

1. Architecture Overview

CRAFTER operates in $R$ rounds. In each round, it fits a lightweight corrector, diagnoses remaining errors on a held-out validation block, mines new candidate features, and filters them through a gate. The final output is a validation-selected corrector (identity, additive, or multiplicative) or the original forecast if no improvement is found.

2. Dual Generators

CRAFTER employs two heterogeneous generators to propose feature specifications:

• Exploration (Compositional Search): A Monte-Carlo Tree Search (MCTS) that composes raw input channels using a fixed vocabulary of transforms (e.g., log, safe_div) and operators (e.g., +, *). It treats channels as "mechanism families" (e.g., volatility, trend) to guide the search efficiently.
• Reasoning (LLM Proposer): A Large Language Model that proposes features based on the run's own evidence (accepted/rejected features, uncovered mechanism families). It generates three types of candidates:
  • LLM-combo: Named combinations of existing channels.
  • LLM-flag: Binary flags based on thresholds.
  • LLM-code: Short executable code for complex logic.
  • Base Proposer: Can invent new named base columns that join the channel pool for subsequent search rounds.

3. Source-Blind Acceptance Gate

A single validation-grounded gate evaluates all candidates regardless of their origin (search vs. LLM). A candidate is accepted only if:

• It explains a portion of the validation residual that the current corrector cannot account for (Spearman correlation $\rho$ exceeds a threshold $\tau$).
• It is not collinear with already accepted features.
• It adheres to per-family and per-round budget constraints.

This design ensures that the pipeline is agnostic to the feature source, allowing for fair comparison between generators and external systems.

4. Validation-Selected Corrector

The surviving features feed a small set of correctors: an identity map (NONE), an additive Gradient Boosted Decision Tree (GBDT), and a multiplicative GBDT. The system selects the configuration with the best validation wMAPE. The "NONE" option acts as a safeguard, ensuring the correction never degrades performance on the validation set.

5. Coupling (CRAFTER†)

The framework supports a coupled mode (CRAFTER†) where the LLM and search feed back into each other. Accepted LLM-generated base columns become searchable in the next round, and the LLM's next proposals are conditioned on the search's accepted/rejected history.

Key Contributions

1. Source-Blind Framework (C1): The paper formulates feature discovery for frozen forecasters as residual correction. By using a single validation gate for heterogeneous sources, CRAFTER serves as both a deployable correction layer and an evaluation instrument. It allows any external feature-engineering system to run through the identical pipeline, attributing forecast changes solely to the feature source.
2. Characterization of Utility (C2): The authors map the conditions under which corrective features help. They find that gains are proportional to the "residual headroom" (exploitable structure in the error). Corrections are most effective on weak backbones with informative covariates (where LLM-named features are critical) and become neutral when a strong backbone has already saturated the series.

Experimental Results

The authors evaluated CRAFTER on six public datasets (EPF, ROSSMANN, ROHLIK, BIZITOBS, FAVORITA, M5) and six frozen backbone families (Timer, Chronos, Moirai, Toto, Chronos-2, Moirai-2.0).

• Performance: CRAFTER consistently outperformed three dedicated feature-engineering baselines (TSFRESH, CAAFE, LLM-FE) across all feature budgets. It roughly doubled the lift provided by corrector-only methods and cut the error of the weakest backbones by up to 27%.
• Robustness:
  • Feature Budget: CRAFTER outperformed baselines even at low feature budgets ($K=5$), proving the gains come from feature quality, not quantity.
  • Fine-Tuning: CRAFTER provided additional lift even when applied on top of fine-tuned backbones, demonstrating it is not a replacement for adaptation but a complementary layer.
  • LLM Backend: Results held across different LLM backends (GPT-5.2 and DeepSeek-3.4-pro), though specific feature sets varied.
• Feature Provenance: Analysis showed that LLM-generated semantic features (combinations and flags) comprised the majority of selected features, while atomic search served as a control for the enumerable space.
• Temporal Transfer: Features discovered on one time-split transferred effectively to subsequent splits, indicating the discovery of persistent residual structure rather than window-specific noise.

Significance and Claims

The paper claims that CRAFTER establishes a new paradigm for deploying frozen forecasters. Instead of retraining or fine-tuning expensive foundation models, practitioners can deploy a lightweight, interpretable correction layer that mines the specific failure modes of the frozen model.

The authors emphasize that the value of corrective features is not universal but context-dependent. The method does not claim to fix all forecasting errors; rather, it provides a systematic way to identify when a frozen model has "headroom" for improvement. The significance lies in:

1. Cost-Efficiency: Correcting a deployed model is cheaper than retraining it.
2. Interpretability: The discovered features (e.g., "renewable_penetration") explicitly name the mechanisms the backbone missed, aiding business decision-making.
3. Fair Evaluation: The source-blind pipeline allows for rigorous attribution of performance gains to specific feature engineering strategies, separating the value of the generator from the value of the corrector.

The paper concludes that while feature budgets amplify gains in unsaturated regimes, they cannot manufacture structure where the residual is already noise. The optimal strategy is to deploy corrective feature discovery when validation residuals remain structured, particularly in weak-backbone or semantic-covariate regimes.