When Offline Selectors Cannot Beat the Best Single Model: A Diagnostic Study on edX Dropout Prediction

Technical Summary: When Offline Selectors Cannot Beat the Best Single Model

Problem Formulation
The paper addresses the problem of offline meta-learning for dynamic model selection, specifically within the context of educational data. The task is framed as a contextual bandit where an agent must select the best-performing base model from a pool of pre-trained classifiers ($M = \{m_1, \dots, m_K\}$) for each individual instance (student-course pair) based on a state vector $s$. The goal is to outperform the strongest single static model by leveraging the fact that different models excel in different regions of the feature space.

The authors investigate why offline selectors, trained on logged data, routinely fail to recover the "oracle headroom"—the theoretical accuracy gain available if one could perfectly select the best model for every instance. They hypothesize three potential causes for this failure, which are often conflated in practice:

1. Algorithmic Failure: The offline reinforcement learning (RL) or behavioral cloning (BC) algorithm is insufficient (e.g., due to conservatism or reward misspecification).
2. Representational Failure: The state representation lacks the necessary information to predict which model will win.
3. Distributional Failure: A shift exists between the oracle-label distribution in the offline buffer and the distribution at evaluation time.

Methodology: A Three-Stage Diagnostic Protocol
To disentangle these causes without further tuning, the authors propose a three-stage diagnostic protocol applied to a shared offline buffer:

• Stage 1: Local Label Consistency (k-NN).
  The authors measure the local consistency of oracle actions using a $k$-nearest-neighbor (k-NN) approach. For each state in the buffer, they check if its neighbors share the same oracle action (the base model that correctly predicts the true label). A low consistency score indicates local representational ambiguity, suggesting that no local selector can reliably distinguish the best model based on the current state features. A held-out 10-NN selector is also trained to calibrate how this local ambiguity translates to test-time performance.

• Stage 2: Algorithm vs. Representation (Paired Ablation).
  To distinguish between algorithmic and representational failures, the authors train two distinct policies on the same buffer:

  1. Behavioral Cloning (BC): A supervised multi-class classifier trained with hard-label cross-entropy.
  2. Offline RL (DQN and CQL): A Deep Q-Network and a Conservative Q-Learning variant trained with a one-step bandit objective and oracle-match rewards.
     If both supervised and RL approaches fail by similar margins, the bottleneck is likely shared (representation or distribution) rather than algorithmic. If they diverge, the algorithm choice is the culprit. CQL is used specifically to test if conservatism is the limiting factor.

• Stage 3: Feature Marginal Value (State Ablation).
  The authors test whether the engineered behavioral features provide signal beyond what is already encoded in the base models' probability vectors. They compare a "full-state" selector (38 dimensions: 28 behavioral features + 5 probabilities + metadata) against a "probabilities-only" selector (5 dimensions). They also test "disagreement-derived transforms" (e.g., standard deviation, pairwise differences) to see if richer representations of the probability vector improve performance.

Experimental Setup
The protocol is applied to edX clickstream data involving 223,505 student-course pairs and 84.5 million events.

• Base Models: Five pre-trained classifiers (Logistic Regression, Random Forest, Gradient Boosting, Calibrated RF, and a Stacking Ensemble) trained to predict course dropout.
• Task: Select the best model for a 14-day prediction window based on a 14-day observation window.
• Oracle: Defined as the base model assigning the highest probability to the true class.

Key Results

1. Oracle Headroom Exists: Across 16 observation/prediction window configurations, the per-instance oracle beats the strongest single base model by an average of 9.7 accuracy points (range 4.5–15.5). On the main configuration, the gap is 6.3 points (Oracle: 0.825 vs. Best Static: 0.762).
2. Selectors Fail to Recover Headroom: Despite the significant oracle gap, all learned selectors (BC, DQN, CQL) cluster tightly around 0.748 test accuracy, failing to exceed the static reference (0.762). This result is robust across buffer sizes (up to 8,000 training examples) and held-out test sets (up to 2,000 examples).
3. Diagnosis of Failure:
   • Not Algorithmic: CQL (conservative) performs similarly to BC (supervised), and DQN does not significantly outperform them. The failure is not due to conservatism or reward engineering.
   • Not Distributional Shift: While buffer-to-test marginal shift ($d_{TV}$) exists (approx. 0.07), it is not the primary driver of the failure.
   • Representational Ambiguity: The local k-NN consistency is low (0.388), indicating that nearby states often map to different oracle actions. This "local representational ambiguity" is the primary bottleneck.
   • Feature Redundancy: Adding engineered behavioral features or disagreement-derived transforms to the state does not improve test accuracy over using the base-model probability vector alone. The 28 behavioral features, while useful for the base models, do not provide marginal signal for the selection task.

Contributions and Significance
The paper makes two primary contributions:

1. C1. A Combined Diagnostic Procedure: The authors assemble established checks (k-NN consistency, paired BC/RL ablation, state ablation, and distribution shift estimation) into a unified protocol. This allows practitioners to diagnose why offline meta-learning fails before investing in further tuning or data collection. It shifts the focus from "tuning the learner" to "diagnosing the data/representation."
2. C2. An Empirical Case Study: The study demonstrates that in a high-stakes educational setting with abundant data, offline selectors cannot beat the best single model due to local label ambiguity, not algorithmic conservatism or distribution shift. The results suggest that simply collecting more data or tuning offline RL hyperparameters will not solve the problem; instead, the state representation must be redesigned or new data sources collected to resolve the ambiguity.

Conclusion
The paper concludes that the failure of offline selectors in this domain is not a limitation of current offline RL algorithms (like CQL) but a fundamental limitation of the available state representation. The "oracle headroom" exists theoretically, but the local ambiguity in the data prevents any selector from reliably accessing it. The authors argue that future iterations should focus on changing the state representation or collecting new data rather than further tuning the offline learner.