Deterministic Decisions for High-Stakes AI. A Zero-Egress Pipeline with the Deployability of RAG and the Accuracy of Machine Learning

Technical Summary: Deterministic Decisions for High-Stakes AI

Problem Statement: Intervention Bias and the Evaluation Gap
The paper identifies a critical failure mode in zero-shot Large Language Model (LLM) agents deployed for high-stakes educational advising: intervention bias. Without task-specific training, these agents systematically recommend action (e.g., tutoring, reminders) when a researcher-defined, hindsight-optimal "oracle" policy mandates inaction. In a study using the Open University Learning Analytics Dataset (OULAD), zero-shot GPT-4o recommended intervention for 73% of students at day 56, whereas the oracle policy designated 70.1% as requiring no action. This results in a 43 percentage-point false-positive rate, generating thousands of unnecessary advisor contacts in large institutions, eroding system credibility, and causing alert fatigue.

Furthermore, the paper identifies an Evaluation Gap: standard LLM-as-judge frameworks (e.g., DeepEval G-Eval) are structurally blind to this bias. These frameworks reward fluency, coherence, and elaborateness—traits exhibited by over-prescribing systems—while failing to penalize incorrect decision quality. Consequently, systems with the worst outcome accuracy can receive the highest evaluation scores, creating a dangerous disconnect between perceived and actual performance in sequential advisory tasks.

Methodology: The EAV-DT Pipeline and Six-Arm Ablation
To address these issues, the authors propose and validate a two-stage architecture decoupling unstructured data ingestion from deterministic decision-making:

1. Stage 1 (Ingestion): Data is normalized into a typed, fixed-dimension Entity-Attribute-Value (EAV) state vector. In this study, Stage 1 is realized via a deterministic mapping of structured OULAD CSV fields to a 12-dimensional vector to isolate decision logic from extraction noise.
2. Stage 2 (Decision): A Decision Transformer (DT) operates on the EAV state vector. The DT is a small (624k parameters), offline-trained model that maps the state and a "Return-to-Go" (RTG) scalar to an intervention action via a deterministic argmax over output logits.

The study employs a six-arm ablation design on 800 students across four temporal cutoffs (days 14, 28, 56, 112):

• Arms A & B: Commercial RAG (Vector) and SQL-augmented RAG using GPT-4o.
• Arm C: EAV state vector presented directly to GPT-4o for decision-making.
• Arms D & E: EAV state vector processed by the ONNX Decision Transformer (Arm E uses a local CPU-native LLM for narrative generation only).
• Arm F: A supervised XGBoost classifier trained on the same oracle labels as the DT, serving as a tabular ML baseline.

Key Contributions and Results

1. Quantification of Intervention Bias:
   Zero-shot LLM arms (A, B, C) exhibit systematic over-prescription. At day 56, Arm C (GPT-4o + EAV) over-prescribed by 43.6 percentage points compared to the oracle. In a 10,000-student institution, this equates to ~4,300 unnecessary contacts per cycle. In contrast, supervised arms (D, E, F) achieved near-zero calibration error, successfully eliminating intervention bias.

2. Supervised Policy Learning Eliminates Bias:
   Both the trajectory-conditioned Decision Transformer (Arms D/E) and the snapshot XGBoost classifier (Arm F) achieved high Expert Policy Fidelity (EPF).

   • Arm D/E (Corrected v3): Achieved 99.8% EPF at day 14 and 95.4% at day 112, with a macro-F1 of 79% and macro-recall of 85% across all five action classes (including rare classes like reduce_load).
   • Arm F (XGBoost): Achieved 99.8% at day 14 but degraded to 89.9% by day 112, suggesting the DT's trajectory conditioning offers an architectural advantage for long-term sequential data.
   • Comparison: The DT significantly outperformed zero-shot LLM arms (47–67% EPF) and the majority-class baseline at most cutoffs.

3. Determinism and Auditability:
   Arms D and E achieved a 0% action flip rate due to the mathematical guarantee of ONNX argmax determinism. Conversely, LLM arms exhibited non-zero flip rates (Arm A: 2.9%, Arm B: 0.8%, Arm C: 0.2%), rendering audit trails unreliable in regulated environments.

4. Hardware Sovereignty and Latency:
   The architecture enables zero-egress, CPU-native inference.

   • Latency: The ONNX DT decision step executes in <5 ms on a commodity CPU, representing a 454.8× speedup over zero-shot LLM decisions (Arm C) and a 2,500× speedup over SQL-RAG pipelines (Arm B).
   • Cost: Arm E (local Granite 4.0 Small) incurred $0.00 API cost and zero data egress, satisfying FERPA/GDPR requirements, while matching the decision fidelity of the cloud-based Arm D.
   • Retraining: The DT can be retrained on new outcome data in under 4 minutes on a single CPU, allowing for hot-swapping of policies without GPU dependency.

5. The Evaluation Gap:
   The study confirms that DeepEval G-Eval rankings are orthogonal to outcome-validated accuracy. DeepEval ranked the SQL-RAG arm (B) higher than the vector RAG arm (A), despite Arm B having significantly worse decision accuracy (47.5% vs. 65.4% at day 14). This demonstrates that fluency-based evaluation is an unreliable proxy for decision quality in high-stakes sequential interventions.

Significance and Scope
The paper claims to demonstrate that zero-shot LLM deployment is structurally miscalibrated for conservative, high-stakes advisory tasks where "doing nothing" is a valid and necessary action. The primary contribution is the empirical validation that supervised policy learning (via DT or XGBoost) reliably eliminates this bias when trained on outcome-labelled data.

The authors propose the EAV-DT pipeline as a "living flywheel": a system where unstructured data is normalized into a typed state (EAV), compiled into a deterministic, sub-5ms decision model (DT), and retrained offline in minutes. This architecture replaces the latency and stochasticity of Agentic RAG with a sovereign, auditable, and cost-free decision engine.

Modesty and Limitations
The authors explicitly state that this study validates Stage 2 (decision quality on controlled, structured EAV input) only.

• No End-to-End Unstructured Claim: The study does not claim the full pipeline works on raw unstructured data; Stage 1 extraction noise (hallucinations, schema drift) is deferred to follow-up work.
• Oracle as Proxy: High fidelity (93–99%) reflects alignment with a researcher-defined hindsight rule, not necessarily superhuman generalization to real-world human advisors or noisy real-world inputs.
• Evaluation Gap: The finding that LLM-as-judge frameworks fail to detect intervention bias is presented as a specific domain failure for sequential interventions, not a general critique of LLM judges for all tasks.

In summary, the paper argues that for high-stakes sequential decision-making, the "RAG-replacement" is not a better prompt or a larger model, but a decoupled architecture that separates deterministic policy learning from narrative generation, ensuring calibration, auditability, and sovereignty.