Deployment-Centered Evaluation: Predicting Query-Level Rejection Risk in a Clinical LLM System

Technical Summary: Deployment-Centered Evaluation of Query-Level Rejection Risk in Clinical LLMs

Problem Statement
The integration of Large Language Models (LLMs) into clinical workflows introduces a critical evaluation gap. Traditional benchmark-centered evaluations prioritize objective "correctness" (e.g., accuracy on medical licensing exams) and aggregate performance across historical queries. However, these static assessments fail to capture real-world utility, specifically whether a clinician will actually accept or "reject" a generated response. Rejection is a noisy construct reflecting low perceived utility, which can occur even when an output is factually correct due to mismatches in stylistic expectations, context, or workflow integration. Furthermore, existing benchmarks rely on densely annotated datasets that are expensive to construct and difficult to refresh, creating blind spots for dynamic, real-world deployment. The core challenge is to predict, before a response is generated, whether a specific user interaction will result in rejection, utilizing the sparse feedback naturally available in deployed systems.

Methodology
The authors propose a deployment-centered evaluation framework centered on a pre-response rejection prediction model embedded within an Electronic Health Record (EHR) system at an academic medical center.

• Task Formalization: The goal is to learn a score function $f(\cdot)$ that predicts the probability of a user rejecting an LLM response ($r_i = 1$) versus accepting it ($r_i = -1$). The input consists of the user query $q_i$ and deployment-specific context: provider type ($p_i$), department name ($d_i$), the specific LLM model used ($m_i$), and the length of the patient record ($l_i$).
• Data Source: The study utilizes 4.5 months of live user interactions (September 2025 – April 2026). From 74,729 total interactions, only 1,196 contained explicit binary feedback ("thumbs up/down"). After filtering for duplicates and missing metadata, the final annotated dataset comprised 878 query-annotation pairs.
• Model Architecture: The authors employ a logistic regression model trained on query embeddings (generated via text-embedding-3-large) combined with one-hot encodings of the deployment context.
  • Feature Selection: A grid search over 31 possible feature combinations (including subsets of embeddings, provider, department, model, and context length) was performed to maximize AUROC on a validation set.
  • Training Protocol: The study utilizes a prospective analysis approach. An initial model is trained on the first 8 weeks of data. For subsequent 19 weeks, the model is dynamically retrained each week by incorporating new data with true labels, simulating a rolling training window to adapt to shifting user-LLM trends.
• Baselines: Performance is compared against models using only query embeddings, LLM-judge classifiers (using GPT-4.1) with and without context, and naive "always accept/reject" classifiers.

Key Contributions

1. Prospective Rejection Prediction: The authors demonstrate the feasibility of training a pre-response model to predict user rejection on future queries using sparse, real-world feedback.
2. Downstream Utility: The rejection score is framed as a deployable decision tool. By adjusting operating thresholds, the same model supports two distinct use cases:
   • Abstention: High-precision filtering (preemptively withholding likely-rejected responses) to reduce verification fatigue.
   • Guardrail Activation: High-recall triggering (flagging potential issues for additional review) to prevent low-quality outputs from reaching the user.
3. Role of Deployment-Specific Context: The paper establishes that leveraging metadata (provider type, department, model) significantly outperforms relying solely on query content.

Results

• Performance: The best-performing model (incorporating Embedding + Provider + Department + Model) achieved an AUROC of 0.719 (95% CI: [0.670, 0.767]) on the prospective test set.
• Comparison: This outperformed the query-embedding-only baseline (AUROC 0.618) and LLM-judge baselines (AUROC ~0.52–0.53).
• Feature Importance: Quantitative analysis revealed that department name and provider type were the most frequent features in top-performing configurations. Removing embeddings while retaining context features resulted in only a slight performance drop, suggesting context is highly informative in data-sparse settings.
• Downstream Metrics:
  • At a high-precision threshold ($\beta = 0.12$), the model achieved 0.88 precision (3 false positives), suitable for abstention.
  • At a high-recall threshold ($\beta = 4.0$), the model achieved 0.99 recall, suitable for guardrail activation.
• Qualitative Insights: Analysis of rejection rationales showed significant variation by role and department. For instance, physicians in cardiology had high rejection rates due to demands for specific clinical details, whereas nurse practitioners often accepted high-level summaries. Similarly, Critical Care physicians rejected queries that Hospital physicians accepted, indicating that acceptance criteria are shaped by departmental workflow and role-specific expectations.

Significance and Claims
The paper claims to offer a starting point for deployment-specific evaluations that move beyond static benchmarks to actionable interventions. Its primary significance lies in shifting the evaluation target from "correctness" to "user acceptance," acknowledging that in clinical settings, utility is defined by whether the output fits the user's specific context and workflow.

The authors modestly note that their findings are derived from a single academic medical center and are constrained by the sparsity of passive feedback (only ~1.6% of users provided feedback). They argue that while their results highlight the feasibility of predicting rejection using deployment-specific context, future work is needed to address selection bias and generalize these findings to broader deployment contexts. The work does not claim to solve all clinical AI safety issues but demonstrates that who is asking, in what context, and for what purpose matters as much as the question itself in determining system success.