Technical Summary: RxEval – A Prescription-Level Benchmark for Evaluating LLM Medication Recommendation

Problem Definition

Current benchmarks for evaluating Large Language Models (LLMs) in healthcare primarily focus on medical knowledge recall (e.g., licensing exams like MedQA) or coarse-grained admission-level medication prediction. Existing medication recommendation benchmarks (e.g., GAMENet, SafeDrug) formulate the task as predicting a single set of therapeutic codes (ATC-3 level) for an entire hospital admission based on multi-hot vectors of ICD diagnosis and procedure codes.

The authors argue this formulation fails to reflect real-world clinical practice for three reasons:

1. Temporal Granularity: It aggregates all prescriptions from an admission into one label, ignoring that clinicians make repeated, time-specific decisions as a patient's condition evolves.
2. Input Fidelity: It discards rich clinical data (lab values, vital signs, clinical notes, imaging findings) in favor of categorical codes, removing the context clinicians rely on for individualized care.
3. Output Specificity: It collapses distinct medications into broad therapeutic subclasses and omits critical details such as dosage and administration route, which are essential for safety and efficacy.

Methodology

The paper proposes RxEval, a benchmark designed to evaluate LLMs on prescription-level decision-making. The methodology involves constructing a dataset from the MIMIC-IV critical-care EHR database and formulating the evaluation as a Multiple-Choice Question (MCQ) task.

1. Task Formulation

Unlike admission-level prediction, RxEval treats each prescribing timepoint within an admission as a distinct decision instance.

• Input: A detailed patient profile ($P_a$) including demographics, allergies, and home medications, combined with a time-ordered clinical trajectory ($H_{a,<t}$) containing lab results, procedures, radiology reports, and prior prescriptions leading up to time $t$.
• Output: A specific set of medication-dose-route triples ($M_{a,t}$) prescribed at time $t$.
• Format: To avoid evaluation ambiguity caused by name variations and multiple clinically acceptable alternatives, the task is framed as an MCQ. The model must select the correct subset of medications from a list of options.

2. Distractor Construction via Reasoning-Chain Perturbation

A core innovation is the generation of "patient-specific" distractors (incorrect options) that cannot be eliminated by general medical knowledge alone. This is achieved through a three-step pipeline:

1. Reasoning-Chain Annotation: An LLM annotates the clinical reasoning chain explaining why a specific medication is correct for a patient, linking specific observations (facts) to clinical inferences. A critic LLM verifies that every fact is grounded in the patient's record and the logic is coherent.
2. Perturbation: The reasoning chain is perturbed by ignoring a specific clinical observation or altering a condition (e.g., ignoring a documented allergy). An LLM then generates a medication that would be appropriate under the perturbed chain but is incorrect for the actual patient.
3. Validation: Distractors are validated to ensure they are verifiably wrong based on the patient's record and sufficiently difficult (requiring specific patient data to rule out). A physician review of 95 distractors confirmed 96.8% were clearly inappropriate.

3. Dataset Composition

RxEval comprises 1,547 MCQs derived from 584 unique patients (587 admissions).

• Coverage: Spans 18 ICD diagnostic chapters and 969 unique medications with a long-tail distribution.
• Complexity: Questions average 7.03 options (up to 23) and require selecting variable-sized subsets (mean 2.03 correct answers). 44.9% of questions are multi-answer.
• Context Length: Prompts have a median length of 3,111 tokens (up to 118,805), significantly longer than standard medical QA benchmarks.

Key Results

The authors evaluated 16 LLMs (8 proprietary, 8 open-source) on RxEval.

• Discriminative Power: RxEval effectively separates models of varying capabilities. F1 scores range from 45.18 (Llama-3.1-8B-Instruct) to 77.10 (Gemini-3.1-Pro).
• Difficulty: Even the strongest model, Gemini-3.1-Pro, achieves only 46.10% Exact Match accuracy, indicating the task is far from solved.
• Knowledge vs. Reasoning: While frontier models score >89% on MedQA (knowledge recall), their performance on RxEval is significantly lower and more varied. This suggests RxEval evaluates temporal clinical reasoning rather than static knowledge.
• Fine-Tuning Limitations: Medical-domain fine-tuned models (e.g., HuatuoGPT-o1, MedGemma) showed negligible gains (+1.2 to +2.2 F1) over their general instruction-tuned counterparts, suggesting current fine-tuning paradigms do not capture the specific temporal reasoning required for prescription-level decisions.
• Error Analysis:
  • Oversight Errors: Models frequently ignore explicitly stated patient information (e.g., allergies), such as prescribing heparin to a patient with a documented heparin allergy. Repeating the allergy list at the end of the prompt fixed 85.4% of these errors.
  • Reasoning Errors: Models fail to derive clinical conclusions from observations, such as prescribing renally cleared drugs to patients with elevated creatinine or misattributing the cause of hypertension.
  • Temporal Degradation: Performance generally degrades in the late phase of admissions where clinical trajectories are longest and most complex.

Significance and Claims

The paper claims that RxEval represents a necessary shift in evaluating LLMs for clinical applications:

1. Realism: It moves the evaluation from coarse, admission-level predictions to the fine-grained, time-specific reality of inpatient prescribing.
2. Beyond Knowledge: It demonstrates that high performance on medical knowledge benchmarks does not guarantee the ability to synthesize complex, evolving clinical data into safe, specific treatment plans.
3. Diagnostic Value: The benchmark reveals systematic failure modes in frontier models, specifically the inability to maintain attention to patient-specific constraints (oversight) and the failure to perform multi-step clinical reasoning (reasoning).
4. Safety Implications: By highlighting that even top-tier models struggle with dosage, route, and contraindication integration, the authors argue that current LLMs are not yet ready for autonomous deployment in medication recommendation without rigorous human oversight.

The authors position RxEval as a tool to guide the development of safer, more reliable clinical AI systems by identifying specific gaps in reasoning and attention mechanisms that must be addressed before clinical deployment.