EveryQuery: Zero-Shot Clinical Prediction via Task-Conditioned Pretraining over Electronic Health Records

Imagine you are a doctor trying to predict what might happen to a patient in the next month. You have their entire medical history: every test, every diagnosis, every pill they've ever taken.

For a long time, the best AI models for this job worked like a crystal ball that only works by guessing. To predict if a patient would get a specific disease, the AI would simulate thousands of "what-if" futures for that patient. It would ask, "What if they get sick? What if they don't? What if they take a different pill?" It would run these simulations over and over, then count how many times the disease appeared to give you an answer.

The Problem with the Crystal Ball:

It's slow: Running thousands of simulations for every single patient takes forever.
It's noisy: If the disease is rare (like a specific type of allergy), the AI might run 20 simulations and never see it happen once. It then has to guess, "Well, maybe 0 out of 20 means 0% chance," which is a terrible guess for a rare event.
It's rigid: You can't just ask the AI, "Will they get diabetes?" You have to set up a complex simulation pipeline just to ask that one question.

Enter "EveryQuery": The Direct Answer Machine

The authors of this paper built a new kind of AI called EveryQuery. Instead of simulating thousands of futures, EveryQuery works more like a smart librarian or a personalized search engine.

Here is how it works, using a simple analogy:

1. The Old Way (Autoregressive Models)

Imagine you want to know if it will rain tomorrow. The old AI model doesn't just look at the clouds and say "Yes." Instead, it generates 20 different versions of tomorrow.

In 19 of those versions, it's sunny.
In 1 version, it rains.
It tells you: "Based on my 20 guesses, there's a 5% chance of rain."
The Flaw: If you ask about something super rare (like a meteor hitting the city), it might generate 20 sunny days and say "0% chance," even if the risk is actually 0.1%. It misses the needle in the haystack because it's too busy looking at the hay.

2. The New Way (EveryQuery)

EveryQuery is different. You don't ask it to guess the future. You give it a specific question (a "query") along with the patient's history.

You ask: "Will this patient have a heart attack in the next 30 days?"
The AI looks: It reads the patient's history specifically looking for signs of a heart attack. It doesn't simulate 20 different futures. It just gives you a direct probability: "There is a 12% chance."
The Magic: It was trained on millions of different questions and answers. It learned that when you ask about this specific code (heart attack), you should look at these specific parts of the medical history.

Why is this a Big Deal?

1. It's Lightning Fast
The old way is like asking a chef to cook 20 different meals to see which one you'd like. EveryQuery is like ordering off a menu. It's about 3,000 times faster. It gives you an answer in a single step instead of running a marathon of simulations.

2. It's Great at Finding Rare Things
This is the paper's biggest win. Because the old AI relies on counting how often a rare event shows up in its simulations, it often misses rare diseases. EveryQuery doesn't need to "see" the event happen in a simulation to know it's a risk. It learns to recognize the signals that lead to that event.

Analogy: If you are looking for a specific rare coin in a pile of sand, the old AI digs 20 random handfuls of sand. If it doesn't find the coin, it says "No coin." EveryQuery is like a metal detector; it scans the whole pile specifically for the signal of that coin, regardless of how rare it is.

3. It's "Promptable" (You can talk to it)
With the old models, you couldn't just ask a new question. You had to reprogram the machine. With EveryQuery, you can just type in a new medical question (as long as it fits a specific format), and it answers it immediately without needing to be retrained. It's like having a chatbot that knows medicine inside and out.

The One Catch (The "Readmission" Problem)

The paper admits the system isn't perfect yet. It's great at asking, "Will the patient get Diabetes?" or "Will they get Flu?"

But it struggles with complex "OR" questions. For example: "Will the patient be readmitted to the hospital?"

This is a tricky question because readmission can happen for any of 70 different reasons (a broken leg, a heart attack, an infection, etc.).
EveryQuery is currently limited to asking about one specific thing at a time. To answer the readmission question, you'd have to ask it 70 separate questions and try to combine the answers, which is clunky and less accurate.
Analogy: EveryQuery is a master at finding a specific key in a room. But if you ask, "Is there any key in the room?" it has to check every single drawer one by one and add up the results, rather than just scanning the room for the concept of "keys."

The Bottom Line

The authors have created a new type of medical AI that is faster, more accurate for rare diseases, and easier to use than the current state-of-the-art models. It trades the ability to "imagine" infinite futures for the ability to give a direct, smart answer to a specific question.

While it still needs to learn how to handle complex "any of these things" questions, it represents a massive step forward in making AI practical for doctors who need quick, reliable answers about patient risks.

1. Problem Statement

Current Electronic Health Record (EHR) foundation models predominantly rely on autoregressive generation (e.g., ETHOS, CoMET). While these models can perform zero-shot prediction by generating synthetic future patient trajectories and aggregating statistics, they suffer from three critical limitations:

Computational Inefficiency: Obtaining a single prediction requires sampling many future trajectories (e.g., $K=20$ ), making inference slow and expensive.
Statistical Noise: Estimates for rare events are highly variable and quantized because low-prevalence events may not appear in sampled trajectories, leading to zero or near-zero probability estimates.
Lack of Promptability: Users cannot directly condition predictions on specific clinical questions (e.g., "Will this patient have a heart attack in 30 days?") without designing custom aggregation pipelines. Existing models are not natively "promptable."

The authors argue that a new paradigm is needed that satisfies Zero-Shot Inference, Efficiency (single forward pass), and Promptability (structured task specification) simultaneously.

2. Methodology: EveryQuery

The authors propose EveryQuery, an EHR foundation model that replaces autoregressive generation with task-conditioned pretraining.

Core Concept

Instead of learning a distribution $p(x)$ (next token prediction) and inferring outcomes via sampling, EveryQuery learns a conditional distribution $p(y | x, q)$ , where:

$x$ : The patient's medical history (tokenized as a sequence of codes).
$q$ : A structured query specifying the clinical task.
$y$ : The binary outcome (event occurrence).

The query is defined parametrically as $q = (c, \Delta t)$ , asking: "Does code $c$ occur within time window $\Delta t$ after the current time?"

Architecture

Backbone: A bidirectional transformer (ModernBERT-base) with 22 layers, 768 hidden dimensions, and ~149M parameters.
Input Construction: The query code $c$ is embedded and prepended to the patient's history sequence $[q; x_1, \dots, x_L]$ . This allows the query token to attend to all patient tokens via standard self-attention, effectively creating a cross-attention mechanism within a unified model.
Prediction Heads: Two separate Multi-Layer Perceptrons (MLPs) map the final hidden state of the query token to:
1. $\hat{y}_{occurs}$ : Probability the event occurs.
2. $\hat{y}_{cens}$ : Probability the data is censored (i.e., the patient record ends before the prediction window).

Pretraining Strategy

Data: MIMIC-IV dataset (converted to MEDS format).
Query Sampling: Queries are constructed by randomly sampling code $c$ from a subset of 10,000 codes (out of 11,467 total) and fixing $\Delta t = 30$ days.
Training Objective: A multi-task loss function combining Binary Cross-Entropy (BCE) for event occurrence and censoring.
- $L = L_{cens} + \lambda L_{occurs}$
- The occurrence loss is masked for censored samples to prevent learning from incomplete data.
Zero-Shot Mechanism: By training on a uniform distribution of random (patient, query) pairs, the model learns to generalize to any query expressible in the language without fine-tuning.

3. Key Contributions

Task-Conditioned Pretraining: Introduces a novel training paradigm for EHRs where the model is explicitly trained to answer arbitrary structured queries, enabling true zero-shot inference.
Efficiency & Determinism: Eliminates the need for trajectory sampling. EveryQuery produces deterministic predictions via a single forward pass, offering a massive speedup over autoregressive baselines.
Rare Event Robustness: Demonstrates that conditioning directly on the target event code allows the model to detect rare events without relying on the statistical frequency of those events in generated trajectories.
Prompt Specificity: Provides evidence that the model's internal representations are organized by the query rather than the patient, confirming that the task specification actively modulates the attention mechanism.

4. Experimental Results

Evaluated on MIMIC-IV against an autoregressive baseline (MEDS-EIC-AR) across 39 randomly sampled prediction tasks (plus 30-day readmission).

Performance: EveryQuery outperformed the autoregressive baseline on 82% of tasks (32/39).
- Mean AUC Improvement: +0.16 (95% CI: [0.10, 0.22]).
- Statistical Significance: $p < 10^{-5}$ (Wilcoxon signed-rank test).
Generalization (OOD): The model generalized seamlessly to Out-of-Distribution (OOD) codes (codes not seen as query targets during pretraining). It won 94% of OOD tasks vs. 71% of In-Distribution tasks, with no statistically significant performance drop.
Rare Event Advantage:
- There is a strong negative correlation ( $\rho = -0.32$ ) between event prevalence and EveryQuery's advantage.
- The autoregressive model's performance degrades significantly for rare events (AUC correlated with prevalence, $\rho=0.64$ ), while EveryQuery's performance remains stable regardless of prevalence.
Efficiency: EveryQuery is approximately 3,000x faster than the autoregressive baseline for a single task (30 seconds vs. 25 hours for the full cohort), as it avoids 20 trajectory rollouts per patient.
Embedding Analysis: UMAP and cosine similarity analysis confirmed that embeddings cluster by query (same query across different patients are highly similar) rather than by patient, validating the task-conditioning mechanism.

Limitation Identified:

Readmission Task: EveryQuery failed on the 30-day readmission task (AUC ~0.56 vs. 0.64 for AR). This is because readmission requires disjunctive reasoning (any of 70 admission codes), which the current single-code query language cannot express natively. Naive aggregation of 70 separate queries failed to recover the implicit information captured by the generative model.

5. Significance and Future Directions

Significance:
EveryQuery challenges the dominance of autoregressive generation for clinical foundation models. It proves that for discriminative tasks, direct task conditioning is superior in terms of speed, stability (especially for rare events), and usability (promptability). It offers a practical solution for clinical decision support where low-variance, fast, and specific predictions are required.

Future Work:

Query Language Expansion: The primary limitation is expressiveness. Future work must extend the query language to support disjunctions (OR logic), conjunctions (AND logic), and numeric predicates to handle complex tasks like readmission or "any cardiovascular event."
Hybrid Approaches: Combining EveryQuery for efficient, targeted predictions with autoregressive models for exploratory or highly compositional tasks.
Broader Evaluation: Testing on multiple datasets and more complex time horizons (beyond the fixed 30-day window used in this study).

In conclusion, EveryQuery represents a shift from "generating futures to find answers" to "asking specific questions to get answers," offering a more efficient and robust framework for clinical AI.