💬 NLP

The Verbose Context Problem in Medical Records

This paper introduces PopMedQA, a benchmark built with the neopatient library to address the verbose context problem in population health analysis, demonstrating that domain-independent methods fail to handle the token inefficiency of longitudinal medical records and highlighting the need for domain-specific solutions.

Original authors: Shiva Kaul, Min-Gyu Kim, Anjum Khurshid, Sriram Vishwanath

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shiva Kaul, Min-Gyu Kim, Anjum Khurshid, Sriram Vishwanath

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a few clues on a desk, you are handed a library containing thousands of books. Every single page in these books is written in a very specific, repetitive code. To understand the story, you have to read every single word of every single page.

This is the "Verbose Context Problem" that the authors of this paper are tackling, specifically in the world of medical records.

Here is a breakdown of the paper's story, using simple analogies:

1. The Problem: The "Wordy" Library

In a hospital, doctors use short codes (like "ICD-10 I21") to describe what happened to a patient (like a heart attack). However, when you feed this data into an Artificial Intelligence (AI) brain, the AI doesn't understand the short code. It needs the full, long description: "ICD-10 I21: Acute myocardial infarction."

If you have one patient, that's fine. But in Population Health, doctors need to look at groups of 10 to 50 patients at once to find patterns.

The Analogy: Imagine trying to find a specific sentence in a book, but the book is actually a stack of 50 dictionaries. The AI has to read millions of words just to find the relevant clues. The paper calls this "verbose" (too wordy). The AI gets overwhelmed, like a student trying to read a whole library in one sitting, and starts making mistakes or missing the point.

2. The Solution: Building a New Test (PopMedQA)

The authors realized that existing tests for AI weren't good enough. Most tests just throw random "junk" words at the AI to see if it can find a needle in a haystack. But medical records aren't junk; they are too much useful information.

So, they built a new test called PopMedQA.

The Analogy: Instead of a generic reading test, they built a specialized "Medical Detective Exam." They created a library of synthetic (fake) patient records that are perfectly realistic but generated by a new tool they built called neopatient.
The Twist: The questions on this exam are designed so you can't just read one patient's file and guess the answer. You have to read all the files at the same time and connect the dots between them. It's like asking, "Which three of these 30 patients are secretly part of the same secret club?" You can't solve it by looking at them one by one.

3. The Experiment: Can the AI Handle the Library?

The authors tested many different AI models (from small ones to the biggest, smartest "frontier" models) on this new exam. They tried to fix the "too many words" problem using three common tricks:

Trick 1: Prompt Compression (The "Summary" Approach): They tried to shrink the text, like summarizing a book into a few sentences.
- The Result: It failed. The AI lost important details. It's like trying to summarize a complex medical history into a tweet; you lose the nuance needed to solve the mystery.
Trick 2: Agentic Decomposition (The "Teamwork" Approach): They tried to have the AI break the problem into smaller pieces, like hiring a team of detectives to each read one page and then report back.
- The Result: It failed. The AI couldn't keep the big picture together. The "team" got confused, and the cost to run them was too high.
Trick 3: Specialized Medical Training: They tested if AI models trained specifically on medical books did better.
- The Result: Surprisingly, no. A general, super-smart AI often did just as well as the medical specialist. The problem wasn't that the AI didn't know medicine; it was that it couldn't process the massive amount of text efficiently.

4. The Conclusion: We Need a New Kind of Map

The paper concludes that the current tools we have for AI (like summarizing text or breaking tasks into small pieces) do not work for this specific problem.

The Takeaway: The authors argue that we can't just force the AI to read faster. We need to change how we give the information to the AI. Instead of giving it a wall of text, we need to find a way to use the specific structure of medical data to help the AI understand the patterns without reading every single word.

In short: The paper says, "We built a hard test to show that current AI struggles when asked to read thousands of pages of medical records at once. Trying to summarize the pages or split the work doesn't help. We need to invent a new way to feed this data to AI that respects the unique structure of medical records."

Technical Summary: The Verbose Context Problem in Medical Records

Problem Definition: The Verbose Context Bottleneck

The paper identifies a critical limitation in applying Large Language Models (LLMs) to population health analytics, termed the verbose context problem. While LLMs offer flexibility over traditional rule-based systems, their utility is hindered when structured medical concepts (e.g., ICD-10 codes, SNOMED CT) are converted into token-inefficient textual representations.

In longitudinal electronic health records (EHRs), a single concept $c$ (e.g., a heart attack) is represented as a string $str(c)$ (e.g., "ICD-10 I21: Acute myocardial infarction"). This string is tokenized and embedded, creating a sequence of vectors. When thousands of such coded events pervade a patient's record, the total context length often exceeds 400,000 tokens. This verbosity inflates context lengths to a point where effective, efficient reasoning becomes computationally prohibitive or impossible for current models. Unlike existing long-context benchmarks that focus on "needle-in-a-haystack" retrieval (irrelevant distractors), this problem arises from the sheer density of necessary, yet verbose, structured data.

Methodology

1. PopMedQA Benchmark

To isolate and evaluate this specific failure mode, the authors introduce PopMedQA, a benchmark designed for reasoning over groups of 10–50 synthetic longitudinal patient records.

Scale: Questions typically range from 64K to 256K tokens.
Decomposition Resistance: A core design feature is that questions cannot be solved by decomposing them into individual patient-level queries. Tasks require holistic, in-context processing of the entire cohort to identify patterns, clusters, or anomalies.
Task Diversity: The benchmark includes nine computational tasks, including Planted Clique (finding cohesive subgroups), Clustering, Outlier Detection, Two-Sample Tests, Top-k selection, and Sorting based on complex clinical trajectories.
Validation: Questions undergo clinician review (M.D.s) for realism, difficulty, and coherence.

2. neopatient: Language-Controlled Generation

To generate the massive volume of synthetic data required (over 25,000 patients, 14M coded events), the authors developed neopatient, a library for language-controlled generation of artificial patient records.

Architecture: Unlike rule-based generators (e.g., Synthea), neopatient uses natural language descriptions to control patient trajectories.
Pipeline:
1. Sampling: An LLM generates a "patient recipe" (demographics and temporal segments) to ensure long-term clinical coherence.
2. Generation: The LLM produces longitudinal medical events (descriptions and target codes) for each segment.
3. Matching: A vector database maps free-text descriptions to standardized medical ontologies (ICD-10, SNOMED, LOINC, RxNorm) to prevent hallucinations.
4. Verification: An LLM validates the final record against the original recipe.
Output: Records are produced in the Medical Event Data Standard (MEDS) format.

3. Experimental Evaluation

The authors evaluated a wide range of models (7B to frontier scale, 128K to 2M context windows) on PopMedQA. They tested four families of interventions to mitigate the verbose context problem:

Prompting Strategies: Baseline (truncated descriptions), raw codes only, and a "codebook" method (mapping unique codes to integer IDs to reduce redundancy).
Prompt Compression: Rendering text to images (Glyph pipeline) and LLMLingua-2 text chunk compression.
Reasoning: Chain-of-thought prompting (increasing inference-time compute).
Agentic Decomposition: Multi-turn interaction systems (MARS, LongCEPO, Claude Code) attempting to break tasks into sub-queries.

Key Results

The evaluation yielded several counter-intuitive findings regarding the state of long-context reasoning in medical domains:

Dual Requirement: High performance requires both clinical competence and long-context capability. Smaller models with long-context windows often failed due to a lack of domain knowledge, while smaller models with domain knowledge failed due to context limits.
Fragility of Generic Compression: Domain-independent prompt compression methods (e.g., text-to-image, LLMLingua-2) significantly degraded performance. These methods appeared to strip away critical information necessary for complex medical reasoning.
Limited Value of Medical Pretraining: General-purpose frontier models often outperformed specialized medical models. This suggests the primary bottleneck is not a lack of clinical knowledge, but the ability to perform robust, multi-hop reasoning over verbose contexts.
Ineffectiveness of Agentic Decomposition: Multi-turn agentic systems failed to achieve absolute performance gains that justified their computational and financial costs. In many cases, decomposition harmed instruction-following and absolute performance compared to monolithic baselines, confirming that PopMedQA tasks are resistant to simple decomposition.

Significance and Claims

The paper claims that the verbose context problem remains a significant, unresolved barrier to population-scale EHR reasoning. The authors assert that:

Domain-Independent Methods Fail: Current general-purpose techniques for handling long contexts (compression, agentic decomposition) are insufficient for the specific structural challenges of medical records.
Opportunity for Domain-Specific Structure: There is a "significant unrealized opportunity" to exploit the specific structure of medical data (e.g., the ontology of codes, the temporal nature of events) within language model inputs to enable robust population-scale reasoning.
Utility of Controlled Generation: The neopatient library demonstrates that language-controlled generation can create realistic, privacy-preserving synthetic cohorts for AI research in population health, bypassing the need for custom simulation code.

The authors conclude that while frontier models demonstrate superior performance, the field must move beyond generic context engineering to develop methods that specifically address the token inefficiency of structured medical concepts.