How much of an LLM-generated clinical corpus is actually new? A production-scale measurement of content redundancy for provenance classification

Technical Summary: "How much of an LLM-generated clinical corpus is actually new?"

Problem Statement
Clinical machine learning is increasingly relying on training corpora generated by Large Language Models (LLMs) rather than manually annotated by clinicians. These synthetic corpora are typically described, shared, and consumed based on their reported scale (e.g., total token count or number of extracted items). However, this practice assumes that volume equates to information content. The authors posit that synthetic corpora generated by applying consistent extraction rules to repetitive clinical text may harbor substantial duplication, where the apparent size of the corpus significantly exceeds the distinct information it carries. This inflation affects how training costs are estimated and, critically, how the distribution of the data is interpreted for clinical use. Prior to this work, the information content of multi-task clinical LLM output had not been measured against its raw size at a production scale.

Methodology
The study analyzes the complete output of a multi-agent clinical extraction pipeline applied to 167,034 patient narratives from the PMC-Patients corpus. The pipeline utilizes a generator model (Llama-3.3-70B) and a verifier model (MMed-Llama-3.1-70B) to produce structured outputs across eleven task channels, ten of which contain free text.

The core methodology involves Provenance-based Redundancy Decomposition (PRD), a token-level classification system that categorizes every output token into five provenance categories:

1. Unique source text: De-duplicated patient narratives.
2. Unique generated content: Model-produced text appearing only once.
3. Duplicated generated content: Model-produced text repeated across records.
4. Copied context: Source narratives reproduced verbatim within per-item context fields.
5. Scaffold: Structural metadata (identifiers, offsets, enumerated values).

To validate these findings independently of the provenance labels, the authors applied lossless compression (using gzip, bzip2, LZMA, and PPMD) to four distinct token streams: the full corpus, the trainable-unique subset, the copied-context stream, and the duplicated-generation stream.

Finally, the authors conducted a downstream controlled experiment to measure the impact of redundancy on model performance. They adapted a clinical encoder (BioClinical ModernBERT) on three versions of the corpus (raw, de-duplicated, and context-removed) under an identical token budget. The adapted encoders were evaluated on two external, human-annotated disease-recognition benchmarks (NCBI-Disease and BC5CDR-Disease) using a linear probe.

Key Contributions

1. Production-Scale Measurement: The paper provides the first measurement of the gap between apparent and actual information content in a multi-task clinical LLM corpus, establishing that raw token counts can overstate content by roughly ninefold.
2. Mechanism Identification: It identifies and separately quantifies two structurally distinct redundancy mechanisms:
   • Context-copy redundancy: Verbatim copying of source narratives into per-item fields (losslessly removable).
   • Generation-duplication redundancy: Repetition of generated text across records due to model regularities (requires de-duplication of content).
3. Reporting Standards and Tools: The authors propose a compact reporting standard for synthetic clinical corpora, introducing the Trainable-Unique Ratio (TUR) and Context-Copy Ratio (CCR). They also release an open-source tool implementing PRD to compute these metrics for any multi-task extraction pipeline.

Results

• Redundancy Magnitude: Of the 2.51 billion generated tokens, only 10.9% (272.6 million tokens) constitutes trainable-unique content. 79.4% is redundant, and 9.7% is scaffold. The redundant content alone is 19.1 times the size of the unique source text.
• Mechanism Distribution:
  • Context-copy redundancy dominates the Question-Answering-with-Reasoning (QAR) and Relation Extraction (RE) channels, accounting for 81.0% and 80.0% of their tokens, respectively.
  • Generation-duplication redundancy dominates the Risk-QA, Recommendations, and Risks channels.
  • The Summary channel serves as an internal control, showing almost no redundancy (0% copied context, 0.0005% duplication), proving that redundancy is a function of channel design rather than an inherent property of LLM extraction.
• Cost Structure:
  • Storage: Copied context accounts for ~7.5 GB of the ~10 GB text footprint. Re-serializing to reference narratives once per patient could reduce this to ~2.5 GB without information loss.
  • Compute: 64% of the free-text generation compute (298.3 million tokens) was spent re-emitting strings the model had already produced. This cost is unrecoverable.
  • Processing: Training on the raw corpus imposes a 9.2× processing multiplier compared to the unique subset.
• Downstream Impact: At an equal token budget, an encoder trained on the de-duplicated corpus outperformed the encoder trained on the raw corpus on both external benchmarks (e.g., +0.029 F1 on NCBI-Disease). The improvement was consistent across adaptation depths and replicated on BC5CDR-Disease.
  • The "context-removed" ablation (removing only copied context) recovered most of the benefit, indicating that the losslessly removable component drives the majority of the downstream cost.
  • Contrary to the hypothesis that redundancy disproportionately harms rare disease recognition, no reliable rare-specific benefit of de-duplication was observed; the gains were consistent overall and concentrated in common diseases.

Significance and Claims
The paper argues that the volume of synthetic clinical corpora is a poor proxy for information content. The findings have direct implications for the reproducibility and fairness of clinical AI:

• Reporting Integrity: Describing a corpus by raw token count (e.g., "2.51 billion tokens") misrepresents its informational richness. The authors recommend reporting unique-content size alongside raw size.
• Training Efficiency: Uncorrected redundancy up-weights longer, more complex presentations, skewing the token-level training distribution. De-duplicating before training is a low-cost, high-impact step that improves model performance without requiring additional data.
• Quality Control: Because redundancy is invisible at the record level (each record is well-formed) but emerges only in aggregate, provenance classification should become a routine quality-control step for synthetic data.
• Design Implications: The variation in redundancy across channels (e.g., Summary vs. QAR) demonstrates that redundancy is not an inevitable property of LLM extraction but a consequence of pipeline design, specifically whether a channel emits many context-bearing items per patient.

The authors conclude that synthetic clinical corpora remain a practical necessity for data at this scale, but their information content must be measured and described honestly to ensure fair and effective use in clinical machine learning.