💬 NLP

Vectors Are Not Neutral: Sensitive-Information Inference from Exported LLM Representations in Summarization

This paper demonstrates that compact vector representations exported from LLM summarization systems can leak sensitive information like patient race even when source documents are restricted, and shows that mitigation strategies such as SurfaceLoRA must be specifically targeted to the exact exposed vector artifact to effectively prevent such inference without compromising utility.

Original authors: Weixin Liu, Bowen Qu, Juming Xiong, Congning Ni, Bradley A. Malin, Zhijun Yin

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Weixin Liu, Bowen Qu, Juming Xiong, Congning Ni, Bradley A. Malin, Zhijun Yin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Digital Shadow" Problem

Imagine a hospital uses a super-smart AI robot to read a patient's long, complicated medical file and write a short summary for the doctor. The original file is locked in a high-security vault; only authorized doctors can see it.

However, to make the system work efficiently, the AI creates a "digital shadow" (a vector) of that file. This shadow is a compact code that helps other parts of the hospital system (like search tools, audit logs, or analytics) work faster. The hospital thinks, "The original file is safe, so this shadow must be safe too."

The paper's main discovery: The shadow is not safe. Even if the original file is locked away, this "digital shadow" still contains hidden clues about sensitive things, like the patient's race. If someone with access to the shadow (but not the vault) analyzes it, they can guess the patient's race with surprising accuracy.

The Experiment: A High-Stakes Test

The researchers tested this using a real-world scenario: Clinical Discharge Summaries.

The Task: The AI reads a patient's hospital notes and writes a "Brief Hospital Course" (a summary of what happened during their stay).
The Sensitive Clue: They used race (as recorded in the hospital's electronic records) as the test subject.
The Test: They took the "digital shadows" the AI created before it even finished writing the summary and asked a simple question: "Can a computer guess the patient's race just by looking at this shadow?"

The Result: Yes. The shadows were full of clues.

The Twist: One Shadow, Two Faces

The researchers looked at two different types of "shadows" the AI creates:

The "Last Word" Shadow (lasttok): This is the snapshot of the AI's brain right at the very last moment before it starts typing the summary. It's like a single photo of the AI's face right before it speaks.
The "Average" Shadow (meanpool): This is a blend of the AI's brain state across the entire prompt. It's like taking a blurry, averaged-out photo of the whole conversation.

The Surprise: The researchers tried to "sanitize" (clean) the "Last Word" shadow so it couldn't reveal the patient's race. They succeeded! The "Last Word" shadow became useless for guessing race.

BUT, the "Average" shadow was still full of clues. Cleaning one type of shadow did not clean the other.

Analogy: Imagine you have a jar of marbles (the data). You paint over the red marbles in the "Last Word" jar so they look white. But the "Average" jar is a mix of all the marbles, and the red ones are still visible there. If you only check the first jar, you think you're safe, but the second jar is still leaking secrets.

The Solution: "SurfaceLoRA"

To fix this, the authors created a new method called SurfaceLoRA.

Think of the AI as a student taking a test.

The Goal: The student needs to write a good summary (Utility).
The Problem: The student's writing style accidentally reveals their race (Leakage).
The Fix: SurfaceLoRA acts like a strict coach. During practice, the coach watches the student write. If the student's writing starts to hint at their race, the coach yells, "Stop! That's a clue!" and forces the student to relearn how to write the summary without those specific clues.

Crucially, the coach only watches the specific jar (the specific vector) that the hospital plans to use. If the hospital uses the "Last Word" jar, the coach trains specifically to hide clues in that jar.

The Results:

Success: When they targeted the "Last Word" jar, SurfaceLoRA made it impossible to guess the race (down to random chance), while the summary remained high-quality.
Failure: When they looked at the "Average" jar (which they didn't target), the race was still easy to guess.

The Golden Rule: Audit the Exact Thing You Use

The paper concludes with a very important warning for anyone building these systems:

You cannot assume that cleaning one part of the system cleans the whole thing.

If your system saves a "Last Word" vector, you must audit and clean that specific vector. If your system saves an "Average" vector, you must audit and clean that specific vector.

Analogy: Imagine you are trying to stop a leak in a boat. If you patch the hole in the front, the boat is still sinking because there's a hole in the back. You have to patch the exact hole where the water is coming in. You can't just patch the front and say, "The boat is safe."

Summary of Key Takeaways

Vectors aren't neutral: The compact codes (vectors) AI creates to summarize text still hold sensitive secrets, even if the original text is hidden.
Cleaning is specific: Fixing the privacy leak in one type of vector does not fix it in another type.
The Fix works (if targeted): The new method, SurfaceLoRA, can effectively hide sensitive info from a specific vector without ruining the quality of the summary.
Don't guess: You must identify exactly which vector your system exports and audit that specific one. You can't assume a general "privacy fix" covers all bases.

Technical Summary: Vectors Are Not Neutral

Problem Statement
Large Language Model (LLM) summarization systems often export compact vector representations of private inputs to downstream workflows for retrieval, monitoring, auditing, or analytics. While the source documents may remain access-restricted, these derived vectors are frequently handled under different access controls. The paper posits that even when raw texts are protected, these exported vectors may still support the inference of sensitive information about the individuals described, creating a residual information-disclosure risk. This issue is framed not merely as a malicious attack vector but as a governance and policy concern: downstream components (e.g., analysts, auditors, service providers) may have access to vectors without authorization to inspect the original text or structured sensitive attributes. If these vectors are treated as derived artifacts that should not reveal subject-level information, successful recovery of sensitive attributes constitutes a privacy failure.

Methodology
The authors instantiate this problem using clinical discharge-summary generation (specifically, Brief Hospital Course or BHC generation) as a high-stakes case study. They use Electronic Health Record (EHR)-recorded race as a controlled sensitive label for auditing.

Artifacts Audited: The study focuses on two specific prompt-derived artifacts computed before generation:
- lasttok: The hidden state of the final prompt token immediately before decoding.
- meanpool: The mean of all non-padding prompt-token hidden states, resembling embeddings used for retrieval or semantic indexing.
Evaluation Protocol: The authors employ post-hoc probing on frozen exported vectors. They train linear (multinomial logistic regression) and nonlinear (two-layer MLP) probes to predict the five-way EHR-recorded race label from the exported vectors. Performance is measured by probe accuracy and LeakageGap (the absolute deviation from the chance baseline of 0.20 in a balanced 5-way setting).
Mitigation Strategy (SURFACELORA): The paper introduces SURFACELORA, a parameter-efficient fine-tuning (PEFT) method targeting specific exported vectors.
- Mechanism: It combines a standard utility objective (summarization via LoRA) with an adversarial objective. A lightweight discriminator is attached to a designated exported vector (e.g., lasttok) during training.
- Gradient Reversal: A Gradient Reversal Layer (GRL) is used to reverse the gradients flowing from the discriminator back to the LoRA adapters. This encourages the model to preserve summarization utility while making the specific targeted vector less predictive of the sensitive attribute.
- Artifact-Specificity: The method is designed to mitigate leakage on the exact vector intended for export. The training-time discriminator targets a specific representation (e.g., lasttok), and the authors argue that mitigation must be aligned with the specific artifact retained in deployment.
Checkpoint Selection: The authors observe a non-monotonic trade-off between utility and leakage during training. Therefore, they propose a deployment procedure where checkpoints are selected based on a held-out validation set (bal_val) using a pre-specified leakage budget (e.g., LeakageGap $\le$ 0.025) and maximizing ROUGE-L, rather than simply using the final training step.

Key Results

Artifact-Specific Leakage: Reducing the recoverability of sensitive information from one exported artifact (e.g., lasttok) does not guarantee reduced recoverability from another (e.g., meanpool). In their experiments, SURFACELORA successfully reduced race recoverability from the targeted lasttok artifact to near-chance levels (approx. 0.203 accuracy) while preserving summarization utility. However, the untargeted meanpool artifact remained substantially predictive (approx. 0.316–0.323 accuracy), demonstrating that mitigation is not transferable across different vector representations.
Insufficiency of Prompting and Post-hoc Removal: Prompt engineering (adding neutrality directives) and post-hoc linear transforms (e.g., PCA removal, INLP) were found insufficient. Prompting failed to remove recoverable signals and introduced meta-term contamination. Post-hoc transforms reduced linear recoverability but failed to eliminate nonlinear leakage.
Trade-off Fragility: The relationship between adversarial strength ( $\lambda$ ) and leakage is non-monotonic. Continuing training beyond the optimal checkpoint can cause "rebound," where leakage increases despite continued adversarial pressure. This necessitates validation-based checkpoint selection.
Output vs. Representation: While SURFACELORA reduced representation-level leakage, it did not render the generated text invariant to sensitive attributes. Output-based attackers could still infer race from the generated summaries at rates above chance, though slightly lower than baselines.

Significance and Claims
The paper claims three primary contributions:

Framing: It frames exported summarization vectors as concrete, artifact-specific audit targets for sensitive-information inference, distinct from the raw text or the model as a whole.
Divergence of Artifacts: It demonstrates that near-chance recovery from one exported vector can coexist with substantial recovery from another, challenging the assumption that sanitizing a canonical token representation implies broader safety.
Targeted Mitigation: It introduces SURFACELORA, an artifact-targeted PEFT method that effectively reduces recoverability on a specific vector (e.g., lasttok) while preserving utility.

The authors conclude that privacy auditing and mitigation for summarization systems must be artifact-specific. A mitigation claim is only meaningful if it identifies both the sensitive information being audited and the exact vector artifact retained, logged, or exposed. If a system exports pooled embeddings (e.g., for retrieval), the audit and mitigation must target that specific representation, as sanitizing the last-token state alone is insufficient. The paper emphasizes that this is a deployment-oriented recipe: audit the specific vector used in the pipeline and apply mitigation aligned with that vector.

Limitations
The authors acknowledge that their study is limited to a specific empirical setting (BHC summarization with EHR-recorded race) and does not establish generality across all sensitive attributes, domains, or model architectures. They note that multi-attribute mitigation (simultaneously protecting race, age, gender, etc.) remains an open problem. Furthermore, they clarify that EHR-recorded race is treated as an administrative category, not a biological attribute, and the findings concern the recoverability of recorded categories and their textual proxies.