💰 quantitative finance

When Summaries Distort Decisions: Information Fidelity in LLM-Compressed Financial Analysis

This paper demonstrates that LLM-based compression of financial documents can distort investment decisions by introducing fidelity losses through decontextualization and model dependency, and proposes an "Agentic Context Compression" framework that generates and audits multiple candidate summaries to preserve decision-relevant context.

Original authors: Hoyoung Lee, Suhwan Park, Seunghan Lee, Jun Seo, Jaehoon Lee, Sungdong Yoo, Minjae Kim, CheolWon Na, Zhangyang Wang, Zach Golkhou, Minkyu Kim, Sotirios Sabanis, Alejandro Lopez-Lira, Dhagash Mehta, So

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hoyoung Lee, Suhwan Park, Seunghan Lee, Jun Seo, Jaehoon Lee, Sungdong Yoo, Minjae Kim, CheolWon Na, Zhangyang Wang, Zach Golkhou, Minkyu Kim, Sotirios Sabanis, Alejandro Lopez-Lira, Dhagash Mehta, Soonyoung Lee, Chanyeol Choi, Wonbin Ahn, Yongjae Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: When Summaries Lie (Even When They Tell the Truth)

Imagine you are a financial investor trying to decide whether to buy a stock. You have a 50-page report from a company (like a quarterly earnings report). It's too long to read, so you ask a super-smart AI assistant to summarize it into a short, 20-bullet list.

You expect the summary to be a "mini-me" of the original report. If the original report says, "We made a lot of money, but we have a huge lawsuit coming up that might wipe it out," you expect the summary to say, "We made money, but a lawsuit is a risk."

The Problem: The paper finds that AI summaries often act like a bad translator who only tells you the good news and forgets the "but." The summary might say, "We made a lot of money!" It sounds true, and it sounds fluent. But because it left out the "lawsuit" part, you might decide to buy the stock (a "Bull" decision), whereas reading the full report would have made you sell it (a "Bear" decision).

The authors call this Information Fidelity. It's not about whether the facts are true; it's about whether the summary keeps the context needed to make the right decision.

The Two Main Culprits

The paper identifies two specific ways these AI summaries mess up the decision-making process:

1. Decontextualization (The "Isolated Fact" Trap)

Imagine you are reading a weather report.

The Headline: "It is 90°F outside." (This is the Headline Fact).
The Context: "But there is a massive heatwave warning, and the humidity is so high it feels like 110°F." (This is the Context).

If an AI summarizes this by just saying, "It is 90°F," it is factually correct. However, if you are deciding whether to go for a run, that summary is dangerous. It stripped away the caveat (the heat warning) that changes the meaning of the fact.

In finance, AI often keeps the "90°F" (e.g., "Revenue grew 20%") but drops the "heat warning" (e.g., "but only because we sold off our best assets, and future growth is flat"). This leads investors to make the wrong move.

2. Model Dependency (The "Different Chefs" Problem)

Imagine you ask three different chefs to summarize the same recipe.

Chef A focuses on the spices and says, "This is a spicy dish."
Chef B focuses on the sauce and says, "This is a sweet dish."
Chef C focuses on the meat and says, "This is a savory dish."

All three chefs are telling the truth about parts of the recipe, but they are all giving you a different picture of the whole meal. The paper found that different AI models (like GPT, Gemini, or Qwen) will compress the same financial document in different ways. One might make a company look "Bullish" (good), while another makes the same company look "Bearish" (bad), just because they chose to highlight different facts.

The Solution: The "Agent" Detective

The authors propose a new method called Agentic Context Compression. Think of this not as a single summarizer, but as a detective agency.

Here is how it works:

Hire Two Detectives: Instead of asking one AI to summarize the document, the system asks two different AI models to create their own summaries.
The Audit: A third "Agent" (a smart AI supervisor) looks at both summaries. It notices where they disagree.
- Detective A says: "The company is doing great."
- Detective B says: "The company is doing great, but they have a debt issue."
Check the Source: The Agent doesn't just guess. It goes back to the original 50-page report (the "Source") and uses a search tool to find the specific paragraph about the debt.
Pick the Winner: The Agent checks which summary is more faithful to the original text. If the original text mentions the debt, the Agent rejects the summary that left it out, even if that summary was shorter or sounded better.

The Results: What They Found

The researchers tested this on real financial documents from big companies (like Apple, Microsoft, etc.).

The Bad News: When they used standard AI summarization, the "decision flip" rate was huge. About 33% of the time, the summary made investors want to buy a stock that the full report suggested they should sell (or vice versa).
The Good News: When they used their new "Agent" method, that error rate dropped significantly (down to about 20%).
The Industry Test: They even tested this in a real-world commercial product used by professional investors. The "Agent" method didn't just reduce errors; it actually improved the accuracy of the investment forecasts compared to the standard summaries.

The Takeaway

The paper concludes that in high-stakes fields like finance, efficiency is not enough. A summary that is short and grammatically perfect is useless if it changes your mind about whether to invest.

To fix this, we shouldn't just ask AI to "summarize." We need to ask AI to "summarize and check its own work against the original source to make sure it didn't accidentally delete the warning labels."

In short: Don't just trust the summary. Trust the summary only if it has been audited to ensure it didn't leave out the "but."

Technical Summary: When Summaries Distort Decisions: Information Fidelity in LLM-Compressed Financial Analysis

1. Problem Formulation

In high-stakes financial domains, decision-makers rely on Large Language Models (LLMs) to compress long documents (e.g., 10-Q filings, earnings call transcripts) into shorter contexts. While existing evaluations focus on reconstruction quality, factual consistency, or surface-level information preservation, they often fail to assess whether the compressed context preserves the decision implied by the original source.

The paper defines Information Fidelity as source-relative decision preservation: a compression has high fidelity if the belief distribution induced by the compressed text remains close to that of the original source. The core failure mode identified is not merely hallucination, but decision distortion, where a fluent and factually plausible summary leads a downstream decision-maker to a different investment judgment (Bear/Neutral/Bull) than the full source text would support. This distortion can arise from:

Decontextualization: Retaining salient headline facts while stripping away necessary caveats, offsets, comparisons, or expectation framing required for correct interpretation.
Model Dependency: Different compression models preserving different subsets of evidence from the same source, leading to systematic, model-specific tilts in the resulting decision.

2. Methodology

2.1 Experimental Setup

Datasets: The study utilizes 300 10-Q MD&A sections and 297 quarterly earnings call transcripts from S&P 100 firms (Fiscal 2025 Q1–Q3).
Decision Model: A fixed decision model ( $E$ ), specifically GEMINI-3.1-FLASH-LITE, maps text to a belief distribution over $\{bear, neutral, bull\}$ .
Metrics:
- Decision Flip Rate: The fraction of documents where the top predicted label changes after compression.
- Total Variation Distance (TVD): Measures the magnitude of belief shift even when the top label remains unchanged.
- Noise Floor: Established by having the decision model re-read the original source without compression to quantify inherent model randomness.

2.2 Compression Approaches Evaluated

Naive Prompt: A single-pass LLM compression selecting the "most material" facts within a fixed bullet budget.
Contextualization: A prompt modification instructing the LLM to preserve details that make facts interpretable (e.g., comparisons, caveats) rather than just standalone salient facts.
Token Pruning: Methods like LLMLingua and LongLLMLingua that drop low-information tokens.
Integrator: Merging summaries from multiple models without source verification.
Agentic Context Compression (ACC): A proposed two-stage agentic process:
- Generation: Multiple candidate compressions are generated from the same source using different models.
- Auditing: An agent audits disagreements between candidates against the original source. It uses tools to grep and read specific source spans to verify if a candidate overclaims (e.g., making evidence seem more bullish/bearish than the source supports).
- Selection: The agent selects the candidate with the lowest overclaim risk and best source-locus coverage.

2.3 Diagnostic Analysis

The authors employ an offline fact inventory (inspired by FActScore and SAFE) to decompose source texts into Headline facts (primary signals), Context facts (calibrators/caveats), and Boilerplate. This allows for the quantification of decontextualization (the disproportionate removal of context facts).

3. Key Results

3.1 Fidelity Loss is Significant

Decision Flips: All one-shot compression baselines significantly exceed the "noise floor" of the decision model. For 10-Q MD&A, naive compression caused a 33.0% decision flip rate, compared to an 11.0% noise floor. For earnings calls, the rate was 23.9% vs. 8.8%.
Belief Drift: Even without a label flip, compression causes significant shifts in belief distributions (TVD), indicating a loss of nuance.
Budget Sensitivity: Increasing the compression budget (more bullets) reduces TVD but does not eliminate decision flips, suggesting that simply adding more space does not solve the issue of which information is selected.

3.2 Diagnostic Patterns

Decontextualization: Under naive compression, the proportion of "context" facts dropped from 25% in the source to 9% in the compressed output for MD&As. Restoring these dropped context facts in a diagnostic "add-back" experiment recovered the original decision in 37% of flipped cases, significantly outperforming the recovery rate of boilerplate or random facts.
Model Dependency: Different compressors induce different decision shifts for the same ticker. For example, one model might shift a stock toward "Bearish" while another shifts it "Bullish," with high-TVD points varying across models. This confirms that compression-induced distortion is not random but systematic and model-specific.

3.2 Effectiveness of Agentic Context Compression (ACC)

ACC Performance: ACC reduced the MD&A decision flip rate to 20.3% and earnings call flips to 18.5%, outperforming both naive compression and the "Integrator" baseline (which simply merges summaries without auditing).
Industry Case Study: In a deployment within a commercial equity-forecasting product, ACC improved forecasting Information Coefficient (IC) by 8.3% over the original-source baseline and 23.8% over naive compression, while reducing source-relative decision flips by 59.5%.

4. Contributions

Information Fidelity Criterion: The paper introduces a decision-centered metric for evaluating context compression, moving beyond factuality to assess whether the source-implied decision is preserved.
Empirical Evidence of Distortion: Using real-world financial data, the study demonstrates that LLM compression frequently alters investment judgments, identifying decontextualization and model dependency as primary mechanisms.
Agentic Context Compression (ACC): The authors propose and validate a source-grounded procedure that generates multiple candidates and audits them against the original source to mitigate decision flips.

5. Significance and Claims

The paper argues that in financial decision-making, preserving isolated facts is insufficient. The value of a summary lies in its ability to retain the contextual qualifiers (caveats, comparisons, framing) necessary to interpret those facts correctly.

The authors claim that:

Compression should be evaluated not just by efficiency or surface factuality, but by its ability to preserve decision-relevant context.
Decision distortion is a systematic failure mode that can be mitigated by auditing multiple compressed views against the source, rather than relying on a single model's summary.
In production systems, maintaining information fidelity is critical for retaining downstream forecasting signals, as naive compression can degrade model performance even when the summary appears fluent and plausible.

6. Limitations

The authors note that their findings are specific to financial disclosures from S&P 100 firms and may not generalize to other domains. Fidelity is measured via a decision model rather than human analysts or realized market outcomes. Additionally, ACC reduces but does not eliminate fidelity loss, particularly if all candidate compressors omit the same critical information.