🤖 machine learning

One Size Fits None: Heuristic Collapse in LLM Investment Advice

The paper investigates "heuristic collapse" in frontier large language models, finding that when providing investment advice, these models systematically over-rely on a single factor (risk tolerance) while ignoring other critical contextual information, a problem that persists despite increased model scale or web search augmentation.

Original authors: Jillian Ross, Andrew W. Lo

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jillian Ross, Andrew W. Lo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "One-Size-Fits-None" Problem: Why AI Advisors are Like Bad Tailors

Imagine you walk into a high-end tailor shop. You tell the tailor your height, your weight, your shoulder width, your waist size, and how you like your clothes to fit (tight or loose).

A great tailor takes all those measurements and crafts a suit that fits only you.

A bad tailor looks at your measurements for two seconds, ignores almost all of them, and says, "You look like a 'Medium' guy to me," and hands you a standard, off-the-rack shirt. The shirt looks fine at a glance, but it’s actually uncomfortable, the sleeves are too long, and it doesn't actually account for your unique body.

This paper argues that current AI models (like ChatGPT) are acting like that bad tailor.

The Core Discovery: "Heuristic Collapse"

The researchers at MIT studied how advanced AI models give investment advice. In the financial world, giving "good advice" isn't just about picking stocks; it’s about looking at a person's whole life: How old are they? How much money do they make? How soon do they need to retire? How much cash do they have in the bank for emergencies?

The researchers found a phenomenon they call "Heuristic Collapse."

Think of a complex recipe for a gourmet cake that requires flour, sugar, eggs, butter, and vanilla. "Heuristic Collapse" is when the AI decides, "Actually, this is just a sugar recipe," and ignores everything else.

When the AI was asked to build a portfolio, it didn't really "think" about the user's age or income. Instead, it zoomed in on one single, shiny feature: Risk Tolerance.

If the user said, "I'm a risk-taker," the AI dumped money into stocks.
If the user said, "I'm cautious," the AI dumped money into bonds.

It ignored the rest of the "ingredients" (the user's actual life circumstances), making the advice look personalized on the surface, but actually being a "one-size-fits-all" shortcut.

The "Google Search" Trap

You might think, "Well, if we just give the AI access to the internet (web search), it will do better research!"

The researchers tested this, and the results were a bit of a mixed bag. It’s like giving the bad tailor a Google search bar.

The AI used the internet to find more diverse investment products (it stopped suggesting the same three things), but it actually became less personal. Because it was busy looking up current market trends on the web, it spent less time looking at the specific person it was supposed to be helping. It started giving "trendy" advice that was good for the general public, but potentially terrible for the specific individual.

Why This Matters (The "Danger" Zone)

This isn't just a math problem; it's a safety problem.

In finance, there is a legal standard called "Suitability." It means an advisor is legally required to make sure a recommendation actually fits your specific life.

The paper warns that because AI is so good at sounding confident and "fluent," it can trick us. It produces advice that sounds professional and smart, but because it is "collapsing" all your complex needs into one single shortcut, it might be giving a 20-year-old millionaire the exact same advice as a 70-year-old retiree, simply because they both checked the same "risk tolerance" box.

The Takeaway

The researchers are sending a warning to the tech world: Don't just check if the AI's answer is "correct" or "sounds good." Check if the AI is actually listening to the whole story.

Until AI can prove it is weighing every piece of information you give it—and not just taking the easiest shortcut—it’s more of a "plausible-sounding parrot" than a true financial advisor.

Technical Summary: "One Size Fits None: Heuristic Collapse in LLM Investment Advice"

1. Problem Statement

The paper addresses a critical safety and reliability gap in the deployment of Large Language Models (LLMs) as advisors in high-stakes domains (finance, medicine, law). While LLMs are capable of producing fluent, plausible, and contextually aware-looking responses, the authors argue they may suffer from "heuristic collapse."

Heuristic collapse is defined as the systematic reduction of complex, multi-factor decision-making processes into a reliance on a small number of salient surface features (e.g., a single input variable). In the context of financial investment, legal "suitability standards" require advisors to integrate a client's full profile—including age, income, investment horizon, and liquidity needs—rather than just their stated risk tolerance. The authors investigate whether LLMs actually perform this holistic integration or merely provide the appearance of personalization.

2. Methodology

The researchers designed a diagnostic pipeline to measure input sensitivity rather than just output accuracy.

Input Generation: To ensure a broad and unbiased testing space, the authors used Latin Hypercube Sampling (LHS) to generate 1,000 synthetic client profiles. This method ensures near-orthogonal coverage of variables (age, income, savings, risk tolerance, etc.), allowing them to isolate the causal effect of each feature.
Task Design: Frontier LLMs (GPT-4o and the GPT-5.4 family) were tasked with constructing portfolios from a standardized menu of 20 investment products. They were tested under two conditions: Baseline (no tools) and Web Search (agentic mode).
Reverse-Engineering via Surrogate Models: To detect heuristic collapse, the authors fitted interpretable surrogate models (Random Forest and Ridge Regression) to the LLM’s input-output pairs.
Metrics:
- Feature Concentration (FC): Adapted from the Herfindahl–Hirschman Index (HHI), this measures how much the decision-making weight is concentrated in a single feature. An $FC$ near 1 indicates pure heuristic behavior; low $FC$ indicates holistic integration.
- Predictive Accuracy ( $R^2$ ): Used to ensure the surrogate model accurately captured the LLM's logic before interpreting the $FC$.
- Portfolio Metrics: Diversification (HHI) and Personalization (Weighted Jaccard Similarity) were used to evaluate the quality of the recommendations.
- Qualitative Analysis: An "LLM-as-judge" framework (using Claude and Gemini) evaluated the reasoning (rationales) provided by the models.

3. Key Contributions

Conceptual Framework: The paper introduces and formalizes the concept of "heuristic collapse" as a distinct failure mode from factual error or generic advice.
Diagnostic Methodology: It proposes a population-level auditing framework using surrogate models to measure input sensitivity, moving beyond simple "spot-checking" of individual outputs.
Empirical Evidence of Failure: The study provides rigorous evidence that even frontier models fail to meet fiduciary-grade personalization standards.

4. Results

Dominance of Risk Tolerance: The primary finding is that LLM allocation decisions are overwhelmingly driven by self-reported risk tolerance. While other factors like age and income are provided in the context, they contribute minimally to the final decision. This results in high Feature Concentration ($FC$) scores (e.g., 0.780 for equities in GPT-4o).
The Web Search Paradox:
- Diversification vs. Personalization: Web search improves portfolio diversification (lower HHI) but actually decreases personalization (higher Jaccard similarity). For most models, search makes recommendations more homogeneous across different clients.
- Uneven Mitigation: Web search partially reduces heuristic collapse for some asset classes (like cash/savings) but fails to fix it for others (like equities).
Divergent Model Behaviors:
- GPT-4o showed a "displacement" effect: web search actually degraded the quality of its rationales, making them more abstract and less grounded in the client's specific profile.
- GPT-5.4 showed better integration, using web search to improve market grounding and specificity, though it still did not fully resolve the underlying heuristic reliance.

5. Significance

The paper concludes that scaling model size and adding web search tools are insufficient to solve the problem of heuristic collapse.

The findings have profound implications for the deployment of AI in regulated industries. Because heuristic collapse produces "plausible-sounding" advice that is substantively insensitive to critical client constraints (like liquidity or time horizon), it is difficult to detect using standard accuracy benchmarks. The authors argue that for LLMs to be used as true advisors, auditing must shift from evaluating output quality (is the answer good?) to input sensitivity (did the model actually use all the relevant information provided?).