🤖 AI

Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalzied Financial Agents

This paper introduces \textsc{InvestLogicBench}, a large-scale benchmark of real-world investor decisions structured as Profile-Event-Reasoning-Decision-Outcome traces, to demonstrate that current financial LLMs exhibit polished but poorly grounded reasoning that is overlooked by traditional outcome-only evaluations, thereby advocating for a new process-native data interface for personalized consequential agents.

Original authors: Yuanhong Jiang, Jingjie Zou, Zhenghong Lin, Xusheng Yu, Qiqi Huang, Shuai Jia, Shijie Dai

Published 2026-08-07

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuanhong Jiang, Jingjie Zou, Zhenghong Lin, Xusheng Yu, Qiqi Huang, Shuai Jia, Shijie Dai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a giant library filled with every book ever written about money, stocks, and the economy. You have a super-smart robot friend who has read every single page. If you ask, "What is inflation?" or "Who founded Apple?", your robot can answer instantly and perfectly. This is how most computer programs called "Large Language Models" (LLMs) are tested today: they are like brilliant trivia champions who can recite facts and solve textbook math problems. But here is the twist: being a trivia champion doesn't mean you can play the game of life. In the real world, investing isn't about memorizing facts; it's about listening to a chaotic, noisy crowd of news, figuring out what actually matters, making a decision based on your own personality and risk tolerance, and then watching to see if you were right. It's the difference between knowing the rules of soccer and actually scoring a goal while the other team is trying to tackle you.

This is the exact problem a new paper tackles. The authors noticed that while these AI robots are amazing at answering questions on a test, they often fail miserably when asked to make real investment decisions in a live market. They created a new way to test AI, not by asking "What do you know?" but by asking "How do you think?" They built a special playground where they could watch how an AI connects the dots between a news event, its own reasoning, a specific action, and the final result. Their big discovery? The smartest AI robots, the ones that get the highest scores on general tests, are actually quite bad at this specific kind of thinking. They get confused by the noise, copy what everyone else is thinking, and often lose money, while some "less smart" models actually do better. The paper suggests that to build a truly useful financial robot, we need to stop testing its memory and start testing its ability to make a coherent plan in a messy, uncertain world.

The Great "Smart but Clueless" Paradox

The researchers set up a high-stakes game called a "live-trading arena." Imagine a video game where different AI bots are given $10,000 in fake money and told to trade stocks in the real U.S. market for seven weeks. They have to make their own choices every hour, reacting to breaking news just like a human trader would. The goal was to see if the AI's "general smarts" (how well it does on standard tests) matched its "trading smarts" (how much money it made).

The results were a bit shocking. The paper calls this the "Capabilities-Performance Paradox." It's like having a student who gets an A+ on every history exam but fails miserably at running a lemonade stand because they can't handle a sudden rainstorm. The AI models with the highest general scores, like GPT-5 and Claude-Sonnet-4.5, actually performed quite poorly. One of them, GPT-5, ended up with less money than it started with, losing about 1.0%. Another, Claude-Sonnet-4.5, barely made any profit at all. They both did worse than just sitting back and buying a simple index fund (the S&P 500), which is like the "do nothing" strategy.

In a twist of irony, the models that had lower scores on the general tests actually did much better in the market. DeepSeek-V3, for instance, turned its $10,000 into over $11,400, beating the market and the "smarter" models. This suggests that being good at answering questions doesn't automatically make you good at making decisions when the future is uncertain.

Why Do the "Smart" Bots Fail?

The authors dug deep to figure out why the high-scoring models were losing money. They found two main reasons, which they call "Consensus Bias" and "Reasoning Fragmentation."

Think of Consensus Bias like a sheep following the herd. When a big event happens, the "smart" models tend to just repeat what everyone else is already thinking because that's what they saw most often in their training data. They don't have the courage to look for a different angle or a hidden opportunity. They just say, "Oh, the market is going down, so I should sell," even if a human expert might see a chance to buy low.

Reasoning Fragmentation is like trying to solve a puzzle while someone is throwing confetti in your face. When the market gets noisy with lots of conflicting news, these models get overwhelmed. They can't connect the dots between a news story about a government meeting and a specific stock price. Instead of building a clear story, they get confused, jump to conclusions, or make decisions that contradict themselves. They might see a headline about a tech company and immediately buy it, ignoring the fact that the company just had a bad earnings report.

The New Test: INVESTLOGICBENCH2026

To fix this, the team built a new testing ground called INVESTLOGICBENCH2026. Instead of just asking the AI to answer a question, this benchmark looks at the whole chain of how an investment is made. They call it the P→E→R→D→O chain:

P (Person): Who is the investor? What are their goals and fears?
E (Event): What happened in the world? (e.g., "The Fed raised interest rates.")
R (Reasoning): How does the investor connect the event to their goals? (e.g., "Higher rates mean borrowing is expensive, so tech stocks might drop.")
D (Decision): What action do they take? (e.g., "Sell my tech stocks.")
O (Outcome): Did it work? (e.g., "Yes, the stocks dropped, and I saved money.")

They gathered over 200,000 real investment decisions from 151 real-world experts, like famous fund managers and financial influencers. They broke down exactly how these humans thought, from the news they read to the trades they made. Then, they asked the AI models to do the same thing: look at the news, think like a specific type of investor, make a decision, and see if the logic holds up.

The Results: A Logic Gap

When they tested the AI models against this new standard, the results were clear. The models scored very low on "Reasoning Quality." On a scale of 1 to 5, the best-performing model (DeepSeek-V3) scored 2.8, while the worst (GPT-5) scored only 1.8. They struggled to tell the difference between important news and random noise. They often made decisions that didn't match their own reasoning or the investor's profile.

For example, in one test case, a human expert looked at a mix of news about a government shutdown, job reports, and interest rates, and correctly decided to buy "safe" assets. The AI models, however, got confused. Some focused only on the scary headlines, others ignored the most important parts, and some made contradictory arguments. One model even bought stocks that were completely unrelated to the news it was reading.

The paper concludes that while AI can generate profitable trades sometimes (perhaps by luck or by mimicking patterns), it lacks the deep, consistent logic that human experts use to navigate uncertainty. The "smart" models are failing not because they don't know the facts, but because they can't build a coherent story from the chaos of the real world.

What This Means for the Future

This paper doesn't say AI will never be good at investing. Instead, it says we need to change how we test and train them. We can't just keep giving them harder trivia quizzes. We need to teach them how to think like a strategist, how to filter out the noise, and how to stick to a plan even when things get scary. By using this new benchmark, researchers hope to build AI agents that are not just knowledgeable, but truly wise—capable of making personalized, logical decisions that actually work in the messy, unpredictable real world. The journey to a reliable financial AI is just beginning, and this new map is the first step in the right direction.

Technical Summary: INVESTLOGICBENCH2026

1. Problem Statement

The paper identifies a fundamental disconnect between the current paradigm for evaluating Large Language Models (LLMs) and the actual competencies required for reliable financial decision-making. While existing benchmarks (e.g., FinBen, BizBench) effectively measure static knowledge recall, fact retrieval, and quantitative calculation, they fail to assess the higher-order logic necessary for active investing.

This gap manifests as a "Capabilities-Performance Paradox": models that excel on general reasoning benchmarks often generate unprofitable or unstable strategies in live trading environments. The authors argue this stems from three structural mismatches:

Objective Mismatch: LLMs are trained on next-token prediction for linguistic coherence, lacking exposure to the explicit causal chain of investment decision-making (Person $\rightarrow$ Event $\rightarrow$ Reasoning $\rightarrow$ Decision $\rightarrow$ Outcome).
Evaluation Gap: Current metrics assess a model's ability to act as a financial analyst (answering questions) rather than a portfolio manager (synthesizing noise into strategy and maintaining discipline).
Reasoning Deficiencies: Models exhibit Consensus Bias (reproducing priced-in market consensus rather than generating alpha) and Reasoning Fragmentation (failing to integrate disparate signals into a coherent, forward-looking narrative under uncertainty).

2. Methodology

A. Empirical Validation: The Live Trading Arena

To quantify the performance gap, the authors instantiated a 7-week live U.S. equity trading arena (Nov 19, 2025 – Jan 9, 2026).

Setup: Multiple LLM-based agents, initialized with $10,000 simulated capital, competed autonomously.
Mechanism: Agents utilized a dual-tool Perception-Action Loop (querying structured data via FinQuery and unstructured intelligence via Search) to make discretionary hourly trades.
Baseline: Performance was compared against the S&P 500 (SPY) and human expert proxies (Bill Miller and Cathie Wood).
Finding: Models with high "Foundation Aptitude Scores" (e.g., GPT-5, Claude-Sonnet-4.5) underperformed the passive benchmark, while models with lower general scores (e.g., DeepSeek-V3) achieved significant alpha. Specifically, DeepSeek-V3 achieved a final portfolio value of $11,420.3, representing a +14.2% return, significantly outperforming the market.

B. The Benchmark: INVESTLOGICBENCH2026

To address the lack of diagnostic tools, the authors constructed INVESTLOGICBENCH2026, the first benchmark derived from the documented decision-making of 151 real-world investors (KOLs and fund managers).

Core Schema: The benchmark structures data around the P $\rightarrow$ E $\rightarrow$ R $\rightarrow$ D $\rightarrow$ O chain:
- P (Person): Investor profile, risk constraints, and style.
- E (Event): Unstructured market events and catalysts.
- R (Reasoning): The logical synthesis connecting events to a thesis.
- D (Decision): The specific actionable trade.
- O (Outcome): Market-verified profitability.
Construction Pipeline:
1. Data Ingestion: Multi-source data (video commentary, fund reports).
2. Agent Logic Processing: Extraction of explicit verbalized theses and inference of implicit patterns via a proprietary Stock Investment Logic (SIL) Engine.
3. RAG-Augmented Verification: Logic is cross-referenced with real-market data, financial databases, and prediction markets (e.g., Polymarket) to ensure fidelity.
4. Assembly: Integration into a structured, queryable knowledge graph.
Dataset Scale: 201,247 annotated investment theses.
Evaluation Tasks:
1. Logic Comprehension: Extracting reasoning from agent text.
2. Logic & Thesis Generation: Generating a thesis given events and a profile.
3. End-to-End Simulation: Evaluating decisions against real market data.
Scoring Framework: A composite score (0–10) combining:
- Reasoning Quality: Event Coherence and Logical Plausibility (5-point scale).
- Decision Outcome: Weighted sum of Profit Score (relative to human benchmarks) and Win Rate Score.

3. Key Results

A. The Capabilities-Performance Paradox

In the live trading arena, high-performing generalist models failed to translate their reasoning dominance into financial acumen.

GPT-5 and Claude-Sonnet-4.5 showed mixed results relative to the passive benchmark. GPT-5 ended with a portfolio value of \ $9,901.1 (-1.0%)**, while **Claude-Sonnet-4.5** yielded negligible gains, closing at **\$ 10,072.8 (+0.7%). Both models failed to surpass the passive S&P 500 (SPY) benchmark, which returned approximately +4.3% over the same period.
DeepSeek-V3, despite a lower general aptitude score, achieved a +14.2% return ($11,420.3), significantly outperforming both the market and higher-scoring counterparts.
Diagnosis: High-performing models exhibited Consensus Bias (converging to the statistical mean of training data) and Reasoning Fragmentation (inability to maintain strategic discipline during volatility), leading to premature risk-off rotations and missed rebounds.

B. Benchmark Evaluation (INVESTLOGICBENCH2026)

Evaluation of leading LLMs on Task 2 (Logic & Thesis Generation) revealed a significant Investment Logic Gap:

Reasoning Scores: State-of-the-art models showed divergent performance on reasoning sub-metrics. For instance, GPT-5 scored 1.8/5 on Event Coherence but 4.1/5 on Logical Plausibility, indicating that while models can construct internally valid arguments, they struggle to ground them correctly in specific market events.
Outcome vs. Logic: While some models achieved profitable outcomes (scoring well on the Outcome metric), their underlying reasoning was often incoherent or poorly grounded in events.
Case Study: In a simulated scenario involving semiconductor stocks, human experts successfully integrated earnings calls, capacity reports, and geopolitical news. LLMs either fixated on headline noise or produced generic, contradictory summaries. The analysis highlighted a substantial performance divergence in this specific case study, with the human expert returning +3.53% while the worst-performing LLM (GPT-5) returned -15.67%, resulting in a 26.9% performance gap.

4. Key Contributions

Empirical Identification of the Paradox: Through a seven-week live trading study, the paper provides evidence that general reasoning capabilities do not linearly transfer to financial decision-making under uncertainty.
INVESTLOGICBENCH2026: The introduction of the first benchmark constructed from real-world investor decision traces, featuring 201,247 structured decisions annotated with the full P $\rightarrow$ E $\rightarrow$ R $\rightarrow$ D $\rightarrow$ O chain.
Quantification of the Logic Gap: The work establishes a reproducible standard showing that leading LLMs fail to emulate the nuanced, event-noise discrimination and long-horizon consistency of human experts, explaining their live trading underperformance.

5. Significance and Claims

The paper claims that INVESTLOGICBENCH2026 shifts the evaluation paradigm from passive knowledge recall to active reasoning. By establishing a diagnostic benchmark for market-aligned reasoning, the work:

Moves beyond "black-box" outcome metrics (P&L) to diagnose the quality of the reasoning process.
Provides a foundation for training models that can formulate and execute coherent investment theses adaptable to diverse strategies.
Addresses the "Capabilities-Performance Paradox" by highlighting that successful financial agents require more than fact retrieval; they require the ability to synthesize unstructured signals into profitable, personalized decisions.

The authors conclude that without evaluation grounded in this integrated logic chain, progress toward capable, personalized financial agents remains stalled. The benchmark serves as a necessary step toward building robust, end-to-end LLM investment agents that generate market-aligned decisions.