Technical Summary: InvestPhilBench

Problem Statement

Large Language Models (LLMs) are increasingly deployed as investment research assistants, yet existing financial benchmarks fail to evaluate whether models can accurately reconstruct and apply the specific procedural decision frameworks of expert investors. Current benchmarks focus on factual retrieval (e.g., FinanceBench), numerical reasoning (e.g., FinQA), or agent-based decision outcomes (e.g., INVESTORBENCH). They do not test if a model understands the sequential, conditional, and "kill-criterion" logic that defines an investor's philosophy (e.g., Warren Buffett's "Circle of Competence" gate or Ray Dalio's regime-dependent logic).

The paper argues that investment philosophy knowledge is distinct from financial calculation or document retrieval because it is procedural, attributed, contradictory, and comparative. Models often fail by:

• Omitting Kill Criteria: Proceeding with analysis after a gate that should terminate the process (e.g., evaluating a pharma company's P/E ratio despite it being outside Buffett's circle of competence).
• Flattening Operationalizations: Confusing distinct definitions of shared concepts (e.g., applying Graham's mechanical "net-net" screen to a question about Buffett's "margin of safety").
• Temporal Conflation: Merging contradictory historical positions (e.g., Buffett's 2001 rejection of airlines vs. his 2016 purchase of them).
• Voice Diffusion: Blending the distinct analytical personas of different investors.

Methodology

1. Cognitive Taxonomy (L1–L8)

InvestPhilBench introduces an eight-layer cognitive taxonomy to evaluate reasoning depth:

• L1–L3 (Factual Retrieval): Principle identification, source attribution, and operationalization recall.
• L4–L5 (Procedural Reconstruction): Reconstructing multi-step decision processes in correct order and comparing frameworks across investors.
• L6–L8 (Generative Application): Recognizing contradictions, applying frameworks to novel scenarios, and extrapolating to unseen macroeconomic developments.

2. Dataset Architecture (v0.6)

The release comprises a primary-source-verified dataset:

• 118 Principle Cards: Reviewer-resolved investment principles with explicit operational criteria and source citations.
• 25 Decision Framework Cards: Structured JSON representations of investor decision processes, annotated with topology metadata (e.g., linear_sequential, parallel_threshold, conditional_branch) and gate logic.
• 243 QA Questions: 197 development and 46 held-out test questions, including 8 expert-difficulty "hard" questions targeting specific gate-level failure patterns.
• Gold Reasoning Programs (GRPs): For L4 and L7 questions, the dataset includes structured "gold programs" defining the exact sequence of gates, verdicts (PROCEED/REJECT/INCONCLUSIVE), and required elements, enabling dual evaluation of execution (final verdict) and program accuracy (reasoning path).

3. Evaluation Protocol

The benchmark employs a three-condition design to decompose performance gaps:

• Condition A (Closed-book): Tests pre-training knowledge.
• Condition B (PKB-augmented/RAG): Tests retrieval-augmented reasoning using top-3 retrieved principle cards.
• Condition C (Oracle): Tests procedural application by injecting the exact ground-truth framework card.

4. Automated Scoring Pipeline (BASP)

To replace qualitative rubrics, the paper introduces the Benchmark Automated Scoring Pipeline (BASP), comprising five algorithmic metrics:

1. OGRS (Ordered Gate Reconstruction Score): Penalizes both missing gates and incorrect ordering.
2. KCCS (Kill Criterion Coverage Score): Measures F1 coverage of the specific conditions that trigger a "kill" (rejection).
3. SAP@k (Source Attribution Precision): Evaluates attribution at entity, document, and context levels.
4. IVP (Investor Voice Precision): Uses embeddings to ensure the model's response retains the specific lexical and conceptual signature of the attributed investor.
5. CKCA (Compound Kill Criterion Accuracy): Evaluates institutional frameworks with multi-component kill criteria.

Additionally, the Failure Mode Detection Protocol (FMDP) algorithmically detects six failure modes (e.g., Hallucinated Gate, Temporal Conflation) with computable rules. A stricter metric, Gate Reconstruction Accuracy (GRA), evaluates per-gate reconstruction (name, verdict, elements) to expose deficits masked by composite scores.

Key Results (v0.6 Sanity Wave)

The paper reports a preliminary "sanity wave" (W1) evaluating four models (Claude Sonnet 4.6, GPT-5.5, Gemini Flash Lite, Gemini 3.5 Flash) under Condition A (closed-book) using a mixed-judge setup.

• Provider-Tier Split: Frontier models (Claude, GPT) significantly outperform cost-efficient models (Gemini) across all layers, with a sharp collapse at L8 for the latter.
• The Composite vs. GRA Divergence:
  • Composite Saturation: The BASP-composite score for frontier models is high (e.g., Claude L4 = 0.932), showing no "cliff" between factual (L3) and procedural (L4) layers. This suggests composite scoring rewards fluent prose and hides procedural gaps.
  • Procedural Deficit Exposed by GRA: When evaluated with the stricter Gate Reconstruction Accuracy (GRA), frontier models drop significantly (L4 GRA ≈ 0.77; L7 GRA ≈ 0.57–0.62). This reveals a residual procedural deficit where models correctly name gates but fail to reconstruct the kill criteria or verdict logic.
• Failure Modes: The most prevalent failure modes are Kill Criterion Omission (FM-3, 42.5%) and Operationalization Flattening (FM-5, 35.0%). Frontier models exhibit low rates of these failures compared to cost-efficient models, which show high rates of procedural collapse.
• Three-Condition Pilot: A small pilot (n=10) suggests that while providing the correct source (Oracle) improves scores over closed-book (Knowledge Gap), targeted retrieval (Condition B) sometimes outperforms full-framework injection (Condition C), likely due to long-context distraction. A residual Application Gap persists even with the correct source in context.

Significance and Claims

The paper positions InvestPhilBench as a benchmark-and-methodology contribution rather than a definitive model leaderboard. Its primary claims are:

1. Methodological Innovation: It is the first benchmark to evaluate LLMs on the procedural reconstruction of expert investment frameworks, moving beyond factual recall or outcome-based evaluation.
2. Metric Design: It demonstrates that composite scoring saturates at the frontier, masking procedural deficits. The Gate Reconstruction Accuracy (GRA) metric is presented as the necessary tool to expose these deficits, distinguishing between fluent prose and correct gate logic.
3. Failure Mode Taxonomy: It provides a quantified taxonomy of systematic errors (e.g., Temporal Conflation, Voice Diffusion) with algorithmic detection rules, offering a diagnostic framework for model improvement.
4. Contamination Resistance: Through dynamic perturbation (metric substitution, temporal displacement) and a "living investor" career-boundary protocol, the benchmark aims to resist data contamination from pre-training corpora.

The authors explicitly state that the empirical results presented are a "preliminary sanity wave" intended to stress-test the metric design. The full multi-model leaderboard, de-confounded unified-judge results, and human-reliability studies are deferred to the v1.0 release. The paper concludes that distinguishing genuine procedural understanding from pattern-matched retrieval is critical for the safe deployment of LLMs in institutional finance, and InvestPhilBench provides the infrastructure to make this distinction.