Can Open-Weight Models Compete on Financial Text Comprehension?

Technical Summary: Can Open-Weight Models Compete on Financial Text Comprehension?

Problem Statement
While open-weight language models from Chinese AI labs have recently closed the performance gap with proprietary frontier models on general benchmarks, their reliability on real-world financial tasks remains untested. A significant challenge in evaluating Large Language Models (LLMs) in 2025–2026 is benchmark contamination, where public datasets are absorbed into training corpora. Furthermore, financial annual reports present a uniquely demanding domain due to their length (exceeding 700 pages), heterogeneity (mixing narrative, regulatory boilerplate, and complex tables), and jurisdictional diversity (covering 22 countries and various accounting standards). Prior benchmarks often relied on short contexts or single documents, failing to capture the complexity of retrieving specific financial metrics from unstructured, multi-page reports.

Methodology
The author updated the Financial Touchstone benchmark, expanding it from 2,878 questions over 480 reports to 2,967 question-context-answer triplets across 495 international annual reports (v1.2). The dataset covers all Global Industry Classification Standard (GICS) sectors and includes reports from 2021–2023.

• Model Evaluation: The study evaluated 20 models from 10 providers, expanding the previous set of 11. This includes recent open-weight models (e.g., Kimi K2.6, GLM 5, DeepSeek V3.2) and proprietary flagships (e.g., Qwen3-Max, Claude Opus 4.6, Gemini 2.5 Pro).
• Pipeline: A Retrieval-Augmented Generation (RAG) pipeline was employed where reports are chunked into 1,000-token windows. The top-5 chunks are retrieved using a fixed embedding model (text-embedding-3-small) and FAISS. Crucially, the retrieval step is part of the task; failures to retrieve relevant context are treated as system failures, not excused as upstream issues.
• Evaluation Metrics: Performance is measured by:
  1. Accuracy: Recall of required information against a human-curated "golden answer."
  2. Hallucination Rate: The share of claims unsupported by the source text.
  3. Failure Taxonomy: Errors are categorized as Retrieval Failures, Model Comprehension Errors, or Hallucinations.
• Quality Control: A model-in-the-loop sanity check and dual-annotation process were used to revise 67 incomplete answers and 15 hallucinated golden answers. The human baseline accuracy is 82.8% with a 2.8% hallucination rate.

Key Results

• Model Performance: Anthropic's Claude Opus 4.6 achieved the highest accuracy (88.4%), followed by Claude Sonnet 4.6 (86.7%). Notably, the open-weight Kimi K2.6 ranked third (83.5%), surpassing the human baseline. Non-reasoning models GLM 5 (82.0%) and Mistral 3 (81.3%) also ranked in the top five, challenging the assumption that reasoning architectures are prerequisites for strong financial comprehension.
• Hallucination vs. Accuracy: Google's Gemini 2.5 Pro maintained the lowest hallucination rate (0.08%), significantly below the human baseline, but ranked 13th in accuracy (76.6%). Conversely, Mistral 3 had a high hallucination rate (13.0%) despite decent accuracy. Kimi K2.6 emerged as a strong balance, ranking third in accuracy with a low hallucination rate (0.13%).
• Error Sources: Information retrieval remains the primary bottleneck, accounting for 48.9% of all failures. Model comprehension errors accounted for 44.7%, and standalone hallucinations for 9.1%. When retrieval fails, accuracy across all models drops drastically from 77.9% to 12.0%.
• Content Filtering: A novel finding documents that Chinese models exhibit geopolitical content filters that refuse legitimate financial questions. Across 23,736 attempts, 18 refusals (0.08%) persisted in the benchmark data. Refusals were triggered by mentions of Chinese political leaders (e.g., in Kweichow Moutai reports), Hong Kong protests, or state-owned banks. Crucially, refusal behavior was found to be route-dependent: identical requests routed through third-party proxies were often answered, while native API calls were refused.

Significance and Claims
The paper claims three substantive shifts in the landscape of financial AI:

1. Open-Weight Competitiveness: Open-weight models, particularly from Chinese labs, have closed the gap with proprietary frontier models in financial text comprehension, with some (Kimi K2.6, GLM 5) outperforming established reasoning models.
2. Reasoning vs. Non-Reasoning: The correlation between "reasoning" architecture and financial performance has weakened. Non-reasoning models (Mistral 3, GPT-4o) and open-weight models are competitive with, or superior to, dedicated reasoning models (e.g., Magistral, Grok 4) on this specific task.
3. Deployment Risks: The study identifies content filtering in Chinese models as a measurable deployment risk for global financial analysis, where legitimate queries regarding specific jurisdictions or entities may be blocked based on geopolitical triggers, a behavior that varies by access route.

The author concludes that while model generation capabilities have improved, the retrieval bottleneck remains the most significant barrier to high-fidelity financial AI. They suggest that improving the RAG stack (e.g., via GraphRAG) offers more potential for performance gains than swapping generators alone. The dataset and evaluation framework are released publicly to support reproducibility.