Technical Summary: FINESSE-Bench

Problem Statement

While Large Language Models (LLMs) are increasingly deployed in financial analysis, reporting, and risk management, robust evaluation of their domain-specific competence remains incomplete. Existing open benchmarks such as FinQA, ConvFinQA, and TAT-QA have advanced financial question answering and numerical reasoning but primarily focus on extracting information from financial reports. They lack an explicit hierarchy of professional difficulty, failing to measure the transition from foundational knowledge to expert-level reasoning. Furthermore, broader resources like FinanceBench, PIXIU, and FinBen, while expanding task coverage, often underrepresent critical applied domains such as technical analysis, derivatives trading, and scenario-based portfolio management. Consequently, there is a gap in evaluating how model performance degrades as task complexity increases and how well models generalize to specialized, professionally relevant scenarios.

Methodology

The authors introduce FINESSE-Bench, a hierarchical benchmark suite comprising eight specialized datasets with a total of 3,993 questions. The design is grounded in two core principles: difficulty hierarchy and domain specialization.

Dataset Construction

The suite integrates three distinct types of data sources:

1. Exam-Oriented Hierarchies: Inspired by professional certifications, the suite includes:
   • CFA-like Levels 1–3: Covering ethics, quantitative methods, financial reporting, valuation, and portfolio management. Level 1 targets foundational literacy; Level 2 focuses on linked item sets and multi-step calculations; Level 3 addresses expert-level strategic synthesis.
   • CMT-like Level 2: Focused on technical analysis theory, chart patterns, and market statistics.
   • CFTe-like Level 1: Covering basic technical analysis concepts (trends, moving averages).
2. Applied Trading Tasks:
   • Trading_TA: Pattern recognition, momentum strategies, and back-testing.
   • Trading_derivatives: Options pricing, Greeks, hedging, and arbitrage strategies.
3. Olympiad-Style Reasoning:
   • VLigaBench-ru: A Russian-language dataset of 324 problems in micro/macroeconomics, financial mathematics, and game theory, emphasizing reasoning and calculation over standard QA.

Evaluation Protocol

The benchmark employs a unified evaluation protocol for heterogeneous task types (multiple-choice, numerical answers, short open-ended responses, and case-linked questions).

• Scoring: For tasks where exact matching is insufficient, the authors utilize an LLM-as-judge paradigm (specifically using GPT-5.2) to assign binary correctness scores to free-form answers, grounded in automated evaluation literature.
• Metrics: Primary metrics include accuracy with 95% confidence intervals computed via bootstrap. The evaluation analyzes performance across three aggregated groups: exam-like, public benchmarks (for comparison), and trading/TA.
• Analysis Dimensions: The study investigates transfer gaps between public and specialized benchmarks, performance degradation across difficulty levels (Level 1 → 3), within-family scaling effects, and the discriminative power of question difficulty (analyzing "unanimous fail," "unanimous success," and "mid-band" question shares).

Key Contributions

1. FINESSE-Bench Suite: The release of eight specialized datasets (3,993 questions) designed to evaluate financial competencies hierarchically, moving from foundational to expert levels.
2. Hierarchical Evaluation Design: A framework that explicitly measures performance degradation as models transition from basic financial knowledge to complex, multi-step expert tasks.
3. Expanded Domain Coverage: Inclusion of underrepresented areas such as technical analysis, derivatives trading, and Russian-language financial reasoning, complementing existing report-centric benchmarks.
4. Unified Protocol: A standardized evaluation approach combining fixed prompting, deterministic inference, and LLM-as-judge scoring for heterogeneous financial tasks.
5. Diagnostic Insights: Empirical evidence demonstrating that strong performance on classical public benchmarks does not guarantee robustness in professionally oriented, specialized financial scenarios.

Key Results

• Transfer Gap: A consistent performance gap exists between classical open benchmarks and FINESSE-Bench groups. Most models exhibit a positive transfer gap (degradation) when moving from public benchmarks to exam-like or trading/TA tasks. For instance, specialized financial models (e.g., Fin-R1, Fino1-8B) and smaller general-purpose models show significant drops (0.27–0.36) in performance on FINESSE-Bench compared to public sets.
• Difficulty Hierarchy: The CFA-like hierarchy effectively captures performance degradation. While some models maintain or slightly improve performance from Level 1 to Level 2, all evaluated models show a distinct decline in performance at Level 3, confirming the suite's ability to test expert-level synthesis.
• Discriminative Power: Unlike classical benchmarks (e.g., TAT-QA, FinQA) which often have high rates of "unanimous success" (questions solved by all models), FINESSE-Bench subsets (particularly CFA-like Levels 2 & 3 and Trading_derivatives) contain a high proportion of "mid-band" questions (10–90% of models solve correctly). This indicates a superior ability to differentiate between models of varying capabilities.
• Model Profiles: No single model dominates all groups. While Claude Sonnet 4.6 leads on public benchmarks, Qwen3.5-Plus-02-15 and Kimi K2.5 demonstrate superior balance and robustness across exam-like and trading/TA groups. The benchmark also reveals finer distinctions within model families (e.g., Qwen3 scaling), where performance gaps are more pronounced on FINESSE-Bench than on public datasets.

Significance and Claims

The paper posits that FINESSE-Bench serves as a necessary complement to existing open financial benchmarks rather than a replacement. Its significance lies in its ability to expose the "transfer gap" between general financial literacy (measured by public benchmarks) and robust, professionally relevant competence.

The authors claim that FINESSE-Bench provides:

• Finer Diagnosis: It reveals strengths and weaknesses in specialized domains (derivatives, technical analysis) that are often masked in aggregated public benchmark scores.
• Scalable Evaluation: The hierarchical design allows practitioners to track model improvement not just in accuracy, but in the retention of quality as task complexity increases.
• Realistic Assessment: By incorporating exam-style structures and applied scenarios, the suite offers a more substantive evaluation of financial reasoning capabilities required for high-stakes deployment.

The authors acknowledge limitations, including the non-commercial license due to data provenance uncertainties, the potential for data contamination, and the current lack of coverage in specific subdomains like regulatory compliance and insurance analytics. They view FINESSE-Bench as an extensible first step toward a more complete standard for evaluating financial LLMs.