🤖 AI

FinProBench: Evaluating Financial AI Agents with Role-Grounded Rubrics Derived from Professional Deliverables

This paper introduces FinProBench, a benchmark for evaluating financial AI agents, and Role-Grounded Rubric Construction (RGRC), a pipeline that derives evaluation criteria from professional deliverables to effectively capture tacit standards and significantly outperform prompt-based methods in role-specialized financial tasks.

Original authors: Ben Wang, Kang Zhou, Lifan Guo, Feng Chen, Chi Zhang

Published 2026-08-06

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Ben Wang, Kang Zhou, Lifan Guo, Feng Chen, Chi Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to be a professional, like a doctor, a lawyer, or a financial analyst. In the world of artificial intelligence, we have these super-smart "agents" that can chat, write code, and solve puzzles. But how do we know if they are actually doing a good job, or just sounding like they are? This is the big question in the field of AI evaluation. For a long time, we've tested AI by asking it simple questions with right-or-wrong answers, like a multiple-choice quiz. But real professional work isn't a quiz; it's a messy, complex project, like writing a 50-page report or building a financial model. To judge this kind of work, we need a "rubric"—a detailed scoring sheet that tells us exactly what makes a piece of work excellent versus just okay. The problem is, most of these scoring sheets are made by guessing what the AI should do, or by looking at the AI's own answers, which is like asking a student to grade their own homework. We need a way to build these scoring sheets based on what real humans actually do in their jobs.

This paper introduces a new way to test financial AI agents called FinProBench, along with a clever method to build the scoring sheets, known as Role-Grounded Rubric Construction (RGRC). Think of RGRC as a detective that doesn't look at the suspects (the AI) to figure out the rules, but instead studies the crime scene (real, professional work documents) to see how the experts actually did it. The researchers gathered over 1,700 real documents—like investment reports and compliance checks—written by actual financial professionals. They used these real-world examples to teach the AI how to grade other AI work.

Here is the twist they discovered: It depends on the job. For common, everyday financial tasks that are already well-known (like writing a standard market report), simply asking the AI to "do a good job" (using a prompt) works almost as well as their fancy new method. However, for specialized, tricky roles where the rules are hidden or very specific (like a niche insurance actuary), the simple "ask nicely" approach fails miserably. In those cases, the RGRC method, which learns from real human deliverables, is a game-changer, catching 99.1% of the necessary standards compared to only 78% for the simple method.

When they put the AI agents to the test against real human experts using these new, high-quality scoring sheets, the humans still won on average, scoring 73.7 out of 100, while the best AI agents hovered around 70. But it wasn't a total blowout. The AI systems had their own superpowers: some were better at digging deep into the math, while others were great at formatting. The humans, however, were the most consistent all-rounders, nailing the structure, the data accuracy, and the compliance rules. The paper suggests that while AI is getting very good, it still needs to learn from the "tacit" (unspoken) secrets of real professionals to truly master complex jobs. The best part? Because they built a "master scoring sheet" for each job role, they could reuse it for many different tasks, saving a massive amount of time—about 6.7 times faster than making a new grading sheet from scratch for every single assignment.

Technical Summary: FinProBench and Role-Grounded Rubric Construction (RGRC)

Problem Statement

The rapid deployment of Large Language Models (LLMs) as AI agents for complex professional tasks has outpaced the development of evaluation frameworks capable of assessing the quality of real-world professional deliverables. Existing financial benchmarks (e.g., FinEval, CFinBench, FinBen) predominantly focus on factual recall, sentiment analysis, or extractive question-answering. They fail to evaluate the production of complete, multi-page professional artifacts such as valuation models, compliance reports, and research papers.

A critical bottleneck in evaluating open-ended outputs is the construction of high-quality rubrics. Current methods—direct generation, contrastive extraction, and iterative refinement—derive criteria from task prompts or model outputs. This approach misses tacit professional standards that are embodied in actual work products but rarely articulated in task descriptions. Furthermore, existing rubric construction lacks a mechanism for role-level reuse, requiring laborious, task-specific authoring that does not scale across occupations or jurisdictions.

Methodology: Role-Grounded Rubric Construction (RGRC)

The paper introduces Role-Grounded Rubric Construction (RGRC), a pipeline designed to distill evaluation criteria directly from authentic professional deliverables written by practitioners in the same occupational role. The pipeline operates in four stages:

1. Deliverable Collection

The foundation is a curated corpus of authentic documents.

Authenticity Criterion: A document is included only if the author's role matches the target occupational role (e.g., excluding academic papers, regulatory filings, or annual reports authored by IR departments).
Source Prioritization: Documents are tiered by authenticity confidence: Tier 1 (Direct institutional authorship), Tier 2 (Proximate roles with overlap), and Tier 3 (Inferred from content).
Corpus Requirements: For each role, a minimum of 20 deliverables from at least 3 distinct institutions spanning ≥2 years are required to capture evolving standards.

2. Competency Extraction

Latent quality dimensions are extracted from the corpus using a two-phase LLM analysis:

Per-Document Analysis: Identifies structural patterns, analytical depth markers, professional conventions, and quality differentiators.
Cross-Document Synthesis: Aggregates findings to identify role-level standards that appear consistently (≥60% frequency) across multiple authors and institutions. This distinguishes universal competencies (e.g., numerical accuracy) from role-specific ones (e.g., DCF model coherence) and implicit standards (e.g., mandatory risk disclosure sections).

3. Rubric Synthesis

Extracted competencies are transformed into scored criteria following four design principles:

Evaluability: Criteria must be independently judgeable with determinate scoring rules, avoiding anchorless subjective descriptions.
Unambiguity: Criteria use concrete observable indicators and quantitative thresholds to ensure high inter-rater agreement (probability ≥0.9).
Non-Hackability: Criteria require multi-step reasoning chains where intermediate results constrain downstream answers to prevent "rubric gaming."
Orthogonality: Criteria are verified to be minimally correlated to prevent mechanical score inflation.
Scoring Schema: Criteria are assigned weights (1–5), polarity (positive/negative), and category tags. A complexity-adaptive mechanism adjusts the number of criteria (38–49) based on task difficulty.

4. Validation

Generated rubrics undergo a three-gate validation process:

Gate 1 (Automated QA): Checks for prompt-rubric coverage, deliverable alignment, principle compliance, and weight distribution.
Gate 2 (Discriminative Review): Verifies the rubric can distinguish authentic references from degraded or model-generated alternatives without awarding points for superficial compliance.
Gate 3 (Cross-Evaluator Consistency): Requires Cohen's $\kappa \ge 0.75$ between independent evaluators (human or LLM) to ensure formulation clarity.

Role-to-Task Reuse: A key architectural feature is the separation of Role-Level Rubrics (normative standards shared across all tasks of an occupation) and Task-Level Rubrics (thin, specific layers). This allows 60–70% of criteria to be inherited, reducing per-task authoring effort by an estimated 6.7×.

The FinProBench Benchmark

RGRC is instantiated as FinProBench, a benchmark built on a corpus of 1,723 curated real-world deliverables spanning:

57 Financial Occupations: Classified into 30 "prior-rich conventional roles" (e.g., equity analysts) and 27 "prior-sparse role-specialized roles" (e.g., specialized compliance officers).
8 Sub-industries: Banking, securities, insurance, asset management, wealth management, fintech, compliance, and data engineering.
161 Deliverable Types.

The initial evaluation set comprises 20 complete tasks across 20 roles and 7 sub-industries, each consisting of a task prompt, a human gold deliverable, and an RGRC-derived rubric.

Experimental Results

The evaluation utilized a panel of four heterogeneous LLM judges (GPT, DeepSeek, Qwen, Gemini) and compared human deliverables against three AI agents (Codex, Qoder, Claude Code).

1. Professional-Standard Coverage

The study compared RGRC against a "Prompt-only" baseline (using general model priors to generate rubrics).

Conventional Roles (30): Prompt-only achieved 89.2% coverage, nearly matching RGRC's 90.7%. This suggests general models already encode conventions for common roles.
Role-Specialized Roles (27): RGRC significantly outperformed Prompt-only (99.1% vs. 78.0%). This indicates that for specialized roles, tacit standards are not encoded in general priors and require grounding in authentic deliverables.

2. Judge Reliability

The rubrics demonstrated high consistency, with a Fleiss' $\kappa$ of 0.76 across the judge panel, meeting the $\ge 0.75$ threshold for reliable automated scoring.

3. Human vs. Agent Performance

Under the gold-held-out role-level rubric:

Human Deliverables ranked first on average (73.7/100).
AI Agents followed with overlapping 95% confidence intervals: Codex (70.3), Qoder (70.2), and Claude Code (69.6).
Dimensional Analysis: Humans led in structural completeness, data accuracy, compliance, and presentation. Agents showed complementary strengths; for instance, Qoder led in analytical depth and risk disclosure, while Claude Code showed strong analytical depth but higher variance. No single system dominated all dimensions.

Significance and Claims

The paper claims that FinProBench and RGRC address three critical gaps in financial AI evaluation:

Shift from Answers to Deliverables: Moving beyond binary correctness to evaluate complete, structured professional work products.
Evidence-Grounded Rubrics: Establishing a paradigm where evaluation criteria are derived from authentic human work rather than prompts or model outputs, successfully recovering tacit professional standards.
Scalability through Reuse: Demonstrating that role-level grounding allows for the reuse of normative standards across tasks, drastically reducing the effort required to construct evaluation suites.

The authors emphasize that while prompt engineering can approximate rubrics for conventional roles where standards are already encoded in model priors, authentic professional grounding is critical for recovering the tacit, role-specific standards of specialized occupations. The framework is presented as jurisdiction-agnostic, relying on the availability of a role-aligned evidence corpus rather than a specific construction methodology.