Technical Summary: APEX–Accounting

Problem Statement

While large language models (LLMs) have demonstrated proficiency in passing professional certification exams (e.g., CPA, CMA) and improving general work quality, there is a lack of benchmarks assessing their ability to perform the actual, day-to-day work of accountants. Existing benchmarks like AuditBench, FinMaster, and AccountingBench often rely on synthetic transactions, text-only simulators, or single-company case studies that fail to capture the repetitive, procedural, and document-heavy nature of month-end close and bookkeeping. The authors argue that current evaluations do not sufficiently test the "real work" of accountants, which involves reconciling accounts, accruing expenses, posting transactions, and producing reports across diverse, complex business contexts.

Methodology

Benchmark Design (APEX–Accounting)

APEX–Accounting is a closed benchmark developed by Mercor in partnership with Ramp, comprising 160 tasks across 10 synthetically generated company worlds.

• World Construction: Each world represents a self-contained company frozen at month-end close, adhering to U.S. GAAP accrual-basis accounting. Worlds were constructed in four stages: scoping, detailed specification (including "trap registers" for seeded contradictions), style guide derivation, and file generation. All files (spreadsheets, PDFs, accounting software exports) are novel and screened against public sources to prevent memorization.
• Task Categories: The 160 tasks are divided into four categories:
  1. Reconciliation (61 tasks): Tying two sources together, identifying breaks, and explaining/correcting them.
  2. Data Entry (26 tasks): Posting transactions, journal entries, and invoices.
  3. Variance Analysis (28 tasks): Comparing actuals against budgets or expectations.
  4. Schedules & Accruals (45 tasks): Building schedules, calculating accruals, and carrying balances.
• Expert Involvement: 42 accounting experts (median 11 years experience, 52% from Big Four firms) authored the tasks, solved them to create "golden responses," and wrote binary, outcome-based grading rubrics. Each task includes a prompt, required input files, a golden response, and a rubric with an average of 13.7 criteria.

Experimental Setup

• Models Evaluated: Nine frontier models were tested, including Claude-Fable-5, Muse-Spark-1.1, GPT-5.6-Sol, and others. Each model executed every task 8 times, generating 11,520 trajectories.
• Harnesses: Two agent harnesses were used:
  • Loop Harness: A standard canonical while-loop-with-tools architecture.
  • Ramp Harness: A specialized harness built with Ramp featuring retry-awareness, tool allowlists, validation, and subagent delegation to mirror real-world accounting constraints.
• Grading: A DeepSeek-v4-Flash model, optimized with a GEPA prompt, served as the judge. It was validated against human expert ground truth (1,687 criteria), achieving 97.1% accuracy (F1 = 0.970). The judge evaluates the final output against binary rubric criteria without access to the intermediate trajectory log.
• Metrics:
  • Mean Criteria@3: The primary metric, averaging the percentage of rubric criteria met across 3 randomly selected runs per task.
  • Pass@k / Pass^k: Measures the capability ceiling (Pass@8: passing at least once in 8 attempts) and consistency (Pass^8: passing all 8 attempts).
  • Cost Ablation: Experiments varying the token budget from $1 to $50 per task to analyze the relationship between spending and performance.

Key Results

Leaderboard Performance

• Top Performer: Claude-Fable-5 (Max) achieved the highest Mean Criteria@3 of 56.4%, followed by Muse-Spark-1.1 (52.6%) and GPT-5.6-Sol (51.5%).
• Consistency Gap: Despite high Mean Criteria@3 scores, consistency remains extremely low. No model achieved a Pass^8 score higher than 2.6% (GPT-5.6-Sol). The highest Pass@8 was 21.5% (Muse-Spark-1.1), indicating that while models can occasionally solve a task, they cannot reliably do so end-to-end.
• Category Difficulty: "Schedules & Accruals" proved the most difficult category, with all models scoring 7–21 percentage points lower than in other categories, reflecting its multi-step, judgment-heavy nature.

Cost and Budget Analysis

• Token Budget Impact: Increasing the budget from $1 to $50 significantly improved scores for expensive models (e.g., Claude-Fable-5 gained +43.4 percentage points), while cheaper models (e.g., Muse-Spark-1.1) saw minimal gains.
• Simpson's Paradox: The study observed an instance of Simpson's Paradox: while increasing the total budget raised scores, within a fixed budget, tasks where models spent more tokens correlated with lower scores. This is attributed to task difficulty; harder tasks consume more tokens but remain harder to solve.
• Harness Impact: Switching from the standard Loop Harness to the specialized Ramp Harness resulted in a negligible average shift (+1.2 percentage points), suggesting that model capability is a stronger driver of performance than the specific agent architecture for these tasks.

Failure Analysis

Analysis of low-scoring trajectories from the top three models revealed a strikingly similar failure profile:

• Reasoning Dominance: Reasoning failures accounted for 59–79% of all annotated errors.
• Specific Failure Modes: The most common sub-failures were non-numeric reasoning (applying faulty logic to non-numeric data) and data handling errors (incorrect filtering, joining, or aggregating).
• Information vs. Logic: Models reliably found the correct input files (information gathering failures were rare). The primary failure mode was mishandling multi-step reasoning over those inputs: substituting wrong authorization logic, dropping correct intermediate results, or failing to carry conclusions through to the final answer.
• Tool Use: No annotated failures involved tool use errors, suggesting that current agent architectures are sufficient for tool interaction, but the underlying reasoning capabilities are the bottleneck.

Significance and Claims

The paper claims that APEX–Accounting provides the first rigorous assessment of frontier models on real-world accounting workflows, moving beyond certification exams to practical application.

• Current Limitations: The results indicate that while frontier models can retrieve information and perform isolated steps, they are not yet capable of reliably closing the books unsupervised. The low Pass^8 scores (max 2.6%) highlight a significant gap between "capability ceiling" and "reliable deployment."
• Future Direction: The authors argue that progress will likely come less from improving agent harnesses (which showed minimal impact) and more from improving the models themselves. Specifically, they suggest the need for accounting-specific training to instill the discipline required to carry multi-step results through, surface contradictions in documents, and refuse to post entries without sufficient evidence.
• Benchmark Utility: As a closed benchmark, APEX–Accounting allows for the evaluation of any frontier model upon request, offering a standardized metric for the industry to track progress in automating skilled knowledge work.