🤖 machine learning

MortarBench: Evaluating Mortgage Loan Origination Agents

This paper introduces MortarBench, a synthetic benchmark for evaluating mortgage loan origination agents that reveals significant performance gaps and biases in current large language models, alongside the CRIT framework which improves accuracy, risk management, and fairness.

Original authors: Matthew Toles, Yunan Lu, Manav Munjal, Bojun Liu, Yuanhao Deng, Stephanie Selig, Derek Rindner, Cheng Li, Zhou Yu

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Matthew Toles, Yunan Lu, Manav Munjal, Bojun Liu, Yuanhao Deng, Stephanie Selig, Derek Rindner, Cheng Li, Zhou Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are applying for a mortgage to buy a house. Before the bank says "yes," a human expert (called an underwriter) has to play a very strict game of "spot the difference." They look at two things: a long list of your bank transactions (what you actually spent) and a form you filled out (what you said you spent). Their job is to make sure these two stories match perfectly and follow complex government rules.

This process is tedious, expensive, and risky. If the human makes a mistake, the bank could lose millions or get fined. So, banks are starting to hire "AI assistants" (large language models) to help do this checking. But here's the problem: nobody knew if these AI assistants were actually good at the job. There was no standardized test to see if they were smart enough or if they were just guessing.

This paper introduces MortarBench, a new "driving test" specifically designed to see how well AI can handle mortgage paperwork.

1. Building the Test (The "Fake" Reality)

You can't just give an AI real people's bank statements because that's a privacy nightmare. So, the researchers built a simulation factory.

The Recipe: They took the "flavor profile" of real bank data (how often people get paid, how much they spend on rent, etc.) and cooked up thousands of fake but realistic bank statements.
The Twist: They didn't just make random data. They used a "mutation" technique. Imagine they wrote a question like, "Does this person have 3 'Buy Now, Pay Later' debts?" Then, they programmatically edited the fake bank statement to ensure the answer was exactly "3." This guarantees the test is fair and the answers are known.
The Result: A massive dataset of 188 complex scenarios where an AI has to read a bank statement and a form, then answer specific questions like "Is this a joint account?" or "List all the large deposits."

2. The Test Results: The AI Struggles

When they ran the top-tier AI models (like Gemini, Claude, and GPT) through MortarBench, the results were mixed:

The Good: The AIs were okay at simple "Yes/No" questions.
The Bad: They were terrible at listing things. If asked to list specific transactions, the AIs often hallucinated (made things up) or missed obvious ones.
The Ugly (Bias): The AIs showed a strange prejudice. If a bank transaction had a name in English, the AI rarely thought it was "foreign." But if the name was in a non-English language (like Arabic, Hindi, or Chinese), the AI assumed it was foreign 77% of the time, even when it shouldn't have. It was like the AI wearing "foreigner glasses" that only worked on non-English names.

3. The Solution: The "Confidence Filter" (CRIT)

The researchers noticed the AI was oversensitive. It was like a smoke detector that goes off every time you toast a piece of bread. It was flagging too many things as "risky" or "foreign" just to be safe.

To fix this, they created a new tool called CRIT.

How it works: Instead of just giving an answer, the AI is forced to pause and rate its own confidence on a scale of 1 to 5. "Am I 100% sure this transaction is a loan? Or am I just guessing?"
The Filter: If the AI's confidence score is too low (below a certain threshold), CRIT throws that answer away.
The Outcome: This simple "confidence filter" acted like a sieve. It caught the AI's wild guesses and filtered them out.
- Accuracy went up (from 77.1% to 80.5%).
- The "oversensitivity" dropped significantly.
- Crucially, the bias got better. The AI became less likely to wrongly flag non-English names as "foreign" just because it was guessing.

4. Why the AI Failed (The Autopsy)

The researchers looked closely at the mistakes and found four main reasons the AI got it wrong:

Misreading the Label: It saw a transaction labeled "Personal Loan" and thought it was a "Buy Now, Pay Later" debt because the words sounded similar.
Math Errors: It saw rent payments that added up to more than the total income listed on the form and assumed the math was right, even though it wasn't.
Lack of World Knowledge: It assumed all wire transfers were international, which isn't true.
Confusing the Rules: It thought a one-time payment was a recurring bill just because it might happen again in the future.

5. The Big Takeaway

The paper concludes that while AI is getting better, it's not ready to replace human mortgage underwriters on its own yet. It's too prone to making up facts and holding biases against non-English names.

However, the CRIT method shows that if we teach AI to "know when it doesn't know," we can make it much safer, more accurate, and less biased. It's a step toward a future where AI can be a helpful co-pilot for humans, rather than a reckless driver.

In short: The paper built a test track for mortgage AI, found that the cars were driving off the road and discriminating against certain license plates, and then installed a "confidence brake" that helped them stay on the road and treat everyone more fairly.

Technical Summary: MortarBench and CRIT

Problem Statement

The mortgage loan origination process is a critical, high-stakes domain involving complex risk assessment and regulatory compliance (e.g., Fannie Mae, Freddie Mac guidelines). While firms are increasingly integrating Large Language Model (LLM) agents to assist loan officers in validating applicant financial profiles against bank statements, the industry lacks a standardized benchmark to evaluate these agents. Existing financial QA benchmarks do not adequately address the specific challenges of mortgage underwriting, which require reconciling unstructured bank statements with structured applicant profiles (ULAD files) to detect inconsistencies, calculate specific financial metrics, and ensure regulatory adherence. Furthermore, the inability to share real financial data due to privacy concerns has hindered the creation of realistic, edge-case-rich evaluation datasets.

Methodology

1. MortarBench Dataset Creation

To address the lack of evaluation standards, the authors introduce MortarBench (Mortgage Transaction Analysis, and Reasoning Benchmark). The dataset generation pipeline employs a financial data synthesis and mutation approach to overcome privacy constraints while ensuring internal consistency:

Data Synthesis: The authors sampled 300 real bank statements to extract anonymous summary statistics (frequencies and average values of transaction types like payroll, housing expenses, crypto, and BNPL). These statistics were used to procedurally generate synthetic bank statements (60-day spans) matching real-world distributions.
ULAD Generation: Uniform Loan Application Dataset (ULAD) files were generated using a template-driven approach based on the Freddie Mac schema. Values in the ULAD (assets, liabilities) were computed directly from the synthetic bank statements to ensure consistency.
Question & Answer Generation: Real-world questions from an in-production loan assistant chatbot were curated and refined by experts. Questions were categorized into three answer types: Boolean, Transaction List (IDs), and Account List (IDs).
Mutation for Edge Cases: To ensure balanced coverage of edge cases, the authors designed mutation functions. For a given question and desired answer (e.g., "3 BNPL transactions"), the system programmatically edits the bank statement and ULAD to force that specific outcome while maintaining document consistency. This allows for the generation of both positive and negative examples for each query.

The final dataset comprises 188 test cases derived from 47 unique questions, with an average of 33.6 transactions per bank statement.

2. The CRIT Framework

Observing that baseline models exhibited severe oversensitivity (high false positives, zero false negatives in some cases), the authors propose CRIT (Confidence Reflection Inference for Transactions).

Mechanism: Instead of directly outputting a list of transactions, the agent generates a confidence score (1–5) for each potential transaction included in its answer. This score reflects the probability of correctness based on the number and strength of assumptions required to justify the inclusion.
Thresholding: The system applies a hard threshold ( $T=5$ ), dropping any transaction with a confidence score below this value.
Objective: This approach aims to align models away from oversensitivity, improve risk management steering, and reduce bias by allowing the system to filter out low-confidence, potentially erroneous inclusions.

Key Results

Performance Metrics

Baseline Performance: State-of-the-art closed-source models performed poorly on MortarBench, with the best baseline (Gemini 3.1 Pro) achieving only 77.1% Exact Match (EM) accuracy.
CRIT Impact: The CRIT framework improved performance across all tested models. For Gemini 3.1 Pro, CRIT increased EM accuracy to 80.5% and F1 to 83.6.
Error Reduction: CRIT provided a 14.8% error reduction over the strongest baseline. The most significant gains were observed in transaction list questions, where baseline models struggled with oversensitivity.
Bias Mitigation: CRIT reduced the "foreign origin" classification gap between English and non-English names from 77.0% to 54.0% in the Gemini baseline.

Failure Analysis

Manual review of 36 erroneous reasoning traces from the baseline Gemini model identified four primary failure modes:

Transaction Misclassification (33.3%): Incorrectly categorizing transactions based on descriptions (e.g., misidentifying a personal loan as BNPL).
Value Matching Errors (27.8%): Failing to correctly match calculated values against stated limits (e.g., rent payments exceeding ULAD totals).
Lack of Domain Knowledge (22.2%): Incorrectly assuming industry norms (e.g., all wire transfers are international).
Constraint Misinterpretation (11.1%): Misinterpreting prompt constraints (e.g., classifying one-time payments as recurring).

Stochasticity and Consistency

The authors analyzed error consistency using Risk Ratio (RR). An RR of 12.7 was found for baseline Gemini, indicating that errors are not due to random stochasticity but rather consistent false beliefs within the model. This suggests that models rely on spurious reasoning patterns rather than genuine understanding.

Bias Analysis

The study revealed systematic biases in LLMs regarding non-English names:

Foreign Origin Bias: Transfers associated with non-English names (especially non-Latin scripts like Arabic, Hindi, and Mandarin) were classified as "foreign origin" in 77.0% of cases, compared to only 13.3% for English names.
Script Sensitivity: The bias was more pronounced for non-Latin scripts (93.3%) than for non-English Latin scripts (52.5%).
IVTS Bias: Users of Informal Value Transfer Systems (IVTS) faced higher recall rates for "foreign" classification compared to formal payment systems, indicating potential unequal treatment.

Significance and Claims

The paper claims the following contributions and significance:

First Specialized Benchmark: MortarBench is the first benchmark designed specifically for mortgage underwriting tasks, filling a critical gap in the industry's ability to evaluate LLM agents.
Realistic Edge Case Coverage: By using a mutation-based pipeline, the dataset successfully generates realistic examples with broad edge case coverage that match real-world distributions, overcoming the privacy barriers of real financial data.
Demonstration of Model Weaknesses: The benchmark exposes that current frontier models struggle with oversensitivity, complex list reasoning, and exhibit significant bias against non-English names and informal payment systems.
Effective Mitigation Strategy: The CRIT framework is presented as a human-centric strategy that not only improves accuracy but also enhances steerability and risk management. By adjusting confidence thresholds, users can trade off between false positives and false negatives based on the specific cost of errors in a financial context.
Call for Domain Expertise: The analysis of error patterns suggests that improving these systems requires more than just larger models; it necessitates additional in-context domain expertise and relevant post-training to correct consistent false beliefs.

The authors conclude that while LLMs show promise for improving efficiency, the current state of technology necessitates human-in-the-loop oversight, particularly for transaction list reasoning and bias mitigation.