💬 NLP

Beyond Semantic Similarity: A Two-Phase Non-Parametric Retrieval Workflow for Corporate Credit Underwriting

This paper presents a two-phase non-parametric retrieval system that addresses the "similarity-utility gap" in corporate credit underwriting by prioritizing analytical usefulness over semantic similarity, successfully reducing document review time from hours to minutes for over 800 analysts.

Original authors: Linus Ng Junjia, Ezekiel Tee Kongquan, Kelvin Heng, Kenneth Zhu Ke, Zhao Jing Yuan

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Linus Ng Junjia, Ezekiel Tee Kongquan, Kelvin Heng, Kenneth Zhu Ke, Zhao Jing Yuan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a financial detective trying to solve a mystery about a company's health. You have a massive library of thousands of documents—annual reports, industry analyses, and legal disclosures—some hundreds of pages long. Your job is to find the specific clues (like a hidden debt or a risky market trend) to decide if you should lend money to that company.

This paper describes a new, smarter way to search that library. Here is the breakdown in everyday language:

The Problem: The "Keyword Trap"

Traditionally, computer search systems work like a librarian who only cares about word matching. If you ask, "How much debt does Company X have?", the computer finds every page that mentions "debt" and "Company X."

But in finance, this is a trap. A page might say, "We have no debt," or "Debt is a common term in our industry," or it might be a boring legal disclaimer that repeats the word "debt" fifty times. These pages match your words perfectly, but they are useless for making a decision. The authors call this the "similarity–utility gap." The computer finds things that look like the answer, but aren't actually useful answers.

The Solution: A Two-Phase Detective Team

The authors built a system that acts less like a keyword matcher and more like a team of human detectives. They split the work into two distinct phases:

Phase 1: The Wide Net (High Recall)

First, the system casts a very wide net. It uses two methods at once:

Keyword Search: Looking for exact financial terms (like "EBITDA" or "liquidity").
Concept Search: Looking for the idea behind the words, even if the specific words are different.

This phase grabs a huge pile of potential pages (50 at a time) to make sure they don't miss anything important. Think of this as a fisherman casting a massive net to catch everything in the ocean.

Phase 2: The Expert Filter (High Utility)

This is where the magic happens. The system doesn't just stop at the pile of fish; it brings in an AI Expert Judge.

The Filter: Before the judge even looks, a lightweight AI checks: "Does this page actually talk about the specific question? Is it just legal noise?" If it's just noise, it gets thrown out immediately.
The Judge: The remaining pages are passed to a smarter AI that asks a different question: "How useful is this for making a lending decision?"
- Instead of asking, "Does this page sound like my question?"
- It asks, "Does this page give me a fact I can use to say 'Yes' or 'No' to the loan?"

This ensures the final list of results contains only the "gold nuggets" of information, not the "dirt" that just happened to have the same words.

Handling the Messy Details

Financial documents are messy. They have long paragraphs, footnotes, and complex tables with merged cells and weird headers.

The "Context-Aware" Tool: If the system finds a simple table, it neatly extracts the numbers. But if it finds a complex, confusing table (like a multi-level financial statement), it doesn't try to guess the numbers. Instead, it grabs the entire table and tags it with a "receipt" (telling you exactly which page and document it came from). This allows a human to verify the numbers later without the computer accidentally breaking the data.

The "On-Premise" Rule

Because this deals with secret bank data, the system doesn't use public cloud servers or outside AI tools. It runs entirely inside the bank's own secure building (on-premise). It's like having a private, secure library where no data ever leaves the building, ensuring total privacy and compliance with strict banking laws.

The Results

The team tested this system with over 800 real financial analysts.

Before: Analysts spent hours digging through documents to find the right facts.
After: The system cut that time down to about three minutes.

Summary

In short, this paper presents a system that stops trying to be a "word-matching machine" and starts acting like a "decision-making assistant." It casts a wide net, filters out the boring legal fluff, and ranks the remaining information based on how helpful it is for making a real-world business decision, all while keeping the data safe and secure.

Technical Summary: Beyond Semantic Similarity in Corporate Credit Underwriting

Problem Statement
Corporate credit underwriting relies on the analysis of long-form, heterogeneous financial documents, including annual reports and industry analyses. Analysts must extract specific financial indicators, assess risk disclosures, and synthesize insights from documents spanning hundreds of pages and multiple languages. While Retrieval-Augmented Generation (RAG) systems have emerged to assist such workflows, standard pipelines prioritize semantic similarity between queries and document passages. In financial contexts, this approach often fails to align with analyst needs; documents frequently contain narrative descriptions, regulatory disclosures, and repetitive boilerplate language. Consequently, similarity-based retrieval may surface passages that share terminology with a query but lack actionable analytical value. The authors identify this discrepancy as the "similarity–utility gap," where topically related content does not necessarily support decision-making.

Methodology
To bridge this gap, the authors propose a two-phase, non-parametric retrieval workflow designed specifically for enterprise decision-support. The system operates entirely on-premise using self-hosted open-source models to ensure compliance with strict data governance policies. The architecture consists of five core components:

Document Ingestion: Financial documents are segmented into structured sections using layout cues. Each segment is indexed with metadata, including source, section title, and page references, to preserve structural information for downstream processing.
Phase 1: Hybrid Candidate Retrieval: This phase aims for high recall across multilingual documents. It employs a hybrid strategy combining keyword retrieval (to preserve precision for financial terminology) and dense semantic retrieval using multilingual embeddings. A TopK value of 50 is used to generate an initial high-recall candidate pool ( $C_0$ ).
Phase 2: Adaptive Retrieval Controller & Utility-Grounded Re-ranking:
- Adaptive Controller: A lightweight language model evaluates candidates based on query intent and document structure. It predicts three metrics for each passage: relevance ( $Rel_i$ ), evidence support ( $S_i$ ), and a utility score ( $U_i$ ). The controller filters the candidate set ( $C_1$ ) by applying a logical condition where evidence support is only considered if relevance is established, effectively pruning semantically related but analytically irrelevant boilerplate.
- Utility-Grounded Re-ranking: The remaining candidates are ranked using an "LLM-as-a-Judge" framework. Instead of measuring similarity to the query, the model evaluates passages based on their decision utility for underwriting. A tunable threshold ( $U_{threshold}$ ) balances precision and recall, filtering for passages containing actionable financial evidence.
Context-Aware Evidence Extraction: To prevent the distortion of meaning or loss of attribution, the system employs distinct extraction strategies based on content structure:
- Narrative Sections: Uses markdown-aware segmentation to preserve headers, bullet lists, and paragraph boundaries.
- Tabular Data: Distinguishes between non-complex tables (parsed to extract specific rows/cells) and complex tables (multi-level headers, merged cells). For complex tables, the system preserves the full table context along with source metadata (document name, page reference) to support manual verification and accurate attribution.

Key Contributions
The paper outlines four primary contributions:

A utility-grounded retrieval framework that aligns passage ranking with decision usefulness in credit underwriting, moving beyond mere semantic similarity.
An adaptive retrieval controller that filters candidate passages using query intent and document structure signals.
A context-aware extraction module that preserves structural information in both narrative and tabular financial content.
An enterprise deployment demonstrating large productivity gains, operating entirely within a regulated, on-premise environment.

Results
The system was evaluated on a multilingual corpus of internal corporate credit documents, with relevance labels curated by senior credit analysts. Due to the proprietary nature of the data, detailed datasets and case studies were not publicly released. However, in a real-world deployment involving over 800 analysts, the system demonstrated significant efficiency gains, reducing document review time from several hours to approximately three minutes. The evaluation indicates that the proposed approach significantly improves retrieval performance compared to traditional naive retrieval systems by surfacing passages containing verifiable financial evidence.

Significance and Claims
The authors claim that the system's significance lies in its ability to optimize for analytical utility rather than just semantic similarity, directly addressing the needs of enterprise workflows in regulated financial environments. By combining hybrid retrieval, adaptive candidate control, and utility-based ranking, the system prioritizes actionable financial evidence. The on-premise deployment ensures that sensitive financial data remains local, satisfying strict governance and auditability requirements.

Future Work
The paper notes that future work will explore enhanced structured data extraction from financial tables. Specifically, the authors suggest integrating specialized Optical Character Recognition (OCR) and Vision Language Model (VLM) pipelines to extract structured numerical information directly from table regions. This would allow the system to capture additional financial metrics embedded in complex layouts and incorporate them into downstream reasoning processes.