💬 NLP

LLM-based Triplet Extraction from Financial Reports

This paper presents a semi-automated pipeline for extracting financial triplets using LLMs and ontology-driven proxy metrics, demonstrating that automatically induced ontologies eliminate schema drift while a hybrid verification strategy significantly reduces hallucination rates and reveals systematic asymmetries in subject-object generation.

Original authors: Dante Wesslund, Ville Stenström, Pontus Linde, Alexander Holmberg

Published 2026-02-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dante Wesslund, Ville Stenström, Pontus Linde, Alexander Holmberg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a financial detective trying to solve a mystery hidden inside thousands of pages of corporate annual reports. These reports are like dense, complicated novels written in "legalese," full of passive sentences and hidden numbers. Your goal is to pull out specific facts—like "Who did what to whom?"—and organize them into a neat, structured list (a Knowledge Graph).

This paper is about building a robot detective (using Large Language Models, or LLMs) to do this job. But there's a catch: the robot is prone to "hallucinations," meaning it sometimes makes up facts that aren't in the text. In finance, making things up is dangerous.

Here is the story of how the researchers built a better system, explained simply:

1. The Problem: The "Lost in the Middle" and the "Make-Believe" Trap

Financial reports are huge. If you feed a whole report to a robot at once, it gets overwhelmed and forgets the important details in the middle (a problem called "lost in the middle"). Also, if you just ask the robot to "find the facts," it might get creative and invent relationships that don't exist because it wants to sound smart.

The Analogy: Imagine asking a student to read a 500-page textbook and write down every fact. If you give them the whole book at once, they might zone out. If you just say "write down the facts," they might start inventing facts to fill the gaps because they are nervous about leaving blanks.

2. The Solution: A "Rulebook" (Ontology)

To stop the robot from making things up, the researchers gave it a strict rulebook called an Ontology. This is like a template that says, "You can only write down facts that fit these specific categories."

They tested two ways to create this rulebook:

The Manual Rulebook (Static): A human expert sat down and wrote a generic rulebook based on one company's report, then tried to use that same rulebook for a different company.
- Result: It was like trying to fit a square peg in a round hole. The second company's report didn't quite fit the old rules, so the robot got confused or ignored the rules.
The Automatic Rulebook (Dynamic): The robot was allowed to build its own rulebook as it read each specific report. It looked at the text, said, "Oh, this report talks about 'Investing Activities,' so I'll add that to my rulebook," and then continued reading.
- Result: This was a perfect fit. The rulebook was custom-made for the specific document, so the robot never got confused or drifted off-topic. It achieved 100% adherence to the rules.

3. The "Fake Fact" Detector: Regex vs. The Judge

Even with a rulebook, the robot sometimes made mistakes. The researchers needed a way to check if the robot was telling the truth without having a "correct answer key" (since no one has manually checked every single fact in these reports).

They compared two ways to check the work:

The "Strict String Matcher" (Regex): This is like a spellchecker that only looks for exact word matches. If the text says "The Group" and the robot writes "The Company," the spellchecker screams "ERROR! You made it up!"
- Problem: This was too harsh. It flagged 65% of the answers as "fake" just because the wording was slightly different, even though the meaning was correct.
The "Hybrid Judge" (Regex + LLM-as-a-Judge): This is a two-step process. First, the spellchecker looks for exact matches. If it doesn't find one, it doesn't immediately fail the robot. Instead, it asks a second, smarter robot (the Judge): "Hey, does 'The Company' mean the same thing as 'The Group' in this sentence?"
- Result: The Judge realized, "Yes, they mean the same thing." This dropped the "fake fact" rate from 65% down to just 1.6%. It stopped the system from punishing the robot for being slightly creative with words, while still catching real lies.

4. The "Passive Voice" Trap

The researchers noticed something weird: The robot was much more likely to make up the Subject (the "Who") than the Object (the "What").

The Analogy: Financial reports love to use passive voice, like "The profit was increased by 5%." They hide the "Who" (the CEO? The board?).

When the robot sees "The profit was increased," it has to guess who did it. It often guesses wrong (hallucinates the subject).
But the "What" (5%) is right there in the text. It's easy to copy-paste.
Lesson: It's harder to be honest about who did something when the sentence doesn't say it, but easy to be honest about what happened.

5. The Takeaway

This paper teaches us three main things for building AI that handles financial data:

Don't use a one-size-fits-all rulebook. Let the AI build a custom rulebook for every document it reads. It works much better.
Don't trust a simple spellchecker. If you want to know if an AI is lying, you need a smart "Judge" to understand the meaning, not just the exact words.
Watch out for passive voice. AI struggles most when humans hide the "doer" of an action in the sentence.

By combining a custom rulebook with a smart two-step checker, the researchers created a system that is much more reliable, accurate, and trustworthy for turning messy financial reports into clean, usable data.

1. Problem Definition

The paper addresses the challenge of extracting structured knowledge (Subject-Predicate-Object triplets) from unstructured corporate financial reports to build Knowledge Graphs (KGs).

The Core Challenge: Financial reports are dense, heterogeneous, and often use passive voice, making manual extraction expensive and unscalable.
The Evaluation Gap: Standard evaluation relies on manually annotated "ground truth" datasets to calculate precision and recall. However, such datasets do not exist for dynamic financial domains, making it difficult to evaluate Large Language Model (LLM) extraction pipelines reliably.
The Hallucination Risk: LLMs are prone to hallucinations (generating facts not present in the source). In finance, where factual adherence is critical, this poses a significant risk. Existing verification methods often fail to distinguish between genuine hallucinations and surface-level variations (e.g., coreference resolution).

2. Methodology

The authors propose a semi-automated pipeline that evaluates extraction quality without ground truth by using ontology-driven proxy metrics.

A. Extraction Pipeline

Data Preprocessing: Reports are segmented into small chunks (5 sentences, zero overlap) to mitigate the "lost-in-the-middle" phenomenon in long contexts.
Prompting: LLMs are prompted to extract triplets strictly adhering to a provided ontology, outputting results in JSON.
Ontology Strategies (The Independent Variable):
1. Manual Ontology ( $O_{Manual}$ ): A static, human-engineered ontology derived from one report (Volvo) and applied to both in-domain (Volvo) and out-of-domain (Elekta) reports.
2. Automatic Ontology ( $O_{Auto}$ ): A fully automated, document-specific induction strategy. An LLM iteratively proposes new entity types and relations for each chunk, merging them into a dynamic ontology specific to that report.

B. Evaluation Metrics (No Ground Truth)

Instead of precision/recall against a gold standard, the pipeline measures:

Ontology Conformance (OC): The percentage of extracted predicates that exist in the predefined relation set $R$ .
Faithfulness (Hallucination Metrics):
- Subject Hallucination (SH): Subject entity not found in source text.
- Object Hallucination (OH): Object entity not found in source text.
- Relation Hallucination (RH): Predicate not in the ontology.

C. Hybrid Verification Strategy

To accurately measure faithfulness without ground truth, the authors propose a two-step verification:

Baseline: Strict Regular Expression (Regex) matching for exact string matches.
Hybrid: If Regex fails, an "LLM-as-a-judge" checks if the entity is semantically present in the text (handling coreference and morphological variations).

3. Key Contributions

C1 (Ontology Comparison): A comparative analysis showing that document-specific automatic ontology induction outperforms static manual ontologies, eliminating "ontology drift" (schema violations).
C2 (Model Architecture Impact): Identification of how different LLM architectures respond to schema constraints. Some models strictly adhere to constraints, while others prioritize semantic fluency over rigid schema adherence, leading to hallucinations.
C3 (Verification Innovation): Validation of a hybrid verification strategy (Regex + LLM-as-a-judge) that drastically reduces false-positive hallucination rates caused by coreference resolution, enabling ground-truth-independent evaluation.

4. Experimental Results

The study evaluated two models (Gemini-2.5 Flash and Llama 4 Maverick) on two reports (Volvo 2024 and Elekta 2022/23).

Verification Impact (RQ3):
- The strict Regex baseline estimated a 65.2% Subject Hallucination (SH) rate.
- The Hybrid method reduced this to 1.6%, proving that most "hallucinations" were actually correct extractions with different surface forms (e.g., pronouns vs. full names).
Ontology Strategy (RQ1):
- Automatic Ontology ( $O_{Auto}$ ): Achieved 100% Ontology Conformance (OC) across all models and reports. It eliminated schema violations entirely.
- Manual Ontology ( $O_{Manual}$ ): Caused significant drift for the Llama model (dropping to ~89% OC) and higher hallucination rates, especially in out-of-domain scenarios.
Model Performance (RQ2):
- Gemini-2.5 Flash: Highly conservative; maintained 100% OC and 0% Relation Hallucination (RH) regardless of the ontology strategy.
- Llama 4 Maverick: More prone to "ontology drift." Under the manual ontology, it generated invalid relations (up to 10.6% RH). However, when paired with the automatic ontology, its performance improved to match Gemini's strict adherence.
Hallucination Asymmetry:
- A systematic asymmetry was found where Subject Hallucinations remained higher than Object Hallucinations even after correction.
- Cause: Financial prose frequently uses passive voice and omits agents, forcing the model to infer subjects (a high-risk operation), whereas objects are often concrete numbers or dates that can be copied verbatim.

5. Significance and Conclusion

Practical Utility: The paper provides a viable framework for extracting financial knowledge in domains where ground truth is unavailable. The hybrid verification strategy is critical for avoiding the overestimation of errors in automated pipelines.
Ontology Design: The study demonstrates that dynamic, document-specific ontology induction is superior to static, human-engineered schemas for LLM-based extraction. It reduces cognitive load on the model and prevents schema drift.
Model Selection: The choice of LLM architecture is critical. Models that prioritize strict constraint satisfaction (like Gemini) perform better with static schemas, while more creative models (like Llama) require dynamic, tailored schemas to prevent hallucinations.
Future Directions: The authors suggest combining stable core schemas with document-specific extensions and expanding evaluation to diverse document types and languages.

In summary, this work establishes that by combining automatic ontology induction with hybrid semantic verification, it is possible to build robust, high-fidelity Knowledge Graphs from financial reports without the prohibitive cost of manual annotation.