💬 NLP

FinTagging: Benchmarking LLMs for Extracting and Structuring Financial Information

This paper introduces FinTagging, the first comprehensive benchmark that decomposes the complex task of XBRL tagging into Financial Numeric Identification and Financial Concept Linking to reveal that while Large Language Models excel at extracting financial entities, they significantly struggle with mapping them to the full US GAAP taxonomy under realistic, structure-aware conditions.

Original authors: Yan Wang, Lingfei Qian, Xueqing Peng, Yang Ren, Keyi Wang, Yi Han, Dongji Feng, Fengran Mo, Shengyuan Lin, Qinchuan Zhang, Kaiwen He, Chenri Luo, Jianxing Chen, Junwei Wu, Chen Xu, Ziyang Xu, Jimin Hu

Published 2026-02-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yan Wang, Lingfei Qian, Xueqing Peng, Yang Ren, Keyi Wang, Yi Han, Dongji Feng, Fengran Mo, Shengyuan Lin, Qinchuan Zhang, Kaiwen He, Chenri Luo, Jianxing Chen, Junwei Wu, Chen Xu, Ziyang Xu, Jimin Huang, Guojun Xiong, Xiao-Yang Liu, Qianqian Xie, Jian-Yun Nie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The Financial "Librarian" Nightmare

Imagine you are a librarian in charge of a massive library with 2 million books (financial reports from companies). Every day, these books are filled with thousands of numbers: profits, debts, salaries, and taxes.

To make these numbers useful for computers, regulators, and investors, every single number needs a specific label (a "tag") from a giant dictionary called US-GAAP. This dictionary has 17,688 different labels.

The Old Way: It's like asking a human to read a book and manually stick a sticky note on every number saying, "This is Revenue" or "This is Debt." It's slow, boring, and humans make mistakes.
The AI Hope: We wanted to use Large Language Models (LLMs)—the super-smart AI brains behind tools like ChatGPT—to do this tagging automatically.

The Catch: Previous tests for these AIs were like giving them a pop quiz with only 5 easy questions. In the real world, the AI has to answer 17,000 questions at once, often looking at messy spreadsheets and confusing text. The old tests didn't show us if the AI was actually smart enough for the real job.

The Solution: FinTagging (The "Two-Step Dance")

The authors of this paper created a new, tougher test called FinTagging. Instead of asking the AI to guess the label in one giant leap, they broke the job down into a two-step dance:

Step 1: The "Fact Finder" (FinNI)

The Analogy: Imagine a detective scanning a crime scene. Their only job is to find the clues (the numbers) and figure out what kind of clue they are.
What it does: The AI reads the document (text or tables) and says, "I found a number: $50 million. It's a money amount, not a percentage or a count of people."
The Result: The AI is actually pretty good at this. It's like a detective who is great at spotting footprints.

Step 2: The "Label Matchmaker" (FinCL)

The Analogy: Now the detective has the clue, but they need to file it in the right drawer in a cabinet with 17,000 drawers. This is the hard part. Is "$50 million" "Cash on Hand"? "Restricted Cash"? "Investment Income"?
What it does: The AI looks at the context (the sentences around the number) and tries to match it to the perfect label in the giant dictionary.
The Result: This is where the AI struggles. Even the smartest models get confused. They might pick a drawer that is close but not quite right. It's like knowing the word "Apple" but not knowing if the person meant the fruit or the tech company without more context.

What Did They Discover?

The researchers tested 13 different AI models (including big names like GPT-4o, Llama, and DeepSeek) on this new, realistic test. Here is what they found:

The "Extraction Gap": The AI is a great detective (Step 1). It can find numbers in messy tables and text almost perfectly.
The "Alignment Gap": The AI is a confused librarian (Step 2). When it has to choose the exact right label from 17,000 options, it often picks the wrong one. It understands the number, but it doesn't fully understand the specific accounting rules behind it.
Old Tests Were Cheating: Previous tests only asked the AI to choose from a tiny list of 100 labels. It's like testing a pilot by only asking them to land on a runway with no wind. FinTagging tests them in a hurricane. When you test them on the full list of 17,000 labels, their performance drops to near zero if you don't use the two-step method.

Why Does This Matter?

Think of financial reporting as the language of money. If the labels are wrong, the "translation" is wrong.

If an AI mislabels a debt as "profit," investors might buy stock in a company that is actually in trouble.
If regulators can't trust the tags, they can't enforce the rules.

The Takeaway:
This paper says, "Don't just trust the AI to do the whole job in one go." We need to build systems where the AI first finds the numbers (which it's good at) and then uses a smart, step-by-step process to match them to the rules (which it needs help with).

They also released the dataset and code (like a recipe book) so other researchers can try to fix this "Label Matchmaker" problem and build better financial AI for everyone.

Summary in One Sentence

FinTagging is a new, realistic test that shows AI is great at finding numbers in financial reports but still needs a lot of help to correctly label them according to complex accounting rules, proving that we need a smarter, two-step approach to automate financial reporting.

1. Problem Statement

The core problem addressed is the costly and error-prone nature of XBRL (eXtensible Business Reporting Language) tagging in financial reporting.

Context: Over 2 million companies annually publish financial reports that must be tagged with standardized US-GAAP concepts (over 17,000 unique concepts) to ensure machine readability for regulators and markets.
Limitations of Existing Approaches:
- Oversimplification: Current benchmarks (e.g., FiNER, FNXL) treat tagging as a flat, single-step classification task over truncated subsets of concepts (e.g., top 139 or 2,800), ignoring the full 17k+ taxonomy.
- Structural Blindness: Existing datasets often strip away the tabular structure of financial documents, despite tables being the primary carrier of numerical data.
- Evaluation Gap: Prior work relies on "closed-set" extreme classification, which fails to evaluate Large Language Models (LLMs) under realistic conditions where the full taxonomy cannot be represented in-context. This leads to hallucinated tags or near-zero performance in full-scope settings.

2. Methodology: The FinTagging Framework

The authors introduce FinTagging, the first comprehensive benchmark designed to evaluate LLMs on structure-aware, full-scope XBRL tagging. They reframe the task from a single-step classification into a two-stage "Extract-and-Align" paradigm:

A. Task Formulation

The task is decomposed into two subtasks:

Financial Numeric Identification (FinNI):
- Goal: Extract numerical entities (Facts) from heterogeneous contexts (narrative text and structured tables) and classify their coarse-grained data types (e.g., monetaryItemType, percentItemType).
- Input: Text or serialized table data.
- Output: A list of {Fact, Type} pairs.
Financial Concept Linking (FinCL):
- Goal: Map each extracted entity to the precise, most specific concept in the full US-GAAP taxonomy (17,688 concepts).
- Mechanism: A Retrieval-Rerank approach.
  - Retrieval: Uses semantic similarity (text-embedding-3-small) to retrieve the top- $k$ candidate concepts from the full taxonomy.
  - Reranking: The LLM analyzes the entity, its type, and the structural context to select the single best tag from the retrieved candidates.

B. Dataset Construction

Source: 142 SEC 10-K filings (2023–2024) covering 11 industry sectors.
Annotation Pipeline:
- Automated Extraction: A rule-based parser reconstructs links between human-readable disclosures and Inline XBRL (iXBRL) metadata to extract 261,457 numerical entities.
- Quality Control: A strategic sampling method (greedy minimal-cover) selected a diverse audit set of ~2,000 instances (sentences and tables) covering all observed concepts.
- Human Verification: Two junior analysts and one senior adjudicator verified the data, achieving a Cohen's $\kappa$ of 0.81 (substantial agreement).
Datasets Released:
- FinNI-eval: 28,787 instances for entity extraction and type classification.
- FinCL-eval: 261,457 query-answer pairs for concept linking against the full 17k+ taxonomy.

3. Key Contributions

New Formulation: Shifts XBRL tagging from "Extreme Classification" to a two-stage "Extract-and-Align" process, enabling evaluation against the full US-GAAP taxonomy (17,688 concepts) rather than truncated subsets.
High-Fidelity Resources: Releases FinNI-eval and FinCL-eval, the first datasets in the financial domain that explicitly incorporate complex tabular structures and require joint reasoning over text and tables.
Diagnostic Insights: Identifies a critical "Knowledge-Alignment Gap". While LLMs excel at extracting numbers, they struggle significantly with fine-grained semantic alignment to specific accounting concepts without a structured framework.

4. Experimental Results

The authors evaluated 13 state-of-the-art LLMs (including GPT-4o, DeepSeek-V3, Llama-3/4, Qwen3, and domain-specific Fino1) in zero-shot settings.

Overall Performance:
- Extraction (FinNI): Large LLMs perform reasonably well. DeepSeek-V3 achieved the best F1 (0.6932), driven by high recall. Models generally generalize well to long-tail entities.
- Alignment (FinCL): Performance drops drastically. Even the best models (DeepSeek-V3, GPT-4o) achieved accuracy below 0.19 in the retrieval-rerank setting.
- Comparison: Traditional PLMs (BERT, FinBERT) fine-tuned on truncated taxonomies outperformed LLMs in micro-F1 on frequent tags but failed to generalize to the full taxonomy, confirming the limitations of closed-set approaches.
Ablation Studies:
- Single-Step Failure: When forced to perform tagging as a single-step extreme classification (selecting from 17k concepts without retrieval), all models collapsed to 0% performance.
- Error Propagation: The full pipeline (Extraction + Retrieval + Rerank) suffers from error cascading. Providing "Gold" entities improved accuracy significantly, proving that entity extraction is a primary bottleneck. However, even with perfect entities, reranking accuracy remained low (<0.21), indicating that fine-grained semantic disambiguation is the intrinsic challenge.
Structure Awareness:
- Retrieval accuracy for table-based entities was significantly lower than for text-based entities.
- Using Structure-Aware Context (SAC) (linearizing table rows) significantly outperformed Fixed-Window Context (FWC), highlighting the necessity of preserving structural information.

5. Significance and Conclusion

Benchmarking Standard: FinTagging establishes a rigorous standard for evaluating LLMs in financial reporting, moving beyond simple QA or extraction to complex, structured regulatory compliance tasks.
Practical Implications: The results suggest that while LLMs are powerful tools for identifying financial facts, they cannot yet replace human expertise in mapping these facts to precise regulatory taxonomies without significant architectural support (e.g., retrieval-augmented generation and structured prompting).
Future Work: The paper highlights the need for models that can better understand the hierarchical semantics of taxonomies and the structural nuances of financial tables. The released code and datasets provide a foundation for developing more reliable, automated XBRL tagging systems.

Availability:

Code: GitHub (The-FinAI/FinTagging)
Datasets: Hugging Face (TheFinAI/fintagging)