💬 NLP

Taxonomy-Aligned Risk Extraction from 10-K Filings with Autonomous Improvement Using LLMs

This paper presents a three-stage LLM-based pipeline for extracting structured risk factors from 10-K filings that aligns with a predefined taxonomy and features an autonomous agent for continuous taxonomy refinement, demonstrating both high extraction accuracy and the ability to capture economically meaningful industry-specific risk profiles.

Original authors: Rian Dolphin, Joe Dursun, Jarrett Blankenship, Katie Adams, Quinton Pike

Published 2026-01-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rian Dolphin, Joe Dursun, Jarrett Blankenship, Katie Adams, Quinton Pike

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a financial detective trying to understand why thousands of different companies might fail. You have a massive library of annual reports (called "10-K filings"), but the "Risk Factors" section in each one is written in messy, unique human language. One company might say, "We worry about the dollar getting too strong," while another says, "Foreign currency swings could hurt us." They mean the same thing, but a computer sees them as totally different problems.

If you just let a smart AI (a Large Language Model, or LLM) read these reports and list the risks, you'd get a chaotic mess of different labels. It's like asking a hundred people to sort a pile of mixed-up toys into boxes, but everyone uses their own names for the boxes. You end up with "spinning wheels," "round things," and "car parts" instead of a neat "Wheels" category.

This paper presents a three-step system to fix that mess and organize the risks into a clean, consistent list, while also teaching the system how to get better on its own.

The Three-Step Pipeline

Think of this process like a high-end quality control line in a factory:

1. The Expert Reader (Extraction)
First, an AI reads the messy text and pulls out every risk it finds. Crucially, it doesn't try to force the risk into a box yet. Instead, it acts like a lawyer: it finds the risk and copies the exact sentence from the report that proves the risk exists.

Analogy: Imagine a detective writing down every clue and the exact page number where they found it, without worrying about which "case file" it belongs to yet.

2. The Semantic Matchmaker (Mapping)
Next, the system uses a "semantic map" (a type of AI that understands meaning, not just keywords) to compare the detective's clues against a pre-made list of official risk categories (a Taxonomy). This list has 140 specific categories, like "Interest Rate Risk" or "Cybersecurity Risk."

Analogy: The system looks at the clue and asks, "Does this sound most like the 'Weather' file or the 'Legal' file?" It uses math to find the closest match.
The Problem: Sometimes, the math gets it wrong. It might force a clue into a box just because it's the "least bad" option, even if it doesn't really belong there.

3. The Strict Judge (Validation)
This is the most important step. A second AI acts as a "Judge." It looks at the clue, the proposed box, and the official description of that box. It gives the match a score from 1 to 5.

Analogy: Imagine a strict teacher grading a student's homework. If the student tries to put a math problem in the "History" folder, the teacher says, "No, that's a 1/5. That doesn't belong here."
The Result: The system throws away any low-scoring matches (the bad fits) and keeps only the high-quality ones. This ensures the final list is accurate.

The "Self-Improving" Feature

The paper introduces a cool new feature: Autonomous Improvement.

Usually, if a category in the list is confusing, a human has to find the error and fix it. Here, the system does it itself.

How it works: The "Judge" AI keeps a log of every time it gave a low score. An automated agent (a robot assistant) looks at these logs, finds patterns, and figures out why the system is confused.
The Fix: The agent rewrites the description of the confusing category to make it clearer.
Real Example: The paper tested this on a category called "Pharmaceutical Approval." The system was getting confused between US rules (FDA) and European rules (EMA). The agent noticed this pattern, realized the description was too US-focused, and rewrote it to include "international agencies."
The Result: This self-correction made the system's ability to distinguish between correct and incorrect matches improve by 104.7%. The system literally taught itself to be better at sorting.

Does It Actually Work? (The Proof)

To prove their system isn't just making things up, the authors tested it on 500 major US companies (the S&P 500). They asked a simple question: "Do companies in the same industry end up with similar risk profiles?"

The Test: They took the cleaned-up risk lists and compared them. They didn't tell the computer which companies were banks or which were drug makers. They just let the computer look at the risks.
The Finding: The computer naturally grouped banks together and drug companies together.
- Companies in the same industry had 63% more similar risk profiles than companies in different industries.
- For example, 83% of banks were flagged for "Interest Rate Risk," while only 22% of all other companies were. Meanwhile, banks rarely had risks about "Raw Materials," which is a huge worry for factories.
The Conclusion: The system successfully extracted real, economic truths about these companies without being explicitly told what industry they were in.

Summary

This paper describes a method to turn messy, unstructured financial text into a clean, organized list of risks. It uses a three-step team (Reader, Matcher, Judge) to ensure accuracy and includes a "self-correcting" robot that fixes its own mistakes over time. The result is a system that can automatically understand the specific dangers facing different types of businesses, just like a human expert would, but at the speed of a computer.

Technical Summary: Taxonomy-Aligned Risk Extraction from 10-K Filings with Autonomous Improvement Using LLMs

Problem Statement

Corporate 10-K filings contain a "Risk Factors" section (Item 1A) that offers critical insights into the uncertainties facing public companies. However, the unstructured nature and substantial length of these documents make manual, systematic analysis at scale impractical. While Large Language Models (LLMs) can extract risk information, a naïve approach yields idiosyncratic, inconsistent labels (e.g., "currency fluctuation risk" vs. "foreign exchange exposure") that prevent systematic comparison. The core challenge is to extract structured risk factors from free-form text while strictly adhering to a predefined, hierarchical taxonomy, ensuring that mappings are both semantically accurate and free of spurious assignments.

Methodology

The authors propose a three-stage pipeline that combines the strengths of LLMs, semantic embeddings, and LLM-based validation to achieve robust, taxonomy-aligned extraction.

1. Three-Tier Risk Taxonomy

The system utilizes a custom hierarchical taxonomy designed specifically for investment analysis, comprising:

7 Primary Categories (e.g., Strategic & Competitive, Financial & Market).
28 Secondary Categories.
140 Tertiary Categories (leaf-level classifications).
Unlike existing taxonomies that may over-subdivide risks (e.g., separating storm types), this taxonomy balances comprehensiveness with the granularity required for financial decision-making.

2. The Three-Stage Pipeline

Stage 1: LLM-Based Risk Extraction

Input: Raw Item 1A text from 10-K filings.
Process: An LLM (Claude 4.5 Sonnet) extracts a comprehensive list of risk factors without constraints to match the taxonomy.
Output: A list of tuples containing a free-form risk tag and a supporting quote from the original text. This "chain-of-thought" approach ensures grounded extraction and provides the text snippets necessary for subsequent semantic matching.

Stage 2: Embedding-Based Taxonomy Mapping

Process: Each extracted supporting quote is embedded using the Qwen3 Embedding 0.6B model, guided by a task-specific instruction ("Classify risk factor text..."). These are mapped to pre-computed embeddings of the 140 tertiary category descriptions.
Mechanism: Nearest-neighbor matching (dot product) assigns each risk to the most semantically similar category.
Limitation: This stage guarantees a mapping for every risk, even if no appropriate category exists, potentially leading to spurious assignments.

Stage 3: LLM-as-a-Judge Validation

Process: A separate LLM instance acts as a judge to evaluate the quality of each proposed mapping (quote to category).
Scoring: The judge rates the fit on a 1–5 scale (1 = Very poor, 5 = Excellent) and provides concise reasoning.
Filtering: Only mappings with scores $\ge$ 4 are retained for the final output. Lower-scoring mappings are logged with their reasoning to inform taxonomy refinement.
Deduplication: The final output retains only the highest-quality mapping for each taxonomy category per filing.

3. Autonomous Taxonomy Refinement

A distinct contribution of the paper is an autonomous workflow for continuous taxonomy improvement:

Feedback Loop: An AI agent analyzes logged low-quality mappings (scores < 4) to identify problematic categories.
Diagnosis: The agent aggregates failure patterns (e.g., geographic scope confusion, post-market vs. pre-market misalignment) based on the judge's reasoning.
Refinement & Testing: The agent proposes candidate description refinements and validates them using an embedding-based testing methodology. This involves measuring the "embedding separation" (the difference in average similarity scores between true positive and false positive test cases) to ensure the new description improves semantic discrimination.

Key Contributions

Robust Extraction Methodology: A hybrid pipeline that combines LLM extraction, embedding-based similarity, and LLM validation to reliably map free-form text to a fixed taxonomy while filtering false positives.
Investment-Specific Taxonomy: A three-tier hierarchy tailored for investment analysis, balancing coverage with practical utility.
Autonomous Maintenance: A systematic method where an AI agent uses evaluation feedback to autonomously identify, diagnose, and refine taxonomy categories, creating a continuous improvement loop.
Empirical Validation: Extensive evaluation on S&P 500 10-K filings demonstrating the economic validity of the extracted risk profiles.

Results and Evaluation

The methodology was evaluated on 2024 10-K filings for 500 S&P 500 companies, extracting 10,688 validated risk factors.

Autonomous Refinement Case Study

In a case study involving the "Pharmaceutical and medical device approval" category, the autonomous agent identified that the original description caused confusion regarding geographic scope (FDA vs. EMA) and timing (pre-market vs. post-market).

Outcome: The agent generated a refined description that explicitly listed multiple health authorities and emphasized pre-market approval.
Metric: This refinement resulted in a 104.7% improvement in embedding separation (from 0.064 to 0.132), significantly reducing false positives.

Industry Clustering Validation

To test if the extracted risks capture genuine economic structure, the authors analyzed risk profile similarity across industries without using industry codes during the mapping process.

Similarity: Companies in the same industry exhibited 63% higher risk profile similarity than cross-industry pairs.
Statistical Significance: The difference was highly significant ( $p < 0.001$ ) with a large effect size (Cohen's $d = 1.06$ ).
Predictive Power: Using risk similarity to predict industry membership yielded an AUC of 0.733 for broad 2-digit SIC sectors and 0.822 for specific 4-digit industries.
Granularity: The signal strengthened with finer industry definitions, confirming that the taxonomy captures specific industry characteristics rather than generic boilerplate.

Sector-Specific Patterns

Analysis of the "Depository Institutions" sector (SIC 60) revealed intuitive patterns:

Interest rate risk appeared in 83% of banks vs. 22% of all companies.
Capital and liquidity requirements appeared in 67% of banks vs. 3% overall.
Conversely, manufacturing-related risks (e.g., raw material availability) were significantly depleted in the banking sector.

Significance and Claims

The paper claims to provide a general framework for extracting structured information from unstructured text while maintaining adherence to predefined categorical systems. The significance lies in:

Solving the "Unstructured-to-Unstructured" Trap: Moving beyond simple LLM extraction that produces inconsistent tags to a system that enforces a consistent, curated taxonomy.
Continuous Improvement: Shifting taxonomy development from a static design problem to an iteratively improving system where quality increases as more documents are processed, reducing the burden of human maintenance.
Economic Validity: Demonstrating that the extracted data is not merely linguistic but captures genuine economic risk dimensions, as evidenced by the strong correlation between risk profiles and industry sectors.

The authors note that while the current implementation focuses on English 10-K filings, the methodology is language-agnostic and applicable to other domains requiring taxonomy-aligned extraction. Limitations include the API costs and latency of LLMs, though these are mitigated by batch processing and caching.