Utilizing Pre-trained and Large Language Models for 10-K Items Segmentation

This study introduces and evaluates two advanced methods, BERT4ItemSeg and GPT4ItemSeg, for segmenting items in 10-K reports, demonstrating that the hierarchical BERT-based model achieves superior accuracy while the LLM-based approach offers greater adaptability to regulatory changes.

The Gist

Imagine you are a detective trying to solve a mystery, but the evidence is hidden inside a massive, chaotic library. This library contains thousands of books called 10-K reports. These are annual reports that public companies in the US must file with the government.

Inside these books, the information is organized into specific chapters, or "Items." For example, Item 7 is the "Management Discussion" (where bosses explain how the company did), and Item 1A is the "Risk Factors" (where they admit what could go wrong).

The Problem:
The library is a mess. Sometimes the chapters are labeled "Item 7," sometimes "Management's Discussion," and sometimes the formatting is weird. In the past, researchers tried to find these chapters using Rule-Based Methods. Think of this like using a rigid metal ruler to measure a squiggly, wiggly snake. If the snake changes shape even a little bit, the ruler breaks, and you miss the measurement. These old methods were brittle, broke easily when companies changed their formatting, and often made mistakes.

The Solution:
The authors of this paper built two new, super-smart "detectives" to find these chapters automatically. They used the latest Artificial Intelligence (AI) technology.

Detective #1: The Super-Reader (BERT4ItemSeg)

This detective is like a highly trained librarian who has read millions of books before.

• How it works: It uses a technology called BERT (a pre-trained language model). Imagine this librarian has a super-power: they can read a single sentence and instantly understand the context of the whole paragraph.
• The Trick: Since 10-K reports are huge (sometimes longer than a novel), the librarian can't read the whole book at once. So, the authors gave the librarian a special strategy: read the book line by line.
  • The librarian reads one line, decides if it's the start of a new chapter, and passes that thought to a "manager" (a Bi-LSTM model).
  • The manager looks at the flow of thoughts from the librarian and says, "Yes, that line is the start of the Risk Factors chapter!"
• The Result: This detective is incredibly accurate (98%+ success rate). It's like having a librarian who never misses a page. However, you have to train this librarian yourself, and they need a powerful computer (a GPU) to work fast.

Detective #2: The Chatbot Genius (GPT4ItemSeg)

This detective is like a genius intern who has never seen a 10-K report before but is incredibly smart and can learn anything just by talking to you.

• How it works: This uses a Large Language Model (LLM) like ChatGPT. Instead of training it for months, you just give it a few examples (like showing it three pages of a book and saying, "See? This is where the Risk Factors start. Now find it in this new book"). This is called "few-shot prompting."
• The Problem: Geniuses sometimes "hallucinate." They might make up a story that sounds true but isn't. If you ask a chatbot to "extract the text," it might rewrite the text slightly, which is bad for legal documents where exact words matter.
• The Trick: The authors invented a clever game called Line-ID Prompting.
  • Instead of asking the chatbot to "write out the chapter," they put a number (an ID) next to every line in the document.
  • They ask the chatbot: "Just tell me the numbers of the lines where the chapters start."
  • Once the chatbot gives the numbers, the computer automatically grabs the exact text from those lines.
• The Result: This detective is very good at adapting. If the government changes the rules tomorrow and adds a new chapter, you just tell the chatbot the new rule, and it figures it out instantly. It's flexible and doesn't need a super-computer, but it costs a little bit of money to use the chat service.

The Showdown

The authors tested both detectives on 3,737 real 10-K reports.

• The Old Way (Rules): Got it right about 90% of the time.
• The Chatbot (GPT4ItemSeg): Got it right about 95% of the time. It's great for new, weird situations.
• The Super-Reader (BERT4ItemSeg): Got it right about 98% of the time. It's the most accurate and reliable for standard work.

Why Does This Matter?

Think of financial research like building a house. If your foundation (the data extraction) is shaky, the whole house will fall down.

• Before this paper, researchers were building houses on shaky, rule-based foundations.
• Now, they have a solid, automated foundation.
• This means investors, auditors, and researchers can trust the data they are analyzing. They can find risks, compare companies, and understand market trends much faster and more accurately.

In a nutshell: The authors replaced a broken, rigid ruler with two smart, flexible AI detectives. One is a precise, trained librarian for the heavy lifting, and the other is a quick-learning genius for new challenges. Together, they make reading financial reports easier, faster, and much more accurate.

Technical Summary

1. Problem Statement

The extraction of specific disclosure items (e.g., Risk Factors, MD&A) from SEC Form 10-K filings is a critical prerequisite for textual analysis in accounting and finance. However, this task faces significant challenges:

• Format Variability: 10-K reports vary widely in document structure, item ordering, and formatting (HTML vs. plain text).
• Limitations of Rule-Based Methods: Traditional approaches rely on regular expressions and keyword matching. These are brittle, difficult to maintain as SEC regulations evolve, and prone to errors when formatting deviates from the norm.
• Context Window Constraints: 10-K filings are often tens of thousands of words long, exceeding the context limits of many standard models.
• Hallucination: Large Language Models (LLMs) often generate text that sounds plausible but does not match the source document exactly, which is unacceptable for regulatory data integrity.
• Lack of Benchmarks: There is no large-scale, manually annotated dataset to systematically evaluate and compare segmentation algorithms.

2. Methodology

The authors propose two novel approaches to address these challenges, utilizing a newly constructed dataset of 3,737 manually annotated 10-K filings (spanning fiscal years 2001–2019).

A. Data Preparation

• Dataset: Randomly sampled 10-Ks converted to a clean text format while preserving line and paragraph boundaries.
• Annotation: 85 annotators labeled lines using a modified BIO (Beginning-Inside-Outside) scheme (e.g., B1 for the start of Item 1, I1 for subsequent lines, O for non-item lines).
• Quality Control: Inter-annotator agreement was verified using Cohen's kappa (0.92), with a dedicated test set held out for evaluation.

B. Approach 1: BERT4ItemSeg (PLM-Based)

This method addresses the context window limitation of BERT (512 tokens) using a hierarchical architecture:

1. Lower Level (Line Encoding): A pre-trained Sentence-BERT model encodes individual lines of text into fixed-length vectors. Since most lines are shorter than 512 tokens, this bypasses the context limit.
2. Upper Level (Sequence Modeling): A Bidirectional LSTM (Bi-LSTM) takes the sequence of line embeddings and predicts the label (B, I, or O) for each line.
3. Advantage: The Bi-LSTM has no fixed sequence length limit, allowing the model to process 10-Ks of arbitrary length while leveraging BERT's contextual understanding.

C. Approach 2: GPT4ItemSeg (LLM-Based)

This method leverages ChatGPT-4o but introduces a novel Line-ID-Based (LIB) Prompting Mechanism to solve hallucination and context issues:

1. Line ID Injection: The input text is pre-processed to attach a unique integer ID to the start of every line.
2. Prompting: The model is asked to output only the Line IDs corresponding to the start of specific items, rather than regenerating the text.
3. Text Reconstruction: A post-processing script uses the returned Line IDs to extract the exact original text from the source file.
4. Few-Shot Learning: The prompt includes a small number of examples (few-shot) demonstrating how to identify item boundaries in partial filings.
5. Advantage: This ensures 100% fidelity to the source text (no hallucination) and allows the model to handle long documents by focusing on boundary detection rather than full-text generation.

3. Key Contributions

1. First Large-Scale Benchmark: Creation and release of a manually annotated dataset of 3,737 10-K filings, enabling systematic evaluation of segmentation performance.
2. Novel Architectures:
   • BERT4ItemSeg: A hierarchical BERT-Bi-LSTM model that overcomes context window limits for long documents.
   • GPT4ItemSeg: A novel LIB prompting strategy that mitigates LLM hallucination in regulatory text extraction.
3. Comparative Framework: A rigorous comparison of PLMs, LLMs, traditional CRF models, and rule-based baselines.
4. Open Source Resources: Release of the dataset, code, and a comprehensive preprocessing pipeline (cleaning HTML, unescaping characters, removing numerical tables) to the research community.

4. Experimental Results

The models were evaluated on "Core Items" (Items 1, 1A, 3, 7) and "Other Items" using Macro-F1 scores.

Model	Core Items (Macro-F1)	Other Items (Macro-F1)	Key Characteristics
BERT4ItemSeg	0.9826	0.9692	Highest accuracy; requires local GPU; robust to format variations.
CRF (Baseline)	0.9788	0.9691	Strong performance; runs on CPU; struggles with new items.
GPT4ItemSeg	0.9552	0.9385	High adaptability; requires API costs; prone to higher variance on rare items.
Rule-Based	0.9048	0.9010	Lowest performance; brittle; high maintenance cost.

• Statistical Significance: BERT4ItemSeg significantly outperformed all other methods on core items. For other items, it was statistically superior to GPT4ItemSeg and Rule-Based methods, though comparable to CRF.
• Few-Shot Sensitivity: GPT4ItemSeg performance plateaued after 5 examples; adding more examples did not improve results but increased token costs.
• Error Analysis: The primary failure modes were random errors, very short reports (lack of context), and unconventional item placements (e.g., items split across sections).

5. Significance and Implications

• Research Reproducibility: By providing a standardized dataset and robust segmentation tools, the study eliminates a major source of noise and irreproducibility in financial text analytics.
• Regulatory Adaptability:
  • GPT4ItemSeg is ideal for rapid adaptation to new SEC rules (e.g., new cybersecurity disclosure items) via prompt engineering.
  • BERT4ItemSeg offers a cost-effective, locally deployable solution for stable, high-volume processing once models are fine-tuned.
• Data Integrity: The LIB mechanism ensures that extracted text is an exact copy of the source, a critical requirement for legal and financial analysis where hallucination is unacceptable.
• Generalizability: The framework can be extended to other filings (10-Q, 8-K, 40-F) and domains, offering a scalable solution for automated disclosure analysis.

In conclusion, the paper demonstrates that combining deep learning (PLMs) with advanced prompting strategies (LLMs) significantly outperforms traditional rule-based methods, providing a robust foundation for the next generation of accounting and finance text analytics.