Grounded Event Extraction from SEC 8-K Filings with a Fine-Grained Taxonomy

Technical Summary: Grounded Event Extraction from SEC 8-K Filings with a Fine-Grained Taxonomy

Problem Statement
Form 8-K filings are the primary mechanism for U.S. public companies to disclose material corporate events. However, the standard machine-readable labels for these filings—32 SEC item codes—are economically coarse. A single item code often conflates distinct economic events (e.g., Item 5.02 covers both the departure of a Chief Executive and the routine retirement of an outside director), while many high-impact disclosures (such as clinical trial results or merger completions) fall into the uninformative "catch-all" Item 8.01 ("Other Events"). While Large Language Models (LLMs) offer the potential to assign fine-grained, economically motivated labels at corpus scale, generic prompting fails to meet two critical requirements for research and application: constraint (outputs must map to a fixed, reliable vocabulary) and auditability (every label must be traceable to source text and assessed for reliability without re-running the model).

Methodology
The authors present a two-stage extraction system designed to tag 8-K filings against a three-tier taxonomy of 119 event types (8 primary, 29 secondary, 119 tertiary). The system prioritizes structural reliability over model scale, utilizing a compact, instruction-tuned language model.

1. Taxonomy Construction: The taxonomy is organized by economic event type rather than SEC item number. It includes specific definitions for each tertiary type (e.g., distinguishing a CEO departure from a CFO departure, or a merger agreement from a merger completion).
2. Stage 1: Constrained Extraction:
   • Schema Validation: The model is prompted with the full taxonomy. Output is checked against a strict schema; any tag not matching an exact taxonomy entry is rejected, triggering a self-correcting retry (up to three times).
   • Verbatim Quote Grounding: Every extracted tag must cite a supporting quote from the filing. A validator checks this quote using fuzzy n-gram matching (4-word sequences). The quote is accepted only if at least 40% of its overlapping 4-word runs appear verbatim in the filing text. This prevents the model from fabricating evidence.
3. Stage 2: Dedicated Scoring (Re-grading):
   • Unlike standard approaches where the extraction model rates its own output inline, this system employs a separate "grader" model.
   • The grader receives only the cited quote and the category definition (not the full filing). It assigns a quality score from 1 to 5 based on how well the quote alone establishes the event and fits the category.
   • This dedicated pass transforms the score into a calibrated reliability dial, avoiding the over-confidence observed when models self-rate during generation.

Key Contributions

1. Verifiable Extraction Recipe: A system where reliability is enforced through schema constraints and verbatim quote grounding, rather than relying on the model's internal judgment.
2. Calibrated Quality Score: The demonstration that a dedicated second pass for scoring is essential. The authors show that inline self-rating saturates (placing ~75% of tags at the top score), whereas the second-stage re-grade spreads tags across the full 1–5 range, creating a usable filter.
3. Economic Validation of Coarseness: An event study proving that SEC item codes are economically coarse and that fine-grained taxonomy tags capture information that item codes lack, without using language models for the market analysis itself.
4. Dataset Release: The release of 601,088 grounded, quality-scored event tags across 292,984 filings (2022–2026).

Results

• Intrinsic Evaluation (LLM-as-Judge): Using a stronger LLM (Opus 4.8) to judge 5,125 stratified tags, the study found that precision rises monotonically with the quality score.
  • Score 1: 12.3% precision, 8.0% unsupported (hallucinated) rate.
  • Score 5: 96.4% precision, 0.0% unsupported rate.
  • Filtering for scores 4 and above retains 55% of tags at 92.7% precision with a 0.1% unsupported rate.
• Ablation Study: The "dial" effect is lost if scoring is done inline. Inline scoring behaves like a binary flag, retaining 73% of tags at only 87.5% precision, unable to reach the higher precision levels achievable by the two-stage method.
• Economic Evaluation (Event Study): Analyzing 182,174 filing events using unsigned, variance-standardized abnormal returns (SAR):
  • Item Code Coarseness: Within Item 5.02, CEO departures show significantly higher market reactions (17.2% of filings move >2 standard deviations) compared to routine officer appointments (9.7%).
  • Taxonomy Value: The most reactive tags (e.g., clinical trial results, merger completions) are predominantly filed under Item 8.01.
  • Variance Explanation: Adding taxonomy tags to item codes increases the explained variance in reaction magnitude by 1.30 percentage points, whereas adding item codes to tags increases it by only 0.41 points.
  • Robustness: Filtering for high-confidence tags (score ≥4) does not alter the economic rankings, confirming that the score effectively filters out noise without discarding economically significant signals.
• Descriptive Validity: The tagged corpus successfully tracks regulatory changes (e.g., the 2023 cybersecurity rule, where ~2/3 of tagged incidents still appeared under Item 8.01 rather than the new Item 1.05), macroeconomic trends (financial distress tags rising with interest rates), and cross-sectional coherence (clinical trial tags concentrated in pharma/biotech).

Significance and Claims
The paper claims to provide a method for extracting fine-grained financial events that is both scalable and auditable. Its primary significance lies in proving that:

1. Reliability requires structural constraints: Verbatim quote grounding and schema validation are necessary to prevent hallucination and ensure outputs map to a fixed vocabulary.
2. Scoring requires separation: A dedicated re-grading pass is methodologically superior to inline self-rating for producing a calibrated, tunable reliability metric.
3. Item codes are insufficient: SEC item codes fail to capture economically distinct events, particularly those in the "Other Events" category. Fine-grained tags derived via this system provide significant additional explanatory power for market reactions.
4. External Validation: The system's quality scores correlate with economic reality; low-scoring tags correspond to low-precision labels and do not drive market reactions, while high-scoring tags isolate distinct economic signals.

The authors conclude that while quote grounding prevents fabricated evidence, it cannot fully prevent fabricated inference (though this residual is small at high confidence scores). The methodology is presented as transferable to other financial disclosures and document processing tasks.