The GELATO Dataset for Legislative NER

1. Problem Statement

Named Entity Recognition (NER) in the legislative domain presents unique challenges compared to general newswire or standard legal domains (like court judgments). Legislative texts feature:

• Specialized Terminology: Concepts such as specific bill numbers, statutory references, government programs, and protected classes are central but absent in general datasets.
• High Density of Citations: Legislative bills contain dense punctuation and references to other laws, complicating tokenization and entity boundary detection.
• Lack of Domain-Specific Ontologies: Existing legal ontologies (e.g., LKIF) are designed for relational logic or case law, focusing on locations and dates, rather than the structural constructs of congressional legislation (e.g., Acts, Funds, Committees).
• Data Scarcity: There is a lack of annotated datasets for U.S. congressional bills, hindering the development of automated extraction pipelines for policy analysis and legislative tracking.

2. Methodology

The authors propose a comprehensive solution involving dataset creation, a novel ontology, and a two-stage modeling pipeline.

A. The GELATO Dataset and Ontology

• Dataset: The authors compiled and annotated 131 bills from the 118th U.S. Congress (House and Senate, bill numbers 100–200).
  • Split: Training (80 House bills), Development (21 House bills), and Test (30 Senate bills).
  • Annotation: Performed by three annotators using Label Studio, with a consensus-based adjudication process. Inter-annotator agreement (IAA) was high (Level 1 F1: ~73–80%).
• GELATO Ontology: A novel two-level NER ontology designed specifically for legislative text.
  • Level 1 (Top-Level): 6 classes: Person, Organization, Document, Act, Abstraction, Class.
  • Level 2 (Subclasses): 30 subclasses tailored to legislative contexts.
    • Person: Member (MBC), Title (TTL), Individual (IND).
    • Organization: Nation, State, Locality, Committee, Agency, Legislative Body, International Institution, Association.
    • Document: Code, Bill, Reference, Parenthetical, Treaty, Report.
    • Act: Public Act, Amendment.
    • Abstraction: Program, System, Infrastructure, Fund, Doctrine, Case, Specification, Session.
    • Class: Protected Class, Non-Protected Class.

B. Modeling Approach: Two-Stage Pipeline

The authors propose a hybrid approach combining fine-tuned transformer models with Large Language Models (LLMs) to handle the hierarchical nature of the ontology.

1. Level 1 Prediction (Sequence Labeling):

   • Models: Fine-tuned BERT and RoBERTa models (base and large variants).
   • Task: Standard token-level classification to identify the top-level entity type (e.g., identifying "Senate" as an Organization).
   • Optimization: Hyperparameter sweeps (learning rate, batch size, epochs, etc.) using Bayesian optimization.

2. Level 2 Prediction (Classification via LLM):

   • Strategy: A "no-training" approach using optimized prompts.
   • Mechanism: Once a Level 1 label is predicted, the context (50 tokens on either side) and the specific Level 1 label are fed into an LLM (Qwen3/Qwen-32B).
   • Prompt Engineering: The system uses DSPy to generate optimized prompts that include specific instructions, the mention, context, and the list of possible Level 2 subclasses. The LLM performs few-shot classification to determine the specific subclass (e.g., classifying "Senate" as Legislative Body).

3. Key Contributions

1. GELATO Ontology: Introduction of the first two-level NER ontology specifically designed for U.S. congressional legislation, bridging the gap between general NER and specialized legal domains.
2. Annotated Dataset: Release of a high-quality, adjudicated dataset of 131 bills with over 130k tokens and 5,800 mentions, publicly available on GitHub.
3. Hybrid Extraction Pipeline: Demonstration of a novel pipeline where smaller, fine-tuned transformers handle coarse-grained entity detection, and LLMs handle fine-grained subclassification via optimized prompting.
4. Baseline Performance: Establishment of strong baselines for legislative NER, showing that RoBERTa significantly outperforms BERT in this domain.

4. Results

Level 1 Performance

• RoBERTa-Large achieved the best performance with an F1 score of 88.72.
• BERT-Large performed significantly worse (F1 67.16), highlighting the importance of vocabulary size and tokenization strategies (Byte-Pair Encoding vs. WordPiece) for legislative text.
• The gap between models suggests that pre-training data diversity and vocabulary size are critical for domain adaptation in legislative texts.

Level 2 Performance

• Performance Drop: Level 2 F1 scores were lower than Level 1 (e.g., RoBERTa-Large dropped from 88.72 to 67.44).
• Error Propagation: Errors in Level 1 cascade to Level 2. If the top-level tag is wrong, the LLM cannot correctly classify the subclass.
• Label Variance: Performance varied significantly by subclass:
  • High F1: Legislative Body (95.27%), Bill (95.24%), Person (96.65%).
  • Low F1: Parenthetical (13.33%), Doctrine (0%), Amendment (0%), and notably Class (27.52%).
• LLM Behavior: The LLM occasionally refused to choose a label if none seemed satisfactory, leading to automatic errors. It also struggled with semantic nuances, such as distinguishing between "Member" (specific to Congress) and general "Name."

Error Analysis

• Class vs. Person: Models frequently confused "Class" (groups of people) with "Person," likely due to pluralization nuances (e.g., "veterans" vs. a specific veteran).
• Vocabulary Impact: RoBERTa's larger vocabulary (50k vs. 30k) and BPE tokenization allowed it to handle legislative terminology better than BERT.
• Data Scarcity: Rare classes like "Treaty" and "Case" had zero mentions in the dataset, making them impossible to evaluate.

5. Significance and Future Work

• Domain Adaptation: The study proves that while general pre-trained models can be adapted to legislative NER, they require domain-specific fine-tuning and larger vocabularies to handle specialized terminology.
• Pipeline Efficiency: The two-stage approach leverages the speed of fine-tuned models for broad detection and the reasoning capabilities of LLMs for fine-grained classification without requiring massive computational resources for full LLM fine-tuning.
• Applications: The GELATO dataset and pipeline enable downstream tasks such as automated legislative tracking, policy analysis, and government transparency initiatives.
• Limitations: The dataset is limited to the 118th Congress (2023–2025) and specific bill ranges. The two-stage pipeline is brittle to Level 1 errors. Future work should explore error correction mechanisms and expanding the dataset to include treaties and court cases referenced in bills.

In conclusion, the paper establishes a new benchmark for legislative NER, demonstrating that a combination of fine-tuned transformers and optimized LLM prompting is a viable strategy for extracting complex, hierarchical entities from U.S. legislative texts.