Benchmarking Large Language Models on Reference Extraction and Parsing in the Social Sciences and Humanities

1. Problem Statement

The paper addresses the critical challenge of bibliographic reference extraction and parsing within the Social Sciences and Humanities (SSH) domain. While citation processing is foundational for scholarly infrastructure (e.g., citation graphs, knowledge bases), existing solutions face significant limitations:

• Evaluation Mismatch: Standard benchmarks rely on clean, English, end-of-document bibliographies. They fail to capture SSH realities where citations are often multilingual, embedded in footnotes, abbreviated, and subject to heterogeneous historical conventions.
• Fragility of Current Systems: Traditional supervised pipelines (like GROBID) struggle with distribution shifts, particularly when dealing with complex page layouts, footnotes, and non-English languages.
• LLM Potential vs. Reality: While Large Language Models (LLMs) show promise, their reliability for structured citation parsing under noisy, real-world SSH conditions remains unquantified against strong baselines.

2. Methodology

Datasets (The Unified Benchmark)

The authors constructed a unified benchmark using three complementary datasets to simulate SSH-realistic conditions:

1. CEX: 112 English journal articles across 27 disciplines. Used as a controlled baseline for cross-domain generalization.
2. EXCITE: 351 German/English documents (SSH focus) featuring explicit variations in citation placement: end-section, footnote-only, and mixed regimes. Includes German-specific abbreviations (e.g., Ebd., a.a.O.).
3. LinkedBooks: A humanities-focused corpus (Venice historiography) with 1,194 manually annotated references. It features high stylistic variation, primary sources, and significant multilinguality (72.3% Italian, 18.3% English, etc.).

Tasks

The study evaluates three tasks of increasing difficulty:

1. Reference Extraction: Detecting and delimiting reference spans within raw text (PDF-to-text).
2. Reference Parsing: Converting gold-standard reference strings into structured JSON fields (Author, Title, Year, Venue, etc.).
3. End-to-End Document Parsing: A pipeline task performing both extraction and parsing directly from raw PDFs.

Systems Evaluated

• Baseline: GROBID (Deep Learning sequence labeling model), a strong supervised pipeline standard.
• LLMs: A diverse set of contemporary models including DeepSeek-V3.1, Mistral-Small-3.2-24B, Gemma-3-27B-it, and the Qwen3-VL family (4B–32B variants).
• Constraints: All LLMs were forced to output strict JSON using a unified schema to ensure comparability. Inputs were converted to Markdown (for PDFs) to isolate parsing capabilities.

Optimization Strategies

The paper investigates two practical levers to improve performance:

1. LoRA Adaptation: Parameter-efficient fine-tuning (Low-Rank Adaptation) on a small, curated SSH dataset (1,832 instances) to adapt models to specific citation styles.
2. Segmentation/Pipelining: Comparing "Single-call" (global context) vs. "Adaptive Pipeline" (per-page segmentation and extract-then-parse) vs. "Semantic Pre-selection" (retrieving only reference-bearing regions).

3. Key Contributions

1. Unified SSH Benchmark: Introduced a comprehensive evaluation framework covering extraction, parsing, and end-to-end processing across three distinct datasets (CEX, EXCITE, LinkedBooks) with a shared schema.
2. Comprehensive Baseline Comparison: Provided a side-by-side analysis of GROBID against diverse LLMs, identifying where LLMs outperform traditional pipelines under distribution shifts (multilinguality, footnotes).
3. Quantification of Adaptation: Demonstrated that lightweight LoRA fine-tuning yields consistent gains, particularly on SSH-heavy benchmarks, and that prompt alignment matters.
4. Hybrid Deployment Strategy: Proposed a routing-based architecture that leverages the speed of GROBID for in-distribution documents while escalating complex, multilingual, or footnote-heavy cases to adapted LLMs.

4. Key Results

Performance Trends

• Extraction: LLMs largely saturate this task once a moderate capability threshold is reached. Differences between top models are minimal; extraction is less of a bottleneck than parsing.
• Parsing & End-to-End: These remain the primary bottlenecks due to "structured-output brittleness" under noisy layouts.
  • GROBID excels on CEX (in-domain English) but degrades significantly on EXCITE (German/footnotes) and LinkedBooks (multilingual/humanities).
  • LLMs (especially larger variants like DeepSeek-V3.1 and Qwen3-32B) demonstrate superior robustness on EXCITE and LinkedBooks, often surpassing GROBID in F1 scores for parsing and end-to-end tasks.
• Scaling Effects: Model size impacts parsing stability significantly. Smaller models (e.g., Qwen3-4B) suffer from higher failure rates (invalid JSON, truncation) on long documents, while larger models maintain structural integrity.

Impact of Optimization

• LoRA Fine-tuning:
  • Yielded consistent improvements across all datasets.
  • LinkedBooks saw the largest gains (~+10% Micro F1), proving that even small SSH-specific supervision sets significantly boost performance on style-rich, multilingual data.
  • LoRA models often performed better with shorter, zero-shot training-style prompts, though some models still benefited from few-shot examples.
• Segmentation:
  • For Extraction, single-call (global context) was generally best.
  • For End-to-End Parsing, the Adaptive Pipeline (per-page segmentation + extract-then-parse) significantly outperformed single-call approaches. It reduced error accumulation and localized failures, improving Micro F1 by ~10-15% on EXCITE compared to single-call.

Error Analysis

• Failure Modes: The dominant errors for LLMs were "Minor" (field omissions/formatting) rather than "Major" (complete hallucination). However, "Structural Errors" (malformed JSON) increased with smaller models and longer contexts.
• Domain Sensitivity: All systems performed worse on SSH/Humanities categories compared to STEM, highlighting the difficulty of handling stylistic variation and specialized terminology.

5. Significance and Future Directions

• Practical Deployment: The paper argues against a "one-size-fits-all" approach. Instead, it advocates for Hybrid Routing: using GROBID for standard, well-structured English PDFs to maximize throughput, while routing complex SSH documents (multilingual, footnote-heavy) to task-adapted LLMs.
• SSH Digitization: The findings validate that LLMs, when properly adapted, are viable for the digitization of SSH collections, a domain historically underserved by automated tools.
• Future Work: The authors suggest extending LoRA adaptation to the full pipeline (layout analysis + extraction + parsing) and developing modular adapters for low-resource languages (e.g., Latin, Polish) to further enhance global coverage.

In summary, the paper establishes that while traditional pipelines are fast and reliable for standard inputs, LLMs combined with LoRA adaptation and smart segmentation strategies offer a robust, scalable solution for the complex, heterogeneous reality of Social Sciences and Humanities literature.