OpenSanctions Pairs: Large-Scale Entity Matching with LLMs

1. Problem Statement

Entity Matching (EM), also known as entity resolution or deduplication, is the task of determining whether two records refer to the same real-world entity. While EM is a core component of data integration, existing benchmarks (e.g., Abt-Buy, DBLP-ACM) are limited to e-commerce or bibliographic domains. They typically feature:

• Clean, structured, monolingual data.
• Homogeneous schemas with fixed field sets.
• Limited scale (few datasets, few sources).

The Gap: Real-world compliance settings, specifically international sanctions screening, involve significantly higher complexity:

• Heterogeneity: Data aggregated from hundreds of sources (293 datasets in this study) across 31 countries.
• Noise & Ambiguity: Missing attributes, inconsistent formatting, and set-valued fields (e.g., multiple aliases, addresses).
• Multilingualism: Names appear in non-Latin scripts (Cyrillic, Chinese, Arabic) requiring transliteration handling.
• High Stakes: Errors lead to regulatory fines or reputational damage.
• Lack of Benchmarks: No public, large-scale benchmark existed for pairwise entity matching on sanctions data.

2. Methodology

A. Dataset Construction: OpenSanctions Pairs

The authors collaborated with OpenSanctions to create the first large-scale EM benchmark for sanctions data.

• Scale: 755,540 labeled pairs covering 1,002,093 unique entities.
• Sources: 293 heterogeneous datasets from 31 countries.
• Labeling Process: Human analysts performed pairwise deduplication. Crucially, labels represent expert human decisions under incomplete evidence rather than definitive ground truth, mirroring real-world operational constraints.
• Data Characteristics:
  • Imbalance: 76.9% positive (match) vs. 23.1% negative (non-match), reflecting a blocking strategy that prioritizes recall.
  • Complexity: Entities have set-valued attributes (aliases, nationalities) and span multiple scripts.

B. Experimental Setup

The study benchmarks three categories of methods:

1. Rule-Based Baseline: The nomenklatura RegressionV1 algorithm used in production. It computes 18 features (token overlap, Levenshtein distance, phonetic matching, date/ID consistency) and uses a logistic regression model.
2. Open-Source LLMs:
   • Llama-3.1-8B-Instruct
   • DeepSeek-R1-Distill-Qwen-14B (locally deployable, no usage costs).
3. Proprietary (Closed-Source) LLMs:
   • Includes GPT-5 Nano, GPT-5.2 Pro, GPT-4o, Claude Opus 4.5, etc.

Prompting Strategy:

• Conflict Detection Framing: Instead of asking for "similarity," prompts were designed to identify contradictions. The default assumption is "Positive" (same entity) unless explicit conflicts (different IDs, dates) are found.
• Optimization: Used DSPy MIPROv2 for Bayesian optimization of instructions and in-context examples (few-shot).
• Evaluation: Stratified sampling (10k and 1k pairs). Primary metric is F1-score, given the class imbalance and the asymmetric cost of false negatives in compliance.

3. Key Results

Performance Comparison

• Rule-Based Baseline: Achieved 91.33% F1. It exhibited high recall (99.42%) but low precision (84.46%), resulting in many false positives (over-matching).
• Open-Source LLMs:
  • DeepSeek-R1-Distill-Qwen-14B (optimized) achieved 98.23% F1.
  • Llama-3.1-8B (optimized) achieved 95.94% F1.
• Proprietary LLMs:
  • GPT-4o achieved the highest score of 98.95% F1.
  • GPT-5.2 Pro reached 98.75% F1.

Key Findings

1. LLMs Surpass Rule-Based Systems: Off-the-shelf LLMs substantially outperform the production rule-based baseline, closing the gap to human expert consistency.
2. Diminishing Returns on Pairwise Optimization: Performance is approaching a practical ceiling (~99% F1). Further tuning of pairwise matching models yields minimal gains.
3. Prompt Optimization vs. Few-Shot:
   • Automatic prompt optimization (MIPROv2) provided consistent, modest gains (e.g., +1.9 F1 for Llama-3.1-8B).
   • Counter-intuitive Finding: Adding in-context examples (few-shot) often degraded performance or provided no benefit when combined with optimized prompts. The authors suggest that demonstrations introduce inductive biases that compete with the optimized instruction strategy.
4. Failure Modes:
   • Rule-Based: Prone to false positives (over-matching common names without unique identifiers).
   • LLMs: Prone to false negatives on cross-script transliteration issues and minor "off-by-one" errors in dates/IDs (over-sensitivity to noise).

4. Contributions

1. OpenSanctions Pairs Dataset: Release of a public, large-scale (755k pairs), heterogeneous entity matching benchmark derived from real-world sanctions data, an order of magnitude larger and more complex than existing benchmarks.
2. Comprehensive Benchmarking: Established baselines comparing legacy rule-based systems against a wide range of open and closed-source LLMs in zero-shot and few-shot settings.
3. Strategic Insight: Demonstrated that pairwise matching is no longer the primary bottleneck in sanctions screening. The research identifies that future value lies in pipeline components (blocking, clustering, uncertainty-aware review) rather than isolated pairwise classification.

5. Significance and Future Directions

• Automation Potential: The results indicate that tasks previously requiring significant manual domain expertise can be reliably automated using LLMs, reducing the burden on human analysts.
• Shift in Research Focus: Since pairwise matching performance is nearing a ceiling, the authors argue the research community should shift focus to:
  • Blocking: Efficiently reducing the candidate set.
  • Clustering: Grouping entities into unified profiles.
  • Uncertainty Handling: Designing workflows where human review is triggered only for low-confidence matches.
• Compliance Impact: By reducing false positives (over-matching) compared to rule-based systems, LLMs can lower the operational cost of compliance screening while maintaining high recall.

Conclusion: The paper establishes that LLMs have matured to a point where they can handle the extreme complexity of international sanctions data, effectively solving the pairwise matching problem and necessitating a re-evaluation of the broader entity resolution pipeline.