Technical Summary: SemHash-LLM

1. Problem Statement

Large-scale document deduplication faces a fundamental trade-off between efficiency and semantic fidelity. Traditional lexical fingerprinting methods (e.g., MinHash, SimHash) are computationally efficient but brittle against paraphrasing, template wrapping, and minor character perturbations. Conversely, pure embedding-based retrieval offers superior semantic awareness but incurs prohibitive computational costs when applied to corpora containing billions of documents. Furthermore, existing solutions often fail to address the heterogeneity of modern web data, which includes long documents, short snippets, boilerplate-heavy pages, and complex containment hierarchies. The challenge is to achieve aggressive deduplication that preserves semantic equivalence without incurring the cost of full neural verification for every document pair.

2. Methodology: SemHash-LLM Framework

The authors propose SemHash-LLM, a multi-granularity framework that unifies semantic projection hashing, attention-weighted lexical sketching, contrastive boundary learning, and selective Large Language Model (LLM) adjudication. The system operates through a cascaded filtering pipeline designed to reduce candidate pairs to less than 1% before applying expensive neural verification.

Core Components

A. Semantic Projection Hashing (SPH)
To bridge the gap between discrete token representations and continuous semantic space, SPH learns locality-sensitive hash functions directly within a distilled LLM embedding space.

• Distillation: A lightweight student encoder is trained via knowledge distillation (combining KL divergence and L2 distance) to mimic a larger teacher LLM, using mean pooling over token hidden states rather than a single classification token.
• Learnable Hashing: Instead of random hyperplanes, the framework learns a family of hyperplanes to map embeddings to compact binary codes. The training objective optimizes for similarity preservation, orthogonality (to prevent redundant bits), and bit balance.
• Optimization: The discrete sign function is approximated using a straight-through estimator with a temperature-annealed tanh function to enable gradient flow during training.

B. Attention-Weighted MinHash (AW-MinHash)
This component addresses "template pollution" where boilerplate (menus, ads, cookies) dominates standard MinHash signatures.

• Importance Weighting: Token importance is derived from aggregated multi-head attention scores across all transformer layers. High attention indicates central content, while diffuse attention signals peripheral or boilerplate text.
• Weighted Sampling: These attention scores, combined with Inverse Document Frequency (IDF), generate weights for n-grams. Consistent weighted sampling is then used to create MinHash signatures that prioritize discriminative content over boilerplate.
• Adaptive LSH: The number of bands in the Locality Sensitive Hashing (LSH) is dynamically adjusted based on the document's attention entropy. Documents with concentrated attention (clear boundaries) use aggressive banding, while those with diffuse attention use finer discrimination.

C. Contrastive Boundary Learning
Recognizing that fixed similarity thresholds fail across diverse document types, the framework learns adaptive decision boundaries.

• Contrastive Projection: A projection network maps combined semantic and lexical features into a space where duplicates cluster tightly.
• Type-Specific Optimization: A boundary network predicts duplicate probability using multiplicative and difference interactions. Thresholds are optimized via margin-based loss for five specific deduplication categories: template-wrapped near-duplicates, distribution-skewed hot-spots, short text perturbations, parent-child containment, and viral fragments.
• Uncertainty Quantification: Monte Carlo Dropout is used to estimate prediction variance. High variance indicates ambiguous pairs requiring further adjudication.

D. LLM-as-Judge Refinement
For the small fraction of candidate pairs (approx. 3%) flagged as uncertain by the automated pipeline, an instruction-tuned LLM acts as a judge.

• Structured Prompting: Documents are presented with explicit criteria and few-shot examples of duplicate types.
• Confidence-Weighted Ensemble: The final decision fuses the automated prediction and the LLM's judgment. The weighting is dynamic: if the automated system is confident, it dominates; if the LLM is confident, its judgment is weighted heavily. This ensures expensive LLM inference is reserved only for borderline cases.

E. Multi-Granularity Fusion Network (MGFN)
To handle different deduplication types, the system hierarchically fuses features from three levels:

• Character-level: Captures fine-grained perturbations via multi-scale convolutions.
• Token-level: Identifies template patterns via bidirectional LSTM.
• Semantic-level: Projects LLM embeddings through a normalized feed-forward network.
• Gated Fusion: A gating mechanism adaptively weights these features based on document characteristics (e.g., character features for short texts, semantic features for long documents).

F. Cascaded Filtering Pipeline
The system reduces the search space through four stages:

1. Bloom Filter: Eliminates exact n-gram duplicates (~40% reduction).
2. Semantic Hash Blocking: Uses SPH codes with Hamming radius to partition documents (~95% reduction).
3. Attention-Weighted LSH: Refines candidates using AW-MinHash signatures.
4. Neural Verification: The MGFN performs final verification.
   The combined pass rate ensures neural computation is applied to less than 1% of all pairs.

3. Key Contributions

• Unified Framework: The paper introduces a system that successfully integrates learned semantic hashing, attention-weighted lexical sketching, and adaptive contrastive learning into a single pipeline.
• Semantic Projection Hashing: A novel method for learning binary codes in continuous embedding spaces that preserves semantic similarity while maintaining the efficiency of hashing.
• Attention-Weighted MinHash: A technique that leverages transformer attention patterns to suppress boilerplate and emphasize informative content in lexical signatures.
• Selective LLM Adjudication: A routing mechanism that applies computationally expensive LLM reasoning only to high-uncertainty cases, achieving high accuracy with minimal cost.
• Multi-Granularity Handling: The framework explicitly addresses five distinct deduplication categories (template wrapping, distribution skew, character perturbation, containment, and viral fragments) through specialized feature extraction and fusion.

4. Experimental Results

Experiments were conducted on a 100GB web corpus from RedPajama. The framework was evaluated against baselines including SimHash, MinHash-LSH, NearDup-BERT, and DedupLM across five deduplication categories (Types A–E).

• Performance: SemHash-LLM achieved a final weighted composite score of 91.05, outperforming the next best baseline (DedupLM at 81.20).
• Ablation Studies: Removing key components resulted in significant performance drops:
  • Without Semantic Projection Hashing (w/o SPH): Score dropped to 79.32.
  • Without Attention-Weighted MinHash (w/o AW-MinHash): Score dropped to 85.18.
  • Without Multi-Granularity Fusion (w/o MGFN): Score dropped to 84.56.
• Efficiency: The cascaded pipeline reduced the candidate space such that neural verification was required for less than 1% of document pairs, while maintaining high recall and precision.

5. Significance and Claims

The paper claims that SemHash-LLM breaks the conventional efficiency-versus-accuracy compromise in document deduplication. By utilizing a cascaded pipeline for pruning and uncertainty-aware routing, the system achieves state-of-the-art detection quality across diverse document types while keeping neural verification costs below 1%. The authors assert that this approach provides a scalable solution for the "first-order systems problem" of curating open pretraining corpora, effectively handling the redundancy that negatively impacts large language model memorization and optimization. The framework is presented as a robust, unified architecture capable of managing the heterogeneity of modern web data without relying on brittle lexical filters or prohibitively expensive full-scale semantic retrieval.