Benchmarking and Stress-Testing Off-the-Shelf Vision Embeddings for Surface Re-Identification in Open-Source Intelligence

Technical Summary: Benchmarking Off-the-Shelf Vision Embeddings for Surface Re-Identification in OSINT

Problem Definition
Open-source intelligence (OSINT) increasingly relies on matching physical surfaces (e.g., granite countertops, wood grain, fabric weaves) across photographs to link anonymous images to specific locations. While a popular framing suggests that pretrained vision models (such as CLIP, DINOv2, or CNNs) can perform this matching with high confidence (often citing figures like "94.7% accuracy"), these claims frequently lack rigorous benchmarking, clear task definitions, or negative controls. This paper addresses the gap between anecdotal confidence and empirical performance by measuring what these off-the-shelf models actually achieve on a precisely defined surface re-identification task.

Methodology
The study defines surface re-identification as the task of retrieving images of the same physical surface instance from a gallery containing many different surfaces. The evaluation utilizes the KTH-TIPS2-b texture database, which comprises 4,752 images of 44 physical surface instances (11 materials, 4 distinct samples per material). Crucially, the dataset includes "hard negatives": images of different samples of the same material (e.g., two different wool swatches) which must be distinguished from the query.

The authors evaluate seven variants of off-the-shelf backbones as frozen feature extractors (no fine-tuning or metric learning):

• CLIP: ViT-B/32 and ViT-L/14.
• DINOv2: ViT-S/14, ViT-L/14, and ViT-S/14 at native 518px resolution.
• CNNs: ResNet-50 and EfficientNet-B0.

All models were tested on a common 224-pixel input (except the native DINOv2 variant) using cosine similarity on L2-normalized pooled embeddings. The evaluation employs standard retrieval metrics (Rank-1, Recall@5, mean Average Precision [mAP]) and verification metrics (ROC-AUC, Equal-Error Rate [EER]). A specific cross-scale analysis was conducted to simulate OSINT scenarios where query and match images are taken at different zoom levels. Statistical significance was determined using paired Wilcoxon signed-rank tests and 95% bootstrap confidence intervals.

Key Results

1. Ranking vs. Top-1 Performance: While Rank-1 accuracy is high across all models (ranging from 94.3% to 98.1%) and Recall@5 exceeds 99.4%, the mean Average Precision (mAP) is significantly lower (0.505 to 0.646). This indicates that while the single best match is often correct, the deeper ranking lists frequently mix in incorrect surfaces, including other samples of the same material.
2. Model Family Superiority: DINOv2 models outperform all other families, including CLIP. Notably, the smallest DINOv2 model (ViT-S/14, mAP 0.604) significantly outperforms the much larger CLIP ViT-L/14 (mAP 0.590). The best overall performer is DINOv2 ViT-L/14, achieving a Rank-1 of 98.1%, an mAP of 0.646, and an AUC of 0.973.
3. Limitations in Real-World Conditions:
   • Cross-Scale Degradation: When the query and gallery images are at different zoom levels, mAP drops for all models (e.g., DINOv2 ViT-L/14 drops from 0.646 to 0.631).
   • Material Dependence: Performance varies sharply by material. Smooth, specular surfaces (e.g., aluminum foil, white bread) are significantly harder to match (mAP as low as 0.489) than strongly patterned surfaces (e.g., lettuce leaf, mAP 0.840).
4. Verification Limits: Even the best model has an Equal-Error Rate (EER) of 7.2%, meaning that at a balanced operating point, roughly 1 in 14 decisions is incorrect.

Key Contributions

• Reproducible Benchmark: The paper establishes a reproducible benchmark on KTH-TIPS2-b where the physical sample is the identity unit, utilizing same-material-different-sample pairs as hard negatives.
• Honest Evaluation: It provides a rigorous, statistically validated evaluation of CLIP, DINOv2, ResNet-50, and EfficientNet-B0 as frozen embedders, reporting retrieval and verification metrics with confidence intervals.
• Cross-Scale and Material Analysis: The study isolates the impact of zoom changes and material properties, demonstrating where surface matching is reliable and where it fails.
• Correction of Common Advice: The results overturn the common recommendation to use CLIP for texture matching, demonstrating that self-supervised models (DINOv2) are superior regardless of model size.

Significance and Claims
The paper concludes that off-the-shelf vision embeddings are useful as lead generators for human analysts but do not support unqualified accuracy claims for automated identification. The high Rank-1 accuracy suggests these tools can surface a strong candidate for a human to verify, but the lower mAP and significant EER indicate they are not reliable enough for standalone proof, particularly across different zoom levels or on smooth surfaces.

The authors emphasize that a "94.7% accuracy" figure is misleading without context; the appropriate summary for OSINT practice is that these models generate leads that require human verification. Furthermore, for surface matching tasks specifically, a self-supervised model (DINOv2) is the recommended default over text-aligned models (CLIP), even when comparing smaller self-supervised models against larger text-image models. All reported numbers are verified to be reproducible via released scripts and are hardware-independent.