AGVBench: A Reliability-Oriented Benchmark of Data Augmentation for Vein Recognition

Technical Summary: AGVBench: A Reliability-Oriented Benchmark of Data Augmentation for Vein Recognition

Problem Statement
Vein recognition is a secure biometric technology constrained by the "small-sample" dilemma inherent in biometric acquisition and the scarcity of annotated data. While data augmentation (DA) is the standard mitigation strategy, most existing approaches inherit policies optimized for natural images. The authors argue that these strategies often overlook the fundamental morphological differences between semantic objects and vascular structures. Aggressive transformations designed for natural images risk obliterating the delicate topological and high-frequency details essential for identity discrimination in vein patterns. Furthermore, the field lacks a standardized, comprehensive benchmark to evaluate how these augmentations affect biometric fidelity, calibration, and security across diverse neural architectures.

Methodology
The paper introduces AGVBench, a large-scale, open-source benchmark designed to systematically evaluate data augmentation strategies specifically for vein recognition. The framework is built on PyTorch and the MMCV ecosystem, featuring a modular architecture that decouples data pipelines, model backbones, augmentation modules, and evaluation scripts.

• Scope: The benchmark evaluates 30 representative augmentation strategies across five public datasets (three palm-vein: VERA220, TJU600, SCUT1100; two finger-vein: FV-USM, SDUMLA-HMT) and seven backbone architectures (including CNNs like ResNet18, MobileNetv2; Vision Transformers like ViT-S, Swin-T; and vein-specific models like FVRASNet, AMPVNet, StarLKNet-S).
• Augmentation Categories: The study covers Single-Image Augmentation (geometric, pixel-level, policy-driven), Multi-Image Augmentation (mixing methods like MixUp, CutMix, PuzzleMix), and Label Enhancement (smoothing, bootstrapping).
• Evaluation Protocol: Beyond standard Top-1 Accuracy, the benchmark employs a six-dimensional reliability-oriented evaluation:
  1. Recognition Performance: Accuracy, Equal Error Rate (EER), and True Acceptance Rate at FAR=0.0001 (TAR@FAR).
  2. Calibration: Expected Calibration Error (ECE) to measure confidence reliability.
  3. Corruption Robustness: Performance under 19 distinct image corruptions (including novel low-intensity levels C1/C2 tailored for vein fragility).
  4. Adversarial Robustness: Resistance to FGSM and PGD attacks.
  5. Occlusion Robustness: Performance under spatial masking.
  6. Efficiency: Training time, memory footprint, GFLOPs, and parameter overhead, summarized via an APEX Rank (Pareto efficiency) metric.

Key Contributions

1. First Large-Scale Benchmark: AGVBench is the first comprehensive framework to systematically evaluate augmentation strategies for vein recognition across multiple modalities and architectures.
2. Revealing Metric Decoupling: The study uncovers a critical inconsistency where methods achieving state-of-the-art clean accuracy (specifically multi-image mixing methods) often exhibit severe calibration errors and high vulnerability to adversarial perturbations.
3. Standardized Codebase: The authors release a unified, open-source library to facilitate reproducible research, standardizing preprocessing, training, and evaluation protocols to eliminate the trial-and-error currently prevalent in the field.

Key Results

• Performance vs. Robustness Trade-off: Multi-image mixing methods (e.g., MixUp, PuzzleMix, StarMixup) consistently dominate Top-1 Accuracy, EER, and TAR metrics, often significantly outperforming baselines (e.g., MixUp on VERA220 with ResNet18 achieved 95.27% accuracy vs. 71.45% baseline). However, these same methods are often poorly calibrated (high ECE) and highly vulnerable to adversarial attacks (e.g., MixUp accuracy dropped to 4.87% under PGD on TJU600).
• Geometric Transformations: Severe geometric transformations (Flip, Rotate, Translation) frequently degrade recognition performance across datasets, likely due to feature misalignment or the destruction of fine-grained vascular topology.
• Modality Dependence: Augmentation effectiveness varies significantly between palm and finger vein datasets. For instance, MixUp excels on palm veins but degrades performance on the SDUMLA-HMT finger vein dataset compared to the baseline.
• Label Enhancement: Label smoothing techniques (e.g., LabelSmoothing, DirichletLabelSmooth) demonstrated the strongest adversarial robustness, often doubling the baseline's resistance to attacks, though they sometimes lagged in raw accuracy compared to mixing methods.
• Efficiency: Methods like MixUp and StarMixup offered the best balance between performance gains and computational overhead, whereas policy-based methods like TeachAugment incurred high training costs with marginal or negative returns.

Significance
The paper argues that accuracy-centric evaluation is insufficient for biometric systems. AGVBench demonstrates that high clean accuracy does not equate to reliability or security. By revealing the decoupling between recognition performance, calibration, and adversarial robustness, the benchmark guides the design of future vein recognition systems that must be not only accurate but also secure and reliable in real-world deployment. The authors posit that AGVBench serves as a foundational resource to bridge the gap between theoretical research and practical implementation, fostering the development of robust, vein-specific augmentation pipelines.