Technical Summary: RF-DETR: Neural Architecture Search for Real-Time Detection Transformers

Problem Statement

While open-vocabulary detectors (e.g., GroundingDINO, YOLO-World) achieve impressive zero-shot performance on standard benchmarks like COCO, they often fail to generalize to real-world datasets containing out-of-distribution classes or imaging modalities not present in their pre-training. Conversely, fine-tuning heavy-weight Vision-Language Models (VLMs) for specific domains improves in-domain performance but sacrifices runtime efficiency due to large text encoders.

Simultaneously, specialist (closed-vocabulary) real-time detectors like D-FINE and RT-DETR offer fast inference but often underperform compared to fine-tuned VLMs. The authors observe that recent specialist detectors are implicitly over-optimized for the COCO benchmark through bespoke model architectures, learning rate schedulers, and augmentation strategies, leading to poor generalization on real-world data distributions (e.g., varying object counts, class numbers, and dataset sizes). Furthermore, standard latency benchmarking across the field suffers from high variance and lack of reproducibility, often attributed to GPU power throttling and inconsistent evaluation protocols (e.g., omitting Non-Maximum Suppression or using different quantization levels for accuracy vs. latency).

Methodology

The paper introduces RF-DETR, a family of scheduler-free, real-time detection and segmentation transformers that utilize Weight-Sharing Neural Architecture Search (NAS) to discover accuracy-latency Pareto curves for any target dataset without re-training.

1. Architecture and Foundation

RF-DETR modernizes the LW-DETR architecture by:

• Backbone Replacement: Swapping the CAEv2 backbone for DINOv2, leveraging internet-scale pre-trained weights to improve generalization on small datasets.
• Scheduler-Free Training: Avoiding complex learning rate and augmentation schedulers that bias models toward specific dataset characteristics (e.g., avoiding aggressive vertical flips that cause false positives in safety-critical domains).
• Instance Segmentation: Adding a lightweight segmentation head that bilinearly interpolates encoder outputs to generate pixel embedding maps, allowing for end-to-end real-time segmentation (RF-DETR-Seg).

2. Weight-Sharing NAS and "Tunable Knobs"

Inspired by OFA, RF-DETR employs a weight-sharing NAS approach where a single base model is trained to support thousands of sub-network configurations. During training, the model uniformly samples random configurations. At inference, specific configurations are selected to target desired latency points. The search space includes the following "tunable knobs":

• Patch Size: Interpolated between sizes (e.g., 8 to 32) using FlexiVIT-style transformations.
• Decoder Layers: Regression loss is applied to all decoder layers, allowing any number of layers to be dropped at inference.
• Query Tokens: Queries are dropped at test time based on confidence scores to vary the maximum number of detections.
• Image Resolution: Positional embeddings are pre-allocated for the largest resolution and interpolated for smaller inputs.
• Windowed Attention: The number of windows per attention block is varied to balance global information mixing and computational efficiency.

This approach acts as a regularizer ("architecture augmentation"), allowing the model to generalize to unseen architectures and configurations not explicitly seen during training.

3. Standardized Latency Benchmarking

The authors identify that GPU power throttling during inference causes significant variance in reported latency. To address this, they propose a standardized evaluation protocol involving:

• Buffering: Inserting a 200ms pause between forward passes to prevent power throttling and ensure stable measurements.
• Artifact Consistency: Evaluating both accuracy and latency using the same model artifact (e.g., FP32 or FP16) to avoid discrepancies caused by naive quantization.
• NMS Inclusion: Ensuring Non-Maximum Suppression is included in latency measurements for fair comparison with end-to-end detectors.

Key Contributions

1. RF-DETR Model Family: A new family of real-time detectors and segmenters that outperform prior state-of-the-art methods on both COCO and the diverse RF100-VL benchmark. Notably, RF-DETR (2x-large) is the first real-time detector to surpass 60 AP on COCO.
2. Weight-Sharing NAS for DETRs: The first application of end-to-end weight-sharing NAS to object detection and segmentation, enabling the discovery of Pareto-optimal architectures for diverse hardware and datasets from a single training run.
3. Reproducible Benchmarking: A critique of current latency evaluation practices and a proposal for a standardized procedure (buffering, consistent artifacts) to improve reproducibility and fair comparison.

Experimental Results

COCO Benchmark

• RF-DETR (Nano): Achieves 48.0 AP on COCO, outperforming D-FINE (Nano) by 5.3 AP at similar latency (~2.3ms).
• RF-DETR (2x-large): Achieves 60.1 AP on COCO, setting a new record for real-time detectors.
• Segmentation: RF-DETR-Seg (Nano) outperforms YOLOv11-Seg (X-Large) while running 4x faster and beats FastInst by 5.4% AP with nearly 10x speedup.

RF100-VL Benchmark (Real-World Generalization)

• RF-DETR (2x-large): Outperforms GroundingDINO (Tiny) and LLMDet (Tiny) by 1.2 AP while running 20x faster.
• Generalization: Unlike YOLO variants which plateau or degrade when scaled, RF-DETR shows consistent improvements with larger backbone sizes (DINOv2-B) and maintains strong performance across the 100 diverse datasets in RF100-VL.
• Fine-Tuning: While fine-tuning RF-DETR on COCO yields limited gains (due to strong regularization from NAS), fine-tuning on RF100-VL provides modest improvements, suggesting the NAS "architecture augmentation" requires more epochs to converge on smaller, diverse datasets.

Ablation Studies

• Backbone: Replacing CAEv2 with DINOv2 improves performance by ~2%.
• Hyperparameters: Using "gentler" hyperparameters (lower learning rate, LayerNorm instead of BatchNorm) is necessary to preserve DINOv2's pre-trained knowledge and enable training on consumer hardware.
• Fixed Architecture: Transferring a COCO-optimized RF-DETR architecture to RF100-VL without re-searching still outperforms LW-DETR, but dataset-specific NAS yields further significant gains.

Significance and Claims

The paper claims that RF-DETR represents a shift from manually designing architectures for specific benchmarks to discovering architectures that generalize across diverse real-world distributions. By combining internet-scale pre-training (DINOv2) with weight-sharing NAS, RF-DETR achieves state-of-the-art performance without the computational cost of re-training for every new hardware target or dataset size.

The authors emphasize that current specialist detectors may be over-optimized for COCO, and that the community should benchmark on diverse datasets like RF100-VL to prevent implicit overfitting. Furthermore, the paper asserts that the lack of reproducibility in latency benchmarking is a critical issue that must be addressed through standardized protocols (buffering, consistent artifacts) to enable fair comparison of real-time models.

To the best of the authors' knowledge, RF-DETR is the first real-time detector to exceed 60 AP on COCO, demonstrating that end-to-end transformers can achieve both high accuracy and real-time inference when properly optimized via NAS.