Le-DETR: Revisiting Real-Time Detection Transformer with Efficient Encoder Design

Le-DETR introduces an efficient backbone (EfficientNAT) and a redesigned hybrid encoder to achieve state-of-the-art real-time object detection performance with significantly reduced pre-training costs, eliminating the need for expensive pre-training while outperforming leading models like YOLOv12 and DEIM-D-FINE in both accuracy and speed.

The Gist

Imagine you are trying to build a super-fast security guard for a busy airport. This guard needs to spot every single person, bag, or suspicious item instantly (low latency) while being incredibly accurate (high accuracy).

For a long time, the best guards were built using two different blueprints:

1. The "YOLO" Guard: Built with traditional, fast-moving bricks (Convolutional Neural Networks). They are fast but sometimes miss subtle details.
2. The "DETR" Guard: Built with a futuristic, all-seeing AI brain (Transformers). They are incredibly smart and accurate but usually very slow to train and require a massive library of books (data) to learn before they can work.

The Problem: The "Expensive Training School"

The paper points out a major headache with the smart "DETR" guards. To make them fast enough for real-time use, previous researchers had to send them to a super-expensive, exclusive training school.

• The Cost: They needed to study 4 million extra images (on top of the standard 1 million) just to get the basics right.
• The Bottleneck: This "school" was so expensive and complex that regular researchers couldn't afford to send their own guards there. It locked innovation behind a paywall of data and computing power. It was like saying, "You can only build a fast car if you first buy a private island to test it on."

The Solution: Le-DETR (The "Smart Local" Guard)

The authors of this paper, Le-DETR, asked a simple question: "Do we really need that expensive school, or did we just build the guard's brain inefficiently?"

They realized the previous designs were trying to use a "global" approach (looking at the whole world at once), which is slow and data-hungry. Instead, they designed a new guard that uses Local Attention.

The Analogy: The Library vs. The Neighborhood

• Old Method (Global Attention): Imagine the guard has to read every single book in a massive library to find one specific fact. It takes forever and requires a huge library (4 million images).
• New Method (Local Attention / Le-DETR): Imagine the guard only needs to look at the neighborhood right next to them. If they are looking for a red hat, they only scan the people within 5 feet. This is much faster, requires less memory, and is actually more accurate for spotting specific things in a crowd.

How They Did It (The "Secret Sauce")

The team built a new engine for their guard called EfficientNAT. Think of it as a hybrid car engine that combines the best of two worlds:

1. Efficient Convolution: Fast, reliable gears for moving around.
2. Neighborhood Attention: A smart radar that only scans the immediate area, ignoring the rest of the world to save energy.

They also redesigned the "decoder" (the part that makes the final decision). Instead of a slow, heavy process, they streamlined it so the guard can make decisions almost instantly.

The Results: Faster, Smarter, Cheaper

The results are like finding a Ferrari that runs on regular unleaded gas instead of rocket fuel.

• Training Cost: They cut the training data requirement by 80%. They only used the standard 1 million images (ImageNet) instead of the massive 5 million required by others. This means anyone can now reproduce their results without needing a supercomputer farm.
• Performance:
  • Their medium-sized model (Le-DETR-M) is faster and more accurate than the current champions (like YOLOv12 and D-FINE).
  • It can spot objects in 4.45 milliseconds (that's faster than a human eye blink) with incredible accuracy.
  • It beats the previous "smart" models (DETRs) while being significantly faster.

Why This Matters

Before this paper, if you wanted the best real-time object detection, you had to be a giant tech company with deep pockets to pay for the massive pre-training.

Le-DETR democratizes the technology. It proves that you don't need a "super-school" to build a super-guard. You just need a better architectural design. It's like showing that you can build a faster car not by buying a bigger engine, but by making the engine run more efficiently.

In short: They took a slow, expensive, data-hungry AI, gave it a "local neighborhood" focus, and turned it into the fastest, most efficient real-time detector on the market, all while saving 80% of the training cost.

Technical Summary

1. Problem Statement

Current state-of-the-art (SOTA) real-time object detection models based on the Detection Transformer (DETR) architecture, such as RT-DETR and D-FINE, face a critical bottleneck: excessive pre-training overheads.

• High Cost: These models rely on backbones pre-trained on massive datasets (e.g., 4 million additional unlabeled images on top of ImageNet-1K) and complex knowledge distillation (KD) schedules.
• Reproducibility Barrier: The reliance on proprietary or non-open-source pre-training pipelines (e.g., specific PaddlePaddle variants like PResNet50 vd ssld) hinders the research community from reproducing results, experimenting with novel backbone architectures, or improving models from scratch.
• Sub-optimal Design: The authors hypothesize that the massive pre-training cost is often a compensation for sub-optimal architectural designs rather than a fundamental requirement for high performance. Furthermore, existing real-time DETRs underutilize modern efficient attention mechanisms, relying heavily on traditional CNN-style blocks (FPN-PAN, RepVGG).

2. Methodology

The paper proposes Le-DETR (Low-cost and Efficient DEtection TRansformer), a framework designed to achieve SOTA performance using only standard ImageNet-1K pre-training (approx. 1M images) by re-engineering the encoder and backbone.

A. EfficientNAT Backbone

Instead of using standard ResNet variants or proprietary backbones, the authors design a novel backbone called EfficientNAT:

• Architecture: It integrates modern efficient convolutions with Neighborhood Attention (NA).
• Stages 1-3: Utilize depthwise separable convolutions (DSConv) and Fused Mobile Convolutions (FusedMBConv) to extract features efficiently.
• Stage 4 (Final Stage): Introduces the EfficientNATBlock, which combines Neighborhood Attention with an MBConv-based Feed-Forward Network (FFN). This allows the model to capture robust local features without the quadratic complexity of global self-attention.
• Scaling Strategy: The authors conducted a systematic study on scaling the backbone (varying block counts in stages 3 and 4). They found that:
  • Small/Medium scales (M, L): Benefit from a balanced distribution of blocks (PA pattern).
  • Large scale (X): Benefits from an early-stage heavy distribution (PC pattern), placing more blocks in the third stage.

B. Hybrid Encoder with NAIFI

The authors redesigned the hybrid encoder to leverage local attention:

• NAIFI Module: They introduced the Neighborhood Attention-based Improved Feature Inference (NAIFI) module. Unlike standard self-attention (AIFI) used in previous DETRs, NAIFI uses local neighborhood attention to optimize feature representation.
• Benefits: This reduces computational overhead and inference latency while improving feature extraction quality.
• Decoder Optimization: The model utilizes a 5-layer decoder (trained with 6 layers) and incorporates training enhancements like Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD). It also employs Flash Attention to accelerate self-attention inference.

3. Key Contributions

1. Identification of Pre-training Dependency: The paper highlights that the heavy pre-training requirements of current real-time DETRs are not strictly necessary for high performance. They demonstrate that SOTA results can be achieved with only ImageNet-1K pre-training, saving ~80% of the pre-training image data.
2. EfficientNAT Backbone: A novel backbone architecture that balances latency and performance by integrating Neighborhood Attention with efficient convolutions, specifically tailored for real-time detection.
3. NAIFI Encoder Redesign: A shift from global self-attention to local neighborhood attention in the encoder, proving that local attention is highly effective for real-time detection tasks, reducing latency and improving accuracy.
4. Reproducibility: By removing the dependency on massive, proprietary pre-training datasets and complex distillation pipelines, Le-DETR significantly lowers the barrier to entry for future research in real-time DETR architectures.

4. Experimental Results

Experiments were conducted on the COCO Val2017 dataset using an RTX 4090 GPU.

• Performance vs. YOLO Series:
  • Le-DETR-M: 52.9 mAP @ 4.45 ms (Outperforms YOLOv12-L by +0.6 mAP with similar speed).
  • Le-DETR-L: 54.3 mAP @ 5.01 ms.
  • Le-DETR-X: 55.1 mAP @ 6.68 ms (Outperforms YOLOv12-X by +0.1 mAP while being 20% faster).
• Performance vs. Real-Time DETR (RT-DETR, D-FINE):
  • Le-DETR-L: Surpasses RT-DETRv2-L by +0.9 mAP (and +2.7 mAP if comparing without KD/extra data) with a 9% speedup.
  • Le-DETR-M: Outperforms DEIM-D-FINE-M by +0.2 mAP with slightly faster inference.
  • Le-DETR-X: Surpasses DEIM-D-FINE-L by +0.4 mAP with only 0.4 ms additional latency.
• Ablation Studies:
  • Replacing EfficientNAT with ResNet50 vd ssld resulted in a 0.7 mAP drop and 0.79 ms latency increase.
  • Replacing NAIFI with standard AIFI resulted in a 0.2 mAP drop and 0.17 ms latency increase.

5. Significance

• Paradigm Shift: Le-DETR challenges the notion that real-time DETRs require massive, expensive pre-training pipelines. It proves that architectural efficiency (better backbone and encoder design) can compensate for reduced training data.
• Community Impact: By open-sourcing the code and weights trained only on ImageNet-1K and COCO, the authors enable the community to reproduce, verify, and build upon these results without access to proprietary datasets.
• Future Direction: The work suggests that future research should focus on optimizing local attention mechanisms and backbone structures rather than scaling up pre-training data, leading to more accessible and efficient real-time detection systems.