Technical Summary: AnyDepth-DETR/-YOLO

Problem Statement

Modern object detectors are typically static, fixed-depth networks optimized for a single operating point. This rigidity necessitates training separate models for different deployment scenarios (e.g., varying resource budgets or latency requirements), making it difficult for a single network to satisfy heterogeneous demands. While adaptive computation methods like early exiting and width adaptation have proven effective in image classification, they transfer poorly to object detection. Unlike classifiers that produce a single global output, detectors rely on a tightly coupled backbone–neck–head pipeline dependent on hierarchical multi-scale features. Naively skipping stages or pruning channels breaks these dependencies, degrading feature quality and localization accuracy. Prior work has not achieved depth-controlled computation in object detection while preserving the full multi-scale feature hierarchy required for robust detection.

Methodology

The authors propose AnyDepth, a framework enabling a single network to span a continuous range of accuracy–efficiency trade-offs by controlling network depth at inference time without retraining. The core design principle is stage-wise modularity: ensuring that the output of a lighter sub-network is interchangeable with that of a fuller sub-network at every stage boundary.

1. Architectural Decomposition

Each stage in both the backbone and neck is divided into two paths:

• Essential Path: Always executes, learning the core feature representation at the given scale.
• Refinement Path: Skippable, providing further refinement when additional computation is available.

This decomposition preserves all hierarchical stages at every depth configuration, unlike conventional early exiting which discards entire stages.

• Residual Stages (e.g., RT-DETR): The refinement path is purely additive ($x_{full} = x_{ess} + \Delta$), sharing tensor shapes and avoiding architectural conflicts.
• CSP-Style Stages (e.g., YOLO): Since concatenation width depends on the number of executed blocks, the authors introduce a switchable aggregation layer. A dedicated weight matrix is used for the essential path when the refinement path is skipped, fully decoupling its gradient stream from the full-path aggregator to prevent conflicting gradients.
• Switchable Batch Normalization: Essential path blocks utilize two independent BN layers (one for full-path execution, one for essential-only) to adjust output distributions appropriately for each mode, ensuring compatibility at stage boundaries.

2. Training Strategy: Self-Distillation

Jointly optimizing all possible sub-networks introduces conflicting gradient signals. To address this, the framework trains only the two extremes:

• Super-net: Executes all full paths.
• Base-net: Executes only essential paths.

These are trained jointly via self-distillation, where the super-net acts as an online teacher for the base-net. The loss function combines ground-truth supervision with distillation losses:

• Prediction-Level Alignment: KL divergence for classification logits and IoU/edge loss for regression boxes.
• Feature-Level Alignment: An explicit loss aligns Global Average Pooled (GAP) channel descriptors between the essential and full paths at every stage boundary, enforcing that downstream stages receive compatible inputs regardless of the path taken.

Target Assignment Conflict Resolution:
A critical challenge is that the super-net and base-net may assign different anchors/queries to the same ground-truth object due to their differing predictions.

• For AnyDepth-DETR: The authors reorder the super-net's foreground queries to align with the base-net's target-matched ordering before applying Knowledge Distillation (KD).
• For AnyDepth-YOLO: Conflicting anchor assignments (where the same anchor is assigned to different GT objects by the two networks) are identified and excluded from the distillation set, restricting KD to unambiguous targets.

Key Contributions

1. Any-Depth Framework: A novel architecture and training method enabling a single model to dynamically adjust depth at inference, covering a continuous accuracy–efficiency spectrum.
2. Stage-Wise Modularity: A design principle ensuring feature compatibility across depth configurations, achieved through essential/refinement path decomposition and specific architectural adaptations (switchable aggregation and BN).
3. Self-Distillation with Conflict Resolution: A training strategy that optimizes only the extreme sub-networks while enforcing modularity through feature and prediction alignment, specifically addressing target assignment mismatches inherent in DETR and YOLO families.
4. First Application to SOTA Real-Time Detectors: The framework is instantiated on RT-DETR and YOLOv12, demonstrating that effective self-distillation is achievable for these complex architectures.

Experimental Results

Evaluated on the COCO benchmark:

• Full-Depth Performance: The super-net configurations match or surpass their respective state-of-the-art (SOTA) baselines with negligible parameter overhead (e.g., <0.09% for RT-DETR, ~3% for YOLOv12).
• Efficiency Gains:
  • AnyDepth-DETR (R-101): Achieves a 1.82× speedup at the cost of only 2.0 AP drop compared to the full model.
  • AnyDepth-YOLO (L): Achieves a 1.57× speedup with only a 1.8 AP drop.
• Generalization: Intermediate sub-networks (mixing essential and full paths) generalize well without direct supervision, offering smooth trade-offs. For instance, reducing encoder depth in AnyDepth-DETR trades 0.9 AP for a 6% FPS gain, while routing all backbone stages through the essential path costs 2.0 AP for a 15% FPS improvement.
• Pareto Frontier: A single AnyDepth model covers the accuracy–efficiency range previously requiring multiple separately trained static models (e.g., covering the range of RT-DETR R-18 to R-101 with one network).

Significance and Claims

The paper claims that AnyDepth provides a practical solution for deploying detectors under dynamic resource constraints, replacing the need for multiple fixed-size detectors with a single modular network. By preserving the full multi-scale feature hierarchy at every depth configuration, the method overcomes the limitations of prior adaptive inference techniques in detection tasks.

The authors emphasize that their approach enables truly adaptive real-time object detection, where depth configurations can be selected at runtime based on resource budgets without retraining. They note that the refinement path primarily improves localization precision rather than detection coverage, validating the modularity of the design.

Limitations Acknowledged:

1. The self-distillation training procedure requires two sequential forward-backward passes per iteration, effectively doubling training time compared to standard models.
2. The current framework selects depth based on a fixed resource budget rather than adapting to input difficulty; integrating a lightweight routing mechanism for input-adaptive selection is identified as future work.