Trajectory-Aware Adaptive Inference in Object Detection Models

Technical Summary: Trajectory-Aware Adaptive Inference in Object Detection Models

Problem Statement
The integration of sensors in autonomous maritime navigation has generated large-scale multimodal datasets, creating a demand for efficient real-time perception. While object detection and trajectory perception are tightly coupled in dynamic environments like maritime navigation, the inference efficiency of object detection models is often overlooked. Standard detectors, such as the YOLO family, apply uniform computational effort to all inputs regardless of scene difficulty. This static approach is suboptimal for resource-constrained environments (e.g., Autonomous Surface Vehicles or ASVs) where latency and energy efficiency are critical. Existing adaptive inference methods, such as multi-exit neural networks, typically rely on internal model confidence signals (e.g., entropy or softmax probability) rather than external contextual data. Consequently, there is a need for an inference strategy that leverages external trajectory data to dynamically adjust computational load based on scene complexity.

Methodology
The authors propose a Trajectory-Aware Adaptive Inference Framework that integrates GPS trajectory data into the inference process of a YOLOv8-based detector. The methodology consists of three primary stages:

1. Detector Adaptation and Scale-Aware Fine-Tuning:

   • The base YOLOv8 model is fine-tuned on a maritime dataset.
   • Recognizing that YOLOv8 possesses an implicit multi-exit structure via its three detection heads (P3, P4, P5) operating at different scales, the authors treat these heads as potential exit points.
   • A scale-aware learning rate strategy is employed. Bounding boxes in the training data are categorized by size (small, medium, large). Learning rates are assigned to each detection head based on the frequency and difficulty of the corresponding object sizes in the dataset. Heads associated with over-represented or challenging object sizes receive higher learning rates to ensure appropriate specialization.

2. Trajectory-Based Difficulty Estimation:

   • Instead of relying on internal confidence, the system utilizes external motion cues derived from synchronized GPS data.
   • Two key metrics are calculated for each frame (or one-second window):
     • Inter-vessel distance ($d_t$): Computed using the Haversine formula.
     • Closure rate ($v_t$): The rate at which the distance between moving objects decreases over time.
   • These metrics serve as proxies for "scene difficulty." Scenarios with closely spaced vessels and high convergence rates are deemed complex, while well-separated, slowly moving scenarios are considered simple.

3. Adaptive Inference Control (Early-Exit Policy):

   • A conditional policy determines which detection heads to activate:
     • Low Complexity: If $d_t > \tau_1$ (e.g., 30m) AND $v_t < \tau_2$ (e.g., 0.5 m/s), only the high-resolution P3 head is activated.
     • High Complexity: Otherwise, the full model (P3, P4, and P5 heads) is executed.
   • This approach allows the system to bypass computationally expensive parts of the network when the trajectory data indicates a low-risk, low-complexity environment.

Key Contributions

• Multimodal Adaptive Inference: The integration of mobility data (GPS trajectories) with computer vision to guide computational resource allocation, moving beyond confidence-based early exiting.
• Scale-Aware Fine-Tuning: A methodology for modulating learning rates per detection head based on the geometric distribution of object sizes in the target domain, optimizing the heads for independent operation.
• Empirical Validation: A comprehensive evaluation of the proposed YOLOv8 model, providing insights into per-head latency-accuracy trade-offs and demonstrating the efficacy of trajectory-driven inference control.

Experimental Results
The framework was evaluated using a dataset comprising synchronized video from onboard ASV cameras and GPS trajectory data from the same platforms (250 GPS records, 3,686 video frames).

• Head Performance: Independent evaluation of detection heads revealed that the P4 head (medium scale) dominates detection counts and achieves the highest standalone performance (mAP@50) across model variants (Nano, Small, Medium). The P3 head showed increased sensitivity to small objects in larger models.
• Computational Efficiency: Activating individual heads yielded significant FLOPs savings (ranging from ~18% to ~33% depending on the head and model size) and speedups (up to 1.61x for P3 in the Nano model).
• Adaptive Strategy Performance: In a realistic maritime scenario with thresholds set at 30m distance and 0.5 m/s closure rate:
  • 3,027 out of 3,686 frames (approx. 82%) were processed using only the P3 head.
  • Median inference latency dropped from 10.097 ms/frame (full model) to 6.686 ms/frame (P3 only).
  • Total processing time for the video sequence decreased from a projected 37.22 seconds to 26.89 seconds.
  • The strategy maintained effective tracking performance, successfully balancing reduced latency with acceptable detection accuracy.

Significance and Claims
The paper claims that the proposed framework offers a flexible trade-off between accuracy and efficiency, making it particularly suitable for real-time, resource-constrained applications like autonomous maritime navigation. By leveraging external trajectory cues, the system achieves input-adaptive computation without requiring complex internal confidence modeling. The authors note that while the approach significantly lowers processing overhead and latency, it preserves predictive performance in dynamic environments.

Future Work
The authors suggest future directions including:

• Generalizing the method to alternative architectures (e.g., DETR).
• Testing robustness under challenging conditions (adverse weather, sensor noise, low visibility).
• Incorporating explicit mappings between GPS coordinates and image coordinates to further enhance the fusion of trajectory and visual information.