Structured Evidence and Vision-Language Models for Interpretable Vision-Only UAV Behavior Analysis

Technical Summary: Structured Evidence and Vision-Language Models for Interpretable Vision-Only UAV Behavior Analysis

Problem Statement
Vision-only counter-unmanned aerial vehicle (counter-UAV) systems offer significant advantages in cost, passive sensing, and compatibility with existing camera infrastructure. However, current pipelines are largely limited to detection and tracking, providing operators with target locations but lacking auditable evidence regarding the UAV's behavior or the urgency of a required response. While operators can see where a UAV is, they cannot easily determine what it is doing (e.g., benign transit, surveillance, approach, or irregular maneuvering) without slow, cognitively demanding manual video review. Existing vision-based pipelines fail to bridge the gap between low-level visual tracking and high-level behavior interpretation.

Methodology
The authors propose a four-layer framework designed to introduce an interpretable evidence layer between a standard tracker and a Vision-Language Model (VLM). The system is strictly vision-only and operates as follows:

1. Detection Layer: A single-class YOLOv8 detector (specifically the YOLOv8s model trained at 1280-pixel resolution) identifies UAVs in each frame.
2. Tracking and Merging: Detections are associated into trajectories using ByteTrack, which leverages low-confidence detections to maintain robustness. To address trajectory fragmentation caused by occlusion or confidence drops, a greedy track-merging algorithm combines short segments into coherent paths based on temporal gaps, spatial distance, and motion continuity.
3. Structured Evidence Extraction: This is the core interpretability component. Each merged trajectory is converted into a fixed-template evidence record containing:
   • Motion Statistics: Average/max speed, speed standard deviation, total displacement, path length, and linearity (ratio of displacement to path length).
   • Scale Features: Bounding box area start, end, and change ratio (proxy for approach/departure).
   • Curvature: Mean and maximum turning angles.
   • Hovering Metrics: Fraction of frames with near-stationary motion.
   • Rule-Based Suggestions: A lightweight rule engine generates preliminary behavior labels (e.g., hovering, circling, cruising, approaching) based on thresholds applied to these metrics.
4. VLM Reasoning: The fourth layer prompts a VLM (Qwen-VL-Max) with a multimodal input consisting of:
   • A 2×3 key-frame mosaic (six uniformly sampled frames).
   • The structured evidence record (serialized text of numeric features and rule suggestions).
   • The model is instructed to output behavior labels, response-urgency estimates, and natural-language rationales grounded in the specific evidence values, formatted as a JSON block.

Key Contributions
The paper's primary contribution is not a novel detector or tracker, but rather the demonstration of how an interpretable evidence interface can transform a conventional detection-tracking pipeline into an auditable behavior-analysis system.

• Structured Evidence over Raw Pixels: The framework explicitly decouples perception from reasoning. Instead of asking the VLM to infer complex temporal dynamics solely from sparse static frames, it provides compact, numerical trajectory facts.
• Auditability: By forcing the VLM to cite specific evidence values (e.g., linearity, scale-change ratio) in its rationale, the system makes the reasoning process inspectable, unlike end-to-end visual prompting where the basis for a label is opaque.
• Hybrid Approach: The system combines rule-based heuristics with the semantic synthesis capabilities of a VLM, allowing the model to refine or override preliminary rule-based suggestions when necessary.

Experimental Results
The framework was evaluated on 20 sequences from the DUT Anti-UAV dataset, yielding 52 valid tracks. A subset of 50 tracks was manually annotated for behavior evaluation across six categories (hovering, circling, cruising, approaching, departing, erratic).

• Performance: The full multimodal configuration (image + structured evidence) achieved the best primary-match accuracy (0.700) and any-match accuracy (0.760).
• Ablation Findings:
  • Image-Only: A variant using only the key-frame mosaic performed poorly (0.220 primary-match accuracy), confirming that sparse static frames cannot reliably infer temporal behavior.
  • Evidence-Only: A variant using only structured evidence achieved 0.680 primary-match accuracy, demonstrating that trajectory statistics are the dominant information source.
  • Comparison: The full configuration outperformed both single-modality variants and the rule engine alone. Specifically, the visual frames provided a measurable gain in geometrically ambiguous cases (e.g., improving "cruising" accuracy from 0.57 to 0.71 compared to evidence-only).
• Class-Specific Performance: The system performed well on "approaching" and "departing" behaviors. "Hovering" remained the most difficult class, often confused with circling when the hovering ratio was not dominant.

Significance and Claims
The paper claims that for vision-only counter-UAV behavior analysis, structured trajectory evidence is the essential information source, while visual frames provide a supplementary but useful gain in ambiguous scenarios. The authors assert that the proposed framework successfully turns low-level tracks into auditable behavior labels and response-oriented explanations.

The work emphasizes that VLMs are effective in this domain only when grounded in explicit trajectory evidence rather than being asked to infer temporal behavior solely from sparse static images. The system offers a practical path toward scalable, interpretable counter-UAV operations where operators can verify the logic behind an alert. The authors remain modest regarding the scope, noting limitations such as the small size of the annotated subset (50 tracks), the use of a single annotator, and the reliance on a limited number of key frames for the image baseline. Future work is suggested to expand the dataset, involve multiple annotators, and explore denser temporal visual inputs.