💻 computer science

Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos

This paper introduces **DroneEyes**, the first pixel-level open-vocabulary dataset for tiny aerial targets, and **SkyAnchor**, a memory-augmented multimodal large language model featuring a semantics-aware token router and hierarchical memory bank to enable real-time, resource-efficient understanding of small objects in streaming aerial videos.

Original authors: Penglei Sun, Yehua Huang, Zhuoli Tao, Xiang Li, Runwei Guan, Yaoxian Song, Kaiyong Zhao, Henghui Ding, Bo Han, Yang Yang, Xiaowen Chu

Published 2026-07-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Penglei Sun, Yehua Huang, Zhuoli Tao, Xiang Li, Runwei Guan, Yaoxian Song, Kaiyong Zhao, Henghui Ding, Bo Han, Yang Yang, Xiaowen Chu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the pilot of a tiny, high-tech drone flying over a bustling city. Your job isn't just to take pretty pictures; you have to answer questions from a human operator in real-time, like "Find that red car" or "Show me the white truck." This is the world of aerial vision, where computers try to understand what they see from the sky. But here's the catch: the things the drone sees are often incredibly small—like a single ant on a giant picnic blanket. To make this even harder, the drone is flying, so it can't wait to see the whole movie before answering; it has to react instantly as the video streams in, frame by frame. This is called streaming video understanding.

Now, imagine trying to describe that tiny ant to a friend while you are running a marathon. You can't carry a giant photo album of every step you've taken (that's too heavy), but if you forget where you were a second ago, you might lose the ant in the crowd. This is the exact problem researchers face with Multimodal Large Language Models (MLLMs). These are super-smart AI brains that can see and read, but they usually get overwhelmed by tiny details or forget the past when the video keeps rolling. This paper tackles the challenge of teaching these AI brains to spot tiny, moving objects in live drone footage without getting confused or running out of battery power.

The Tiny Target Problem

The researchers started by realizing that existing AI tools were terrible at this specific job. Most AI models are trained on videos of people and dogs in parks, where the subjects are big and easy to see. When you point an AI at a drone video, it tries to squish the whole image down to save space, treating a tiny car the same as a huge building. The result? The tiny car gets lost in the blur.

To fix this, the team first built a new training ground called DroneEyes. Think of this as a massive library of 2,140 high-definition drone videos, but with a special twist: every single tiny object in every single frame is outlined with pixel-perfect precision. They didn't just draw boxes around things; they traced the exact shape of a tiny silver car or a distant pedestrian. This dataset includes over 176,000 pairs of questions and answers, teaching the AI to not only find these tiny things but to describe them in detail, like "That's a red sculpture in a circular courtyard."

Meet SkyAnchor: The Drone's New Brain

With this new data, the team built a new AI model named SkyAnchor. If the old models were like a student trying to read a book while juggling, SkyAnchor is like a student with a super-organized notebook and a magic highlighter. It solves the two main headaches of drone vision with two clever tricks:

The Magic Highlighter (Semantics-Aware Token Router):
Normally, when an AI looks at a video, it breaks the image into thousands of tiny puzzle pieces (called tokens). It treats every piece equally, even the boring background sky. SkyAnchor uses a "router" that acts like a smart highlighter. It scans the image and says, "Hey, that patch of sky is boring, let's ignore it," but "That tiny car is important, let's keep all its details!" This allows the AI to focus its brainpower on the tiny targets without needing to process the whole massive image, saving energy and keeping the details sharp.
The Two-Notebook System (Hierarchical Memory Bank):
Since the drone is flying, the AI needs to remember what it saw a few seconds ago to know where the object is now. But it can't remember everything forever, or its memory would fill up instantly. SkyAnchor uses a two-layer memory system:
- The "Who" Notebook (Semantic Memory): This stores the identity of the object. It remembers, "That is a silver sedan." This part stays in the memory for a long time so the AI doesn't forget what it's tracking.
- The "Where" Notebook (Tracking Memory): This is a short-term, sliding window that only holds the last few seconds of movement. It remembers, "The car just moved to the left."
  By separating "who it is" from "where it is," SkyAnchor can track a target smoothly even as the drone zooms in and out, without getting confused or losing the object.

What They Found

The team tested SkyAnchor on their new DroneEyes dataset and compared it to the smartest AI models currently available. The results were impressive. On the task of describing objects, SkyAnchor scored twice as high as the best general-purpose models. When it came to finding and outlining the tiny objects (referring segmentation), it beat the next best model by 15.3%, even though SkyAnchor is much smaller and lighter.

But the real test was whether this AI could handle a completely new environment it had never seen before. The researchers threw it at a different dataset called SkyFind, which features maritime (ocean) scenes. Without any extra training, SkyAnchor improved the previous best results by 25.9% in finding objects accurately. This suggests the model learned a general skill for spotting tiny things, not just memorizing the training videos.

Real-World Flight

Finally, the team didn't just leave SkyAnchor in a computer lab. They put it on a real drone flying over a university campus. They optimized the software so it could run on a small, portable computer (an NVIDIA Jetson AGX Orin) without needing to send data to the cloud. The result? The system ran 3.05 times faster than a standard setup, allowing the drone to answer questions and draw outlines in real-time while flying.

In videos from the real world, SkyAnchor successfully tracked a red SUV, a silver car hidden in a gravel road, and even a koi fish in a pond, all while the drone was moving and the view was changing. It didn't get lost in the shadows or confused by similar-looking objects.

The Bottom Line

This paper suggests that by combining a new, high-quality dataset with a smart memory system and a focus-preserving router, we can make AI much better at spotting tiny things in live drone videos. It proves that you don't need a giant, heavy computer to do this; a streamlined, memory-aware model can fly on a drone and help operators find exactly what they are looking for, right now. While the paper shows strong results on specific datasets and real-world tests, it also hints that future work will focus on making the system even faster and teaching it to control the drone's flight path automatically to keep the target in view.

Technical Summary: Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos

1. Problem Statement

The paper addresses the challenge of online referring video object segmentation (RVOS) in streaming aerial videos captured by Unmanned Aerial Vehicles (UAVs). Unlike offline processing, UAVs must respond in real-time as frames arrive sequentially without access to future data. This setting presents two intrinsic challenges:

Tiny Target Perception: Aerial targets often occupy a minuscule portion of high-resolution frames (e.g., <0.24% of the image area). Existing Multimodal Large Language Models (MLLMs) typically apply uniform visual compression (e.g., fixed-ratio token merging), which treats small targets and large backgrounds equally, causing the loss of fine-grained details essential for small object identification.
Streaming Context vs. Resource Constraints: Understanding a continuous video stream requires temporal context to prevent target drift or disappearance. However, onboard hardware is resource-constrained, making it infeasible to cache the entire visual history of an ever-growing stream. Existing memory mechanisms (e.g., linear scaling in SAM 3) fail to sustain long-term identity or are too heavy for edge deployment.

Furthermore, the paper identifies a lack of suitable data: existing aerial datasets (e.g., SkyFind, AerialMind) rely on coarse bounding boxes or static images, while daily-life datasets (e.g., MeViS) focus on large objects. None provide pixel-level, open-vocabulary, per-frame segmentation masks for streaming aerial scenarios.

2. Methodology: SkyAnchor

The authors propose SkyAnchor, an MLLM framework designed specifically for small-object understanding in streaming aerial video. The system operates on a vision-language backbone (Qwen2.5-VL) and a mask decoder (SAM 2), enhanced by two core components:

A. Semantics-Aware Token Router

To address the "tiny target" challenge under a fixed visual token budget:

Mechanism: The router partitions dense spatial tokens into non-overlapping blocks. Instead of uniform merging, a lightweight scoring network evaluates the semantic salience of each token within a block.
Aggregation: Tokens are merged via softmax-weighted aggregation, assigning higher weights to target-relevant regions and lower weights to background.
Training: An auxiliary routing loss ( $L_{route}$ ), supervised by ground-truth masks, guides the router to preserve discriminative features of small objects while compressing background noise.

B. Hierarchical Memory Bank

To address the "streaming context" challenge while managing memory:

Semantic Memory Layer (Long-term): Stores persistent, object-level appearance features (anchors) extracted from key "anchor frames" (updated every $N$ frames). This layer preserves target identity (e.g., "silver sedan") to prevent long-term drift and aids the VLM in reasoning.
Tracking Memory Layer (Short-term): Maintains a Fixed-Size First-In-First-Out (FIFO) sliding window of recent spatial states (tracking memory). This layer supports short-term mask propagation between anchor frames using the SAM 2 decoder.
Integration: The two layers decouple temporal information, allowing the model to maintain stable understanding without storing the entire history.

C. Edge Deployment Strategy

The system employs an asynchronous inference schedule:

The heavy VLM backbone runs only on sparse anchor frames (e.g., every 16th frame) to update semantic anchors.
The lightweight SAM 2 decoder runs densely on every frame, utilizing cached tokens and the hierarchical memory bank to propagate masks.
Optimization includes TensorRT compilation, CUDA Graph capture for memory attention, and asynchronous scheduling to maximize throughput on edge devices (NVIDIA Jetson AGX Orin).

3. Key Contributions

Data: DroneEyes Dataset

The authors constructed DroneEyes, the first large-scale dataset for pixel-level, open-vocabulary referring segmentation in streaming aerial videos.

Scale: 2,140 high-definition drone videos (1080p–4K) and 176,623 QA pairs.
Granularity: Dense per-frame segmentation masks (pixel-level) rather than bounding boxes.
Task Levels:
- Level 1 (L1): Object Description (given a region, describe the object).
- Level 2 (L2): Referring Expression (given a text query, segment the object).
Characteristics: Features an extremely low average object-to-image ratio (0.24%), making it a rigorous benchmark for tiny object detection.

Method: SkyAnchor Framework

Introduction of the Semantics-Aware Token Router to preserve fine-grained details of tiny targets under compression.
Design of the Hierarchical Memory Bank to balance long-term identity preservation with short-term tracking efficiency in streaming scenarios.

Empirical Results

DroneEyes Performance: SkyAnchor (3B parameters) outperforms all existing general-purpose MLLMs and specialist segmentation models.
- L1 (Description): Achieves a 2× improvement in GPT scores over the strongest baselines.
- L2 (Segmentation): Achieves a J&F score of 38.10%, surpassing the previous best (Sa2VA-26B, 33.05%) by 15.3% despite using significantly fewer parameters.
- Small Object Robustness: Shows a marked advantage on ultra-tiny objects (relative area < $10^{-4}\%$ ), outperforming baselines by up to 10.8 percentage points.
Cross-Domain Generalization: Evaluated on the SkyFind benchmark without fine-tuning. On the unseen maritime test split, SkyAnchor improved IoU@0.5 by 25.9% and IoU@mean by 95.5% over the previous domain-specific SOTA (AerialREC), demonstrating strong robustness to domain shifts.
Real-World Deployment: Deployed on a UAV with an NVIDIA Jetson AGX Orin. The optimization pipeline achieved a 3.05× speedup over vanilla PyTorch, reaching 5.8 FPS (172ms latency), enabling cloud-free online monitoring.

4. Significance and Claims

The paper claims that SkyAnchor provides a practical solution for online aerial perception, bridging the gap between the high computational demands of MLLMs and the resource constraints of UAVs.

Data Contribution: It fills a critical void in the literature by providing the first dataset with pixel-level, open-vocabulary annotations for streaming aerial video, moving beyond the limitations of bounding-box-based datasets.
Methodological Insight: It demonstrates that simply scaling up MLLM parameters is insufficient for aerial small-object tasks; instead, specialized architectural designs (semantics-aware compression and hierarchical memory) are required to handle extreme foreground-background imbalance and streaming constraints.
Practical Viability: The successful deployment on real hardware validates that complex multimodal reasoning and segmentation can be performed onboard without cloud offloading, supporting applications in surveillance, search-and-rescue, and infrastructure inspection.

The authors conclude that future work will focus on further hardware-aware optimization to match native capture rates and exploring the integration of segmentation outputs for active camera control and autonomous navigation.