💻 computer science

Moving Like a Human: Ego-Motion-Normalized Temporal Signatures for Real-Time Aerial Person Tracking on Milliwatt-Class Hardware

This paper introduces EMTS-Det, an ultra-lightweight, real-time aerial person tracking system that achieves high accuracy on milliwatt-class hardware by replacing learned temporal processing with analytically computed, ego-motion-normalized motion signatures, enabling robust detection and tracking on devices like the Raspberry Pi Zero 2W where traditional models fail.

Original authors: Akbar Anbar Jafari, Cagri Ozcinar, Gholamreza Anbarjafari

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Akbar Anbar Jafari, Cagri Ozcinar, Gholamreza Anbarjafari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Invisible Tracker: How Drones Learn to Follow Without Getting Dizzy

Imagine you are trying to follow a friend through a crowded, chaotic dance floor while standing on a wobbly, spinning platform. If you just look at a single snapshot of the room, your friend might look like a tiny, blurry smudge, indistinguishable from a pile of coats or a shadow. This is the daily struggle for drones trying to "follow-me" a person. They fly on tiny, battery-powered computers that are incredibly weak compared to the supercomputers in our phones, and the wind makes the drone shake violently. To make matters worse, the person they are chasing is often so far away that they only take up a few pixels on the camera screen—too small for standard software to recognize just by looking at their clothes or face.

The solution isn't to build a smarter brain; it's to change how the drone sees the world. Instead of trying to memorize what a person looks like, this paper suggests teaching the drone to notice how a person moves. It's the difference between trying to identify a runner by their shirt color (which changes with the wind and lighting) versus noticing that their legs are moving in a rhythmic, walking pattern while the background scenery rushes past. By mathematically canceling out the drone's own shaking and spinning first, the system turns a confusing, blurry video into a clear signal that says, "Hey, that thing is moving on its own." This allows even the tiniest, cheapest computer chips to track a human in real-time, something that was previously thought impossible on such limited hardware.

The Paper: "Moving Like a Human"

The researchers behind this paper, working with drones that have very limited computing power (specifically "milliwatt-class" hardware like the Raspberry Pi Zero 2W), set out to solve a tricky problem: how to make a drone follow a person without getting lost or needing a supercomputer. They found that trying to use standard, heavy-duty AI detectors (like the famous YOLOv8n) on these tiny drones was a recipe for disaster. Those standard detectors are too slow, running at a sluggish 1.95 frames per second, and they fail miserably when the person is small and the camera is shaking.

Instead, they built a new system called EMTS-Det. Think of this system as a five-step detective team that works together to spot a human:

The Stabilizer (Stage A): First, the system figures out exactly how the drone is moving and spinning. It's like a dancer who knows exactly how much they are twirling so they can ignore that motion.
The Motion Filter (Stage B): Next, it takes the video and mathematically "subtracts" the drone's movement. If the drone spins left, the image spins right to cancel it out. What's left? Only the things that are moving on their own. In this filtered view, a walking person pops out as a bright, glowing blob, while stationary rocks or swaying trees disappear.
The Tiny Detective (Stage C): This is the main AI brain, but it's incredibly small—only 22,000 parameters (compared to millions in standard detectors). Because the "motion blob" is already so clear, this tiny brain doesn't need to work hard to guess what the object is. It just spots the center of the moving blob.
The Tracker (Stage D): Once it finds the person, it locks on using a mathematical tool called a Kalman filter. This acts like a mental prediction, guessing where the person will be next even if the wind blows the drone around or the person briefly disappears behind a bush.
The Gait Inspector (Stage E): Finally, to make sure it hasn't accidentally locked onto a swaying tree branch or a cloud shadow, a small "verifier" checks the movement pattern. It asks, "Does this object move like a human walking?" If the answer is no, it rejects the target.

What They Found (and What They Rejected)

The results were impressive. On a Raspberry Pi Zero 2W, their system ran at 31.85 frames per second with high accuracy, while the standard YOLOv8n detector crawled along at 1.95 frames per second. In a real-world test with 1,000 drone videos, their system successfully tracked people with a 0.462 accuracy score (AP25), whereas the standard detector only managed 0.172.

Crucially, the paper explicitly ruled out several ideas that many people might assume would help:

They rejected "Learning" the motion: The team tested adding complex "temporal shift" modules (AI that learns to look at past and future frames to understand motion). They found that these actually made the system worse. Because the motion was already cleaned up mathematically in Stage B, the AI didn't need to learn it; trying to learn it just wasted energy and confused the system.
They rejected "Appearance" as the main clue: When they tried to train the system using only the visual look of the person (ignoring the motion data), the system collapsed. On real-world footage, the "appearance-only" version failed completely, proving that for tiny, distant targets, how a person looks is not enough; you need to know how they move.
They rejected standard calibration: They discovered that the usual way of setting up these tiny AI chips (using a "moving average" to guess the range of numbers) caused the system to fail silently. By switching to a "min-max" method, they fixed the accuracy, which had been cut in half by the standard method.

The Bottom Line

The paper demonstrates that you don't need a massive, expensive computer to track a person from the sky. By using simple geometry to remove the drone's own shaking first, and then letting a tiny, specialized AI look for the specific "signature" of human walking, you can achieve real-time tracking on hardware that costs very little. The system successfully locked onto a target in just 1.3 seconds during a field test and maintained a 97.9% success rate in keeping the lock, even when the person was hidden behind obstacles.

The authors are careful to note that while their system is a major step forward, it still has limits. For instance, if a person stands perfectly still for a long time, the system might lose them because it relies on motion. However, for the specific job of "follow-me" drones, where the target is usually moving, this approach of "geometry first, learning second" proves to be a powerful and efficient solution.

Technical Summary: EMTS-Det for Real-Time Aerial Person Tracking

Problem Statement
The paper addresses the challenge of "follow-me" person tracking on consumer and industrial drones, specifically targeting hardware with milliwatt-class compute budgets (e.g., Raspberry Pi Zero 2W). The problem is defined by three simultaneous constraints:

Small Target Size: At typical follow distances, a person occupies only 10–60 pixels in a 640×512 frame, rendering single-frame appearance features ambiguous and indistinguishable from background clutter.
Aggressive Ego-Motion: Sub-250g drone airframes experience violent motion due to wind gusts that outpace consumer gimbals, causing rapid camera attitude changes.
Severe Compute Limits: Companion computers in this class offer only 2–4 effective int8 GFLOP/s. Standard detectors like YOLOv8n (approx. 8.7 GFLOPs) run at ~2 FPS, while faster, smaller models (e.g., MobileNet-SSD) fail to detect small objects accurately.

The authors argue that the missing evidence for detection is not appearance but temporal motion, which must be extracted analytically rather than learned, as the available FLOP budget cannot support learning both ego-motion invariance and small-object appearance simultaneously.

Methodology: EMTS-Det Pipeline
The proposed system, EMTS-Det (Ego-Motion-normalized Temporal Signature Detection), is a five-stage pipeline designed to fit within a ~100 MFLOPs/frame budget.

Stage A: Ego-Motion Estimation: Estimates inter-frame camera motion as a 4-DoF similarity transform (translation, rotation, scale) using sparse Lucas-Kanade tracks with RANSAC. It falls back to phase correlation for texture-poor scenes.
Stage B: Input Representation: The previous frame is warped by the estimated transform to create three input channels:
1. Luminance ( $L_t$ ): The current grayscale frame.
2. Residual Motion Magnitude ( $R_t$ ): The magnitude of the difference between the current frame and the warped previous frame (Gaussian smoothed).
3. Signed Temporal Difference ( $D_t$ ): The raw difference between current and warped previous frames.
  Crucially, this representation analytically removes ego-motion, ensuring that bright pixels in the motion channels correspond only to independently moving objects.
Stage C: Micro-Detector: A stateless, anchor-free detector with 21,941 parameters (7.6 MFLOPs/frame). It uses a depthwise-separable backbone to output a center heatmap, box geometry, and an 8-D re-identification embedding. The network is deliberately stateless; temporal evidence is provided entirely by the input channels, not learned temporal layers.
Stage D: Target-Locked Tracking: A constant-velocity Kalman filter operates in stabilized coordinates (transformed by the accumulated ego-motion). This ensures that camera shake perturbs the coordinate transform rather than the target's motion model. Association uses a two-key gate: Mahalanobis distance and embedding similarity.
Stage E: Temporal-Signature Verification: An 8,289-parameter 1-D CNN verifies tracks every 8th frame. It analyzes a 16-frame sequence of optical flow descriptors (ROI flow) to confirm the object moves with human articulation (e.g., gait frequency oscillation in legs vs. whole-body translation).

Training Strategy
Due to the lack of public datasets with dense labels at the specific operating point (10–60 px targets with aggressive ego-motion), the authors employ a synthetic-motion curriculum:

Tier 1: Articulated human actors rendered in Blender and composited onto real drone footage to ensure realistic ego-motion.
Tier 2: Static aerial imagery (VisDrone) converted to pseudo-video with simulated drone motion and gait proxies.
Hard Negatives: Person-free clips from real deployment environments (swaying vegetation, shadows) to train the system to reject false positives.
Key Constraint: Every synthetic clip is passed through the deployed Stage A/B code to generate motion channels, eliminating the train/deploy distribution gap.

Key Contributions and Findings

Input Representation over Learned Temporal Machinery: The paper demonstrates that when input channels carry ego-normalized motion evidence, adding learned temporal modules (like Temporal Shift Modules) reduces accuracy. The deployed detector is stateless because the analytic motion channels already provide the necessary short-horizon temporal evidence.
Quantization Calibration Failures: The authors document and repair two silent calibration failures in int8 deployment:
- Moving-Average Range Estimation: Standard moving-average calibration washes out rare, large activations in regression heads (box size), halving accuracy. Min-max calibration is required.
- Stateful Graph Calibration: For streaming networks with cached states, calibrating with zero-initialized caches clips steady-state temporal information. Propagated cache states must be used during calibration.
Generalization vs. Overfitting: On a held-out VisDrone-DET split (real imagery never seen in training), the deployed EMTS-Det achieves 0.444 AP25, while a YOLOv8n fine-tuned on the exact same training mixture drops to 0.103 AP25. This indicates that appearance-based models overfit to the synthetic training domain, whereas the motion-based representation generalizes to real-world conditions.
Performance on Milliwatt Hardware:
- On a Raspberry Pi Zero 2W, the full pipeline runs at 31.85 FPS (31.4 ms/frame) with 0.462 AP25 and 0.714 recall.
- In contrast, YOLOv8n on the same hardware runs at 1.95 FPS with 0.172 AP25.
- The system achieves 97.9% lock recall on a 57-second field sequence with zero false re-locks after occlusions.

Significance and Claims
The paper claims that high-accuracy aerial tracking on milliwatt-class hardware is achievable not by shrinking modern detectors, but by redefining the problem: removing ego-motion analytically and presenting the network with pre-computed motion evidence. This reduces the learned workload to a tiny, stateless detector and a lightweight verifier.

The authors emphasize that temporal evidence belongs in the input representation, computed analytically for free, rather than in learned temporal machinery. This approach allows the system to transfer effectively to real-world data where appearance-based models fail, even when trained on identical datasets. The work also provides critical guidance for deploying stateful and regression-heavy networks in int8, addressing silent failure modes that standard calibration pipelines miss.

Limitations Acknowledged
The authors note that per-stage on-device profiling for thermal behavior is pending, the synthetic bench shares a generative process with training (mitigated by held-out real data), and the system degrades if a target remains stationary for extended periods (starving the motion channels). Additionally, the verifier's gait model is trained on walk cycles and untested on crawling or heavily encumbered gaits.

The Invisible Tracker: How Drones Learn to Follow Without Getting Dizzy

The Paper: "Moving Like a Human"

What They Found (and What They Rejected)

The Bottom Line

Technical Summary: EMTS-Det for Real-Time Aerial Person Tracking

More like this