💻 computer science

Follow Everything: A Leader-Following and Obstacle Avoidance Framework with Goal-Aware Adaptation

This paper presents a unified framework for robust leader-following that employs segmentation-based detection to handle arbitrary leader forms and a goal-aware adaptation mechanism with graph-based planning to ensure continuous tracking and obstacle avoidance even when the leader temporarily leaves the robot's field of view.

Original authors: Qianyi Zhang, Shijian Ma, Boyi Liu, Jianhao Jiao, Dimitrios Kanoulas

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qianyi Zhang, Shijian Ma, Boyi Liu, Jianhao Jiao, Dimitrios Kanoulas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot that acts like a loyal hiking companion. Its job is simple: follow a leader (a person, another robot, or even a sign) wherever they go, carrying supplies or just keeping up. But in the real world, this is surprisingly hard. If the leader steps behind a tree, the robot might forget who it's following. If the leader takes a shortcut over a hill, the robot might try to climb the hill too, even if it's a wheeled robot that can't climb.

This paper introduces a new system called "Follow Everything" that solves these problems using three main tricks. Think of it as giving the robot a super-memory, a smart brain for decision-making, and a map that sees the whole picture.

1. The "Distance Memory Bank" (Seeing Through the Fog)

The Problem: Traditional robots use cameras to spot a leader. If the leader walks behind a box and comes back out, the robot often thinks, "Oh, a new person!" and loses track of the original leader. It's like trying to recognize a friend in a crowd, but every time they step behind a pillar, the robot forgets their face.

The Solution: The researchers gave the robot a special "Distance Memory Bank."

How it works: Instead of just remembering what the leader looks like, the robot remembers where the leader was at different distances. It keeps a "snapshot" of the leader's appearance for every 1-meter interval (e.g., "what they look like at 2 meters," "what they look like at 5 meters").
The Analogy: Imagine you are looking for a friend in a park. If you only remember their face, you might get confused if they walk behind a tree. But if you also remember, "My friend looks like this when they are far away, and like this when they are close," you can instantly recognize them the moment they pop back out, no matter how far they were.
The Result: Even if the leader disappears for a moment, the robot doesn't get confused. It knows exactly who to look for and where to find them.

2. The "Smart Brain" (Knowing When to Chase, Follow, or Back Up)

The Problem: Most robots just blindly try to get closer to the leader. But sometimes that's dangerous. If the leader is far away, the robot should speed up. If the leader is right in front of it, it should slow down. If the leader walks backward, the robot should back up too, or it might crash.

The Solution: The robot has a "Goal-Aware Adaptation" system. It constantly asks itself, "What is the situation right now?" and switches modes like a video game character changing classes:

Chasing Mode: The leader is far away or out of sight. The robot says, "I need to catch up!" and speeds up to the last known location.
Following Mode: The leader is close and visible. The robot says, "I'll keep a safe, comfortable distance and walk beside you."
Retreating Mode: The leader walks backward toward the robot. The robot says, "Whoa, backing up!" and moves backward to avoid a collision.
Planning Mode: The leader disappears around a corner. The robot doesn't just stop; it says, "I'll go to the last place I saw you and wait there."

3. The "Shortcut Finder" (Not Just Copying Footsteps)

The Problem: Some robots are "copycats." If the leader walks a long way around a big rock to get to a door, the robot tries to walk that same long path, even if there is a clear, shorter path right next to it.

The Solution: The robot uses a Graph-Based Planner.

How it works: Instead of just copying the leader's path, the robot looks at the whole map of obstacles (like a spiderweb of possible routes). It calculates many different ways to get to the goal and picks the one that is the fastest and safest.
The Analogy: Imagine a leader walking a long, winding path around a puddle. A copycat robot would walk the whole long way. This new robot looks at the puddle, sees a dry bridge nearby, and says, "I can take the bridge and get there faster," without waiting for the leader to show it the way.

The Results: A Much Better Companion

The team tested this system in simulations and with a real legged robot (like a robot dog) in both indoor and outdoor settings. They compared it to other methods and found:

Success Rate: It successfully followed the leader 96.9% of the time (compared to as low as 11.9% for other methods).
Lost Sight: The robot lost sight of the leader for much less time.
Crashes: It crashed into obstacles or the leader 65% less often than other methods.
Distance: It kept a much better, safer distance, averaging just 2.0 meters away.

In short, this paper teaches robots to be less like blind followers and more like intelligent, adaptable companions who can remember who they are following, know when to speed up or slow down, and always find the smartest path to stay together.

Technical Summary: Follow Everything: Goal-Aware Adaptation and Graph-Based Planning Towards Arbitrary Leader Following

Problem Statement

Existing leader-following robotics systems face two primary limitations: a lack of generalization to arbitrary leaders and a failure to maintain tracking when the leader temporarily exits the robot's field of view (FOV). Traditional detection models rely on fixed object categories (e.g., "pedestrian"), causing the robot to lose the leader's identity (ID) once they reappear after occlusion. Furthermore, standard planning approaches often fail to adapt to dynamic leader-follower relationships, such as when a leader moves out of range, retreats, or when the follower must navigate complex obstacles without simply replicating the leader's historical path. These issues lead to high rates of visual loss, collisions, and inefficient trajectories.

Methodology

The paper proposes "Follow Everything," a unified framework integrating perception, state adaptation, and trajectory planning to address these challenges.

1. Perception: Segmentation with Distance Frame Buffer

To enable tracking of arbitrary leaders (humans, vehicles, robots) without category constraints, the system employs a segmentation model (based on SAM2) rather than traditional detection.

Temporal Memory Buffer ( $B_T$ ): Maintains high-confidence historical leader embeddings to bootstrap tracking.
Distance Frame Buffer ( $B_D$ ): A novel component designed to handle the specific challenge of leaders leaving and re-entering the FOV. The buffer divides the distance range between the follower and leader into equal intervals. It stores the highest-confidence leader embedding for each specific distance interval.
Mechanism: When a leader reappears, the system retrieves embeddings from the buffer corresponding to the current distance. This prevents the "bootstrap" process from being misled by low-quality, partial features captured just before the leader disappeared, significantly improving re-identification accuracy.

2. Goal-Aware Adaptation

The framework dynamically governs the robot's planning states based on the leader's visibility, motion, and relative position. Five distinct states are defined:

Chasing: Activated when the leader is visible but outside the costmap range. The robot maximizes speed to close the gap, utilizing a goal line derived from the costmap edge.
Following: Activated when the leader is visible and within the costmap. The robot maintains a safe distance (dynamically adjusted via a Kalman filter's Normalized Innovation Squared) and faces the leader, avoiding unnecessary detours.
Planning: Activated when the leader is lost (e.g., occluded). The robot targets the leader's last known position and orientation at maximum speed to recover visual contact.
Retreating: Activated when the leader approaches the robot. The robot moves backward to maintain a safe margin.
Switching: Activated when a new leader is assigned (e.g., via text prompt), resetting buffers and transitioning to the appropriate state.

3. Graph-Based Planner

To ensure efficient and safe navigation, the environment is modeled as a topological graph where obstacle clusters are nodes and shortest collision-free connections are edges.

Trajectory Generation: The planner generates candidate trajectories by traversing the graph. It considers homotopy classes to avoid redundant paths (e.g., clockwise vs. counter-clockwise detours).
Optimization: Candidate trajectories are optimized for time, subject to kinematic constraints, dynamic/static obstacle avoidance, and the specific goal/velocity constraints provided by the Goal-Aware Adaptation module.
Selection: The trajectory with the lowest time cost and highest similarity to the previous optimal path is selected, ensuring smooth transitions between states.

Key Contributions

Arbitrary Leader Tracking: Introduction of a segmentation module augmented with a Distance Frame Buffer, enabling robust re-identification of leaders of any shape or type after occlusion.
State-Driven Adaptation: Proposal of a Goal-Aware Adaptation mechanism that dynamically switches between Chasing, Following, Planning, Retreating, and Switching states to handle diverse interaction scenarios.
Efficient Navigation: Development of a Graph-Based Planner that generates and optimizes candidate trajectories, allowing the robot to find shorter, safer paths rather than merely replicating the leader's historical trajectory.

Results

The framework was evaluated through simulations (Gazebo) and real-world experiments using a legged robot follower with diverse leaders (mobile robots, UAVs, pedestrians, stop signs) in indoor and outdoor settings. The method was compared against four baselines: Alaa (SAM2 + PID), MPC-YOLO, and two ablation variants (FE-N-DFB and FE-N-GP).

Performance Metrics (Follow Everything vs. Best Baseline):

Follow Success Rate: Improved to 96.9% (vs. 81.3% for the best baseline, FE-N-GP).
Average Leader Loss Time: Reduced to 10.7% (vs. 20.0% for FE-N-GP).
Collision Rate: Reduced to 1.8% (vs. 16.3% for FE-N-GP and 66.9% for Alaa).
Average Distance: Reduced to 2.0 m (vs. 2.6 m for FE-N-GP).

In specific scenarios, such as following a UAV through a forest or a pedestrian in a factory with partial occlusions, the system demonstrated superior robustness in re-identification and obstacle avoidance compared to methods relying solely on temporal memory or standard detection.

Significance

The paper claims that "Follow Everything" provides a unified solution for robust leader-following that overcomes the limitations of category-specific detection and rigid planning. By integrating distance-aware feature buffering with adaptive state management and graph-based optimization, the framework significantly enhances the robot's ability to track arbitrary leaders, recover from occlusions, and navigate complex environments efficiently. The authors posit that this approach makes leader-following more practical for real-world applications such as supply carrying, customer guidance, and patrol assistance, where leaders may be non-standard objects or may frequently disappear from view.