🤖 AI

UESF-Bench: Benchmarking and Probing for Unified Embodied Seeking and Following

This paper introduces UESF-Bench, a large-scale benchmark for unified embodied human seeking and following that addresses the limitations of existing evaluations by requiring agents to first locate a language-described target before persistently tracking them, and proposes the SeekFollow-VLA framework to effectively manage the transition between these two phases.

Original authors: Kun Yu, Jianhua Yang, Yixiang Chen, Changwei Wang, Hongyuan Yu, Yan Huang, Fushuo Huo, Ya Jing, Zhumin Chen, Keji He

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kun Yu, Jianhua Yang, Yixiang Chen, Changwei Wang, Hongyuan Yu, Yan Huang, Fushuo Huo, Ya Jing, Zhumin Chen, Keji He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to be the ultimate sidekick. In the world of robotics, there's a big difference between just "seeing" and "understanding." For a robot to be truly helpful, it needs Embodied AI—a brain that lives inside a body, allowing it to move through the real world, not just stare at a screen. It also needs Vision-Language-Action skills, which is a fancy way of saying it must look at what's happening, listen to what you say, and then physically move to do something about it. Think of it like a game of hide-and-seek where the robot has to listen to a clue like "Find the guy in the red jacket," run around a house to find him, and then stick close by without bumping into him. Why does this matter? Because in real life, people don't always stand right in front of a robot waiting to be followed. Often, they are hiding in another room, or the robot has lost sight of them. If a robot can only follow someone it can already see, it's not very useful. It needs to be able to hunt them down first, then switch gears to follow them, all while figuring out who is who in a crowded room.

This paper introduces a new challenge and a new solution for exactly that problem. The authors, a team of researchers from various universities and companies, realized that most existing tests for robot followers were too easy: they assumed the target person was already standing right there, visible from the start. They argue that this misses the most important part of the job: the search. To fix this, they built a massive new playground called UESF-Bench (Unified Embodied Seeking and Following Benchmark). It's like a giant video game level with over 1.43 million different scenarios, featuring 4,800+ unique digital people and 770+ different environments. In this benchmark, the robot gets a single instruction like, "Find the short man with brown hair in the kitchen and follow him," and it has to do two things: first, actively search for him when he's out of sight, and second, seamlessly switch to following him once found, even if he gets lost in a crowd again.

The researchers found that simply telling a robot to "do both" isn't enough; it gets confused. They tested three different ways to build the robot's brain. The first was a "Single Head," where the robot tried to use one brain for both searching and following. The second was a "Dual Head" with a basic switch, and the third was a "Task-Driven Router," a smarter system that acts like a traffic cop, explicitly telling the robot when to stop searching and start following based on what it sees. The results showed that the "Single Head" approach was a disaster, with the robot succeeding in only 4% of the single-person tests. Even the basic "Dual Head" struggled, staying at 5%. However, the "Task-Driven Router" (which they call SeekFollow-VLA) was a game-changer. In single-person tests, it succeeded 35% of the time, and in the much harder multi-person tests (where fake people are hiding to trick the robot), it still managed a 20% success rate, far beating the others.

The paper suggests that the key to success isn't just having a better camera or a faster brain, but having a clear internal signal that knows when the job has changed from "hunting" to "herding." The researchers showed that their smart router learns to act like a detective first, scanning the room with high intensity, and then instantly switches to a loyal bodyguard mode the moment it spots the right person. They also noted that while this new method is much better at finding and following, it does bump into things slightly more often in crowded rooms, suggesting that being a great hunter sometimes makes you a bit clumsy. Ultimately, the paper argues that to build robots that can truly help us in the real world, we need to stop testing them on easy, static tasks and start challenging them with the messy, confusing reality of finding someone who is hiding, and then sticking with them through the chaos.

Technical Summary: UESF-Bench and SeekFollow-VLA

Problem Definition

Current embodied AI benchmarks for human following, such as EVT-Bench, typically assume the target person is visible at the start of an episode. This simplification overlooks a critical real-world requirement: an agent must often first locate a language-described target who is initially out of view and then seamlessly transition to persistent following in dynamic environments.

Existing research treats human search and human following as separate tasks, often relying on strong environmental priors or explicit cues. Furthermore, they fail to address the continuity of interaction, specifically the challenge of implicit phase transitions. An agent must autonomously determine when to switch from searching to following, and conversely, whether to resume searching or attempt local recovery when a target is temporarily lost. This paper formulates Embodied Visual Seek-and-Follow as a unified task where an agent must handle semantic-guided exploration, reliable behavior switching, and delayed identity grounding under distractors, all based on a single fused natural language instruction.

Methodology: SeekFollow-VLA

To address these challenges, the authors propose SeekFollow-VLA, a Vision-Language-Action (VLA) framework designed for unified seek-and-follow.

Architecture

Backbone: The model utilizes a pre-trained Vision-Language Model, Qwen3-4B, as its backbone.
Visual Encoding: It employs a dual-tower visual encoder (DINO-V3 and SigLIP) to process egocentric RGB observations. Features are extracted at a unified resolution (384×384) and fused. To balance detail and efficiency, the system uses a sliding window mechanism: fine-grained tokens for the current frame and coarse-grained tokens for historical frames ( $k=31$ ).
Multimodal Fusion: Visual tokens are projected into the language latent space via a cross-modality projector and concatenated with fused instruction tokens.

Action Head Designs

The core innovation lies in the investigation of three alternative action head architectures to manage the implicit transition between seeking and following:

Single Head: A shared MLP maps the multimodal representation directly to waypoints. This serves as a baseline but risks "mode confusion" between the distinct objectives of searching and tracking.
Dual Heads with Phase-Aware Router: Two separate heads (Seeking and Following) are combined using routing weights learned solely from the final waypoint prediction objective.
Dual Heads with Task-Driven Router (Proposed): This variant introduces explicit supervision for the routing process. In addition to routing weights, the router predicts task-type logits ( $g_T$ $g_{T}$ ) supervised by a ground-truth stage label ( $z_T$ $z_{T}$ ).
- $z_T = 0$ (Seeking) until the target is first successfully found (defined as being within 1.7m and facing the agent).
- $z_T = 1$ (Following) thereafter.
- Loss Function: The training objective combines the waypoint prediction loss ( $L_{wp}$ ) with an auxiliary stage-aware cross-entropy loss ( $L_{stage}$ ), weighted by $\beta_{nav} = 10$ . This explicitly encourages the model to learn correct task-aware routing between modes.

Key Contributions

Task Formulation: The paper defines embodied visual seek-and-follow as a unified task requiring an agent to locate a language-described target initially out of view and then transition to persistent following without explicit phase signals.
UESF-Bench: The authors construct the Unified Embodied Seeking and Following Benchmark, a large-scale dataset containing 1.43 million samples across 770+ environments and 4,800+ diverse avatars. It covers both single-person and multi-person (crowded) scenarios, addressing the lack of a unified evaluation standard for this specific task.
SeekFollow-VLA Framework: A VLA framework featuring a task-driven routing mechanism that enables latent phase inference and transition modeling, effectively reducing mode confusion between seeking and following behaviors.
Empirical Validation: Extensive experiments demonstrate that the task-driven dual-head design significantly outperforms single-head and phase-aware dual-head baselines, particularly in challenging multi-person environments.

Experimental Results

Experiments were conducted on the UESF-Bench test set using metrics including Task Success Rate (TSR), Collision Rate (CR), Following Ratio (FR), and Search SPL.

Single-Person Setting:
- DualHead-TD achieved a TSR of 0.35, significantly outperforming SingleHead (0.04) and DualHead-PA (0.05).
- It also achieved the highest FR (0.92) and Search SPL (0.53).
- Results indicate that explicit task-driven supervision is necessary to resolve behavioral ambiguity and improve search efficiency.
Multi-Person Setting:
- Performance generally declined due to distractor interference, but DualHead-TD maintained the highest TSR (0.20) compared to 0.04 for the other baselines.
- DualHead-TD achieved a Search SPL of 0.55, demonstrating superior target identification and search efficiency even in crowded scenes.
- While DualHead-PA showed a slightly higher FR (0.85 vs. 0.82), DualHead-TD's superior TSR and Search SPL highlight its effectiveness in the unified task, where successful transition and target recovery are critical.
Routing Dynamics: Visualization of the router weights confirmed that the Task-Driven Router learns a clear functional separation: assigning near-100% weight to the seeking head during exploration and sharply transitioning to the following head upon target discovery, maintaining stability thereafter.

Significance and Claims

The paper claims that UESF-Bench is the first large-scale benchmark for unified language-guided human seeking and following in embodied settings. By moving beyond the assumption of initial target visibility, the work addresses a more realistic form of embodied interaction.

The authors argue that treating search and follow as independent tasks fails to capture the continuity of real-world interaction. The proposed SeekFollow-VLA demonstrates that a unified framework with explicit task-driven routing can effectively manage the implicit phase transitions and delayed identity grounding required for robust human-robot interaction. The results suggest that for complex embodied tasks, architectural inductive biases that explicitly model stage transitions are crucial for performance, particularly in environments with distractors.