Technical Summary: Follow Instructions with Social and Embodied Reasoning (FISER)

Problem Formulation

The paper addresses the challenge of enabling AI agents to follow natural language instructions in collaborative, embodied human-robot tasks. While standard grounded language learning maps instructions to actions, it often fails in real-world scenarios where human instructions are inherently ambiguous. Humans assume listeners possess prior knowledge about their hidden goals and intentions, leading them to omit critical information for efficiency (Grice 1975; Sperber and Wilson 1986).

The authors formalize this as a Human-Robot Markov Decision Process (MDP). In this setting, a human performs a trajectory of actions $\tau_{t'}$ in a shared environment before issuing a natural language instruction $u$. The robot must infer the human's hidden internal reward function $R_h$ and specific sub-goal $p^*$ to determine the correct robot task $G_r$. The core difficulty lies in the fact that the instruction $u$ alone is insufficient; the robot must leverage the partially observable human trajectory and the context of the environment to disambiguate the request.

Methodology: The FISER Framework

The authors propose Follow Instructions with Social and Embodied Reasoning (FISER), a framework that explicitly decomposes the instruction-following problem into two distinct phases: Social Reasoning and Embodied Reasoning.

1. Modeling Human Intentions

The framework models the human's hidden goal $G_h$ as a set of logical predicates. The human selects a sub-goal $p^*$ (a specific part of the overall plan) requiring assistance and generates an instruction $u$ to specify a robot task $G_r$. The robot cannot observe $G_h$, $p^*$, or $G_r$ directly; it must infer them from the observed trajectory $\tau_{t'}$, the final state $s_{t'}$, and the instruction $u$.

2. Two-Phase Reasoning

• Phase 1: Social Reasoning (Task Recognition): The robot disambiguates the natural language instruction into a concrete, executable robot task $G_r$. This phase involves:
  • Robot's Task Recognition: Estimating $G_r$ given $s_{t'}$, $\tau_{t'}$, and $u$.
  • Human's Plan Recognition (Optional): An enhanced stage where the model explicitly predicts the human's sub-goal $p^*$ (e.g., "storing books in a box") before determining $G_r$. This is modeled as predicting a tuple of logical predicates (Quantifier, Subjective, Verb, Objective).
• Phase 2: Embodied Reasoning (Grounded Planning): Once $G_r$ is established, the problem reduces to a standard goal-conditioned planning task. The robot generates a sequence of actions $\{a_t\}$ to achieve $G_r$ within the environment.

3. Model Implementation

The authors implement a Transformer-based model trained from scratch (due to the excessive token lengths of the input data, which exceed the context limits of existing small pre-trained models).

• Architecture: The model utilizes $3N$ encoder layers. The first $2N$ layers constitute the Social Reasoning phase, and the final $N$ layers constitute the Embodied Reasoning phase.
• Modality Interaction: A specialized module fuses four input streams: World State Description, Human Trajectory, Language Instruction, and Model's Past Outputs.
• Training Strategies:
  • End-to-End (E2E): The model is trained to output actions directly, with an auxiliary loss applied to intermediate predictions.
  • Multi-Staged (MS): The Social Reasoning and Embodied Reasoning modules are trained separately. The Social module predicts $G_r$ (and optionally $p^*$), which is then fed as a ground-truth input to the Embodied module during evaluation.

Experimental Setup

The framework is evaluated on the HandMeThat (HMT) benchmark, a text-based household environment containing ambiguous instruction-following tasks across four difficulty levels:

• Level 1: Pure planning (no ambiguity).
• Level 2: Requires inferring goals from human trajectories.
• Level 3: Requires pragmatic reasoning to resolve linguistic ambiguity.
• Level 4: Requires strong priors to resolve inherent ambiguities where information is insufficient.

The authors compare their approach against:

1. Baselines: Human performance, hand-coded heuristics, and a Seq2Seq neural network.
2. State-of-the-Art (SOTA): The previous best method (Cao et al. 2024) which uses iterated goal inference.
3. Large Language Models (LLMs): GPT-4 Turbo prompted with Chain-of-Thought (CoT), including standard prompting, prompt engineering (PE), and CoT prompts mimicking the FISER structure (predicting intermediate steps).

Key Results

The experiments yield three primary findings:

1. Separation of Reasoning Improves Performance: Models that explicitly separate social and embodied reasoning (FISER) significantly outperform end-to-end approaches. The Multi-Staged + FISER model achieves a 64.5% average success rate on the HMT test set, setting a new state-of-the-art.
2. Small Models Outperform Prompted LLMs: Despite GPT-4 Turbo's access to vast common-sense knowledge, it fails to match the performance of the small-scale Transformer models trained from scratch. Even with CoT prompting and domain-specific knowledge, GPT-4 achieves less than half the success rate of the FISER model on ambiguous levels (2–4).
3. Failure Mode Analysis:
   • GPT-4 struggles primarily with Planning Failures (hallucinating object locations) and Incorrect Intentions (misaligning common sense with the specific human goal).
   • FISER models effectively avoid planning failures and redundant behaviors but still face challenges in correctly identifying the target object in highly ambiguous scenarios.
   • Filtering irrelevant objects improves GPT-4's performance slightly but does not close the gap, indicating that the limitation is not just object selection but the fundamental ability to perform the required social and embodied reasoning.

Significance and Claims

The paper claims that the key insight for solving ambiguous instruction-following tasks is explicitly modeling human intentions as intermediate reasoning steps. By treating the human as a partially observable agent with hidden goals, rather than just a disembodied instruction issuer, the system can better leverage context to resolve ambiguity.

The authors assert that:

• Pre-training is insufficient: General pre-trained LLMs, even with CoT prompting, lack the specific domain knowledge and reasoning structure required for complex social and embodied tasks in shared environments.
• Structured reasoning is superior: Decomposing the problem into social (intent inference) and embodied (action planning) reasoning phases yields better results than monolithic end-to-end training or direct prompting.
• Efficiency: Training a small, domain-specific model from scratch is more effective and reliable than attempting to prompt massive, general-purpose LLMs for these specific collaborative tasks.

The work establishes that for AI agents to be truly helpful in human environments, they must move beyond simple language grounding to incorporate social reasoning that infers the "why" behind human actions before executing the "how."