GRAIL: Goal Recognition Alignment through Imitation Learning

1. Problem Statement

Goal Recognition (GR) is the task of inferring an agent's underlying objective from a partial sequence of observed actions. While critical for human-robot interaction and multi-agent coordination, existing GR methods suffer from two primary limitations:

1. The Optimality Assumption: Most classical and Reinforcement Learning (RL)-based GR approaches assume agents act optimally (e.g., following shortest paths or maximizing a reward function). This fails when agents exhibit systematic biases (e.g., preferring scenic routes over shortest paths) or suboptimal behavior (noise, heuristics, cognitive limitations).
2. Computational Latency: Many methods require running a planner or RL rollout for every candidate goal during inference, creating bottlenecks for real-time, interactive applications.

The paper argues that real-world agents often deviate from idealized rationality in predictable ways, and current GR systems fail to align with these actual behavioral models.

2. Methodology: GRAIL Framework

The authors propose GRAIL (Goal Recognition Alignment through Imitation Learning), a framework that reframes GR as a collection of Imitation Learning (IL) problems. Instead of learning a reward function or a planner, GRAIL learns a specific policy for each candidate goal directly from demonstration trajectories.

Core Workflow

The framework operates in two distinct phases:

A. Offline Policy Learning (Training)
For each candidate goal $g \in G$, the system trains a dedicated policy $\pi_g$ using a dataset of demonstrations ($D_g$) where an agent was pursuing goal $g$. The framework supports three IL backbones:

• Behavioral Cloning (BC): Supervised learning on state-action pairs to maximize the likelihood of expert actions.
• Generative Adversarial Imitation Learning (GAIL): Uses adversarial training to match the occupancy distribution of the learner's policy with expert trajectories.
• Adversarial Inverse Reinforcement Learning (AIRL): Learns a reward function and a policy simultaneously to explain expert data.

Crucially, these policies absorb the specific "fingerprint" of the agent's behavior, including biases and suboptimal quirks, rather than assuming a theoretical optimal path.

B. Online One-Shot Inference (Testing)
When a new partial trajectory $\tau_{obs}$ is observed:

1. The system evaluates the trajectory against every learned policy $\pi_g$ in a single forward pass.
2. It computes a similarity score $S_g(\tau_{obs})$ using a Negative Average Mean Squared Error (MSE) metric:
   $$S_g(\tau_{obs}) = -\frac{1}{T} \sum_{t=0}^{T-1} \|\pi_g(s_t) - a_t\|^2$$
   (Note: This metric measures how well the policy's predicted action matches the observed action at each step.)
3. The goal with the highest score is selected: $\hat{g} = \arg\max_{g \in G} S_g(\tau_{obs})$.

Key Innovation: This approach eliminates the need for planners, environment rollouts, or reward modeling at inference time, enabling fast, "planner-free" recognition.

3. Key Contributions

1. Novel Formulation: The first framework to treat Goal Recognition as a set of Imitation Learning problems, allowing the system to learn directly from potentially suboptimal or biased demonstrations.
2. Alignment with Real Behavior: By learning policies directly from data, GRAIL captures systematic biases (e.g., route preferences) that optimal-planning-based methods miss.
3. Efficiency: The inference phase is computationally lightweight, requiring only a forward pass through neural networks, making it suitable for real-time applications.
4. Unified Approach: It bridges the gap between Goal Recognition, Inverse Reinforcement Learning (IRL), and Imitation Learning (IL), showing that IL algorithms can be repurposed for one-shot goal inference without explicit reward recovery.

4. Experimental Results

The authors evaluated GRAIL on two domains: MiniGrid (discrete navigation) and PandaReach (continuous robotic control). They compared GRAIL variants (BC, GAIL, AIRL) against state-of-the-art baselines: GRAQL (Q-learning based) and DRACO (PPO-based).

Performance Highlights:

• Systematically Biased Optimal Behavior: In scenarios where agents take optimal paths but with consistent biases (e.g., preferring a park route), GRAIL achieved near-perfect F1-scores (1.0), whereas the Q-learning baseline (GRAQL) struggled (F1 $\approx$ 0.3–0.5) because it assumed uniform optimality.
• Suboptimal Behavior: Under noisy or heuristic-driven behavior, GRAIL variants consistently outperformed baselines, showing F1-score improvements of 0.1 to 0.3. BC-GRAIL was particularly effective, internalizing systematic deviations.
• Optimal Behavior: Even in clean, optimal settings, GRAIL remained competitive, often matching or slightly outperforming RL-based methods, especially as the number of candidate goals increased.
• Efficiency:
  • Training: BC-GRAIL trained significantly faster than RL baselines (e.g., ~5s per goal in MiniGrid vs. ~300s for GRAQL).
  • Inference: GRAIL inference was dramatically faster than DRACO, as it required no environment interaction or rollouts.

5. Significance and Implications

• Robustness to Real-World Data: GRAIL addresses the "reality gap" where human and agent behavior is rarely perfectly optimal. It is the first GR system explicitly designed to handle systematic biases and suboptimality without treating them as noise.
• Scalability: The "one-shot" inference capability makes GRAIL viable for time-sensitive applications like autonomous driving or assistive robotics, where latency is critical.
• Ethical Alignment: By inferring goals based on actual observed behavior rather than idealized rational models, GRAIL offers a more accurate and potentially safer way to align AI systems with human intentions, reducing the risk of misinterpreting human preferences.

Limitations & Future Work:
The current framework assumes a closed set of known candidate goals and fixed environment dynamics. Future work aims to extend GRAIL to open-set goal recognition (discovering new goals) and handling dynamic shifts in environment dynamics. Additionally, the authors note that while the current study uses synthetic data, applying this to complex, perception-heavy real-world inputs remains a challenge.