ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraints

1. Problem Statement

Current embodied AI agents typically operate under the assumption that object affordances (the usability of objects for specific actions) are static and fully specified in the instructions. However, real-world environments are dynamic; objects may become temporarily unavailable due to state changes (e.g., a microwave is occupied, a cloth is dirty) or external factors not explicitly mentioned in the user's command.

Existing benchmarks (like ALFRED) and methods (like FILM or CAPEAM) fail in these scenarios because:

• They lack mechanisms to detect unmet preconditions (e.g., realizing a cloth must be cleaned before being put in a drawer).
• They struggle with dynamic state transitions, often attempting to execute inapplicable actions or failing to recover without explicit error signals.
• They rely on literal instruction following rather than commonsense reasoning about latent constraints.

The core problem is how to enable embodied agents to infer implicit, time-varying affordance constraints and adapt their planning accordingly without explicit re-specification of the goal.

2. Methodology

A. The DynAfford Benchmark

To address the lack of evaluation standards for this problem, the authors introduce DynAfford, a new benchmark built on the AI2-THOR simulator.

• Dataset: Contains 2,628 expert demonstrations and 10,106 natural language annotations across 57 scenes (kitchens and bathrooms).
• Dynamic Affordance Injection: Unlike static benchmarks, DynAfford modifies the initial state of objects to violate implicit preconditions (e.g., initializing a microwave as "occupied" or a cloth as "dirty") while keeping the instruction text and high-level goal unchanged.
• Task Types: Covers six complex interaction types: picking, placing, cleaning, heating, cooling, and stacking.
• Splits: Includes "Seen" (training distribution) and "Unseen" (novel scenes) splits, with both static (idealized) and dynamic (realistic) settings to isolate the impact of affordance reasoning.

B. The ADAPT Framework

The authors propose ADAPT (Affordance-Aware Action Selection), a plug-and-play decision-time inference module that augments existing planners. It operates in two sequential stages:

1. Stage I: Affordance State Inference

   • Before executing a planned high-level action, the agent checks if the target object's latent preconditions are met.
   • Vision-Language Model (VLM): Uses a LoRA-finetuned LLaVA-1.5-7B model.
   • Multimodal In-Context Learning (MICL): To improve accuracy, the model receives a triplet of images: (1) a reference of the object in an available state, (2) a reference in an unavailable state, and (3) the current egocentric observation. This allows the model to reason about usability via visual comparison.
   • Output: A binary classification of whether the action is executable (Available) or not (Unavailable).

2. Stage II: Applicability Resolution

   • If the action is deemed unavailable, ADAPT does not replan from scratch (which breaks long-horizon coherence). Instead, it:
     • Defers the original action.
     • Queries a Large Language Model (LLM) with the current observation, the inferred constraint, and the deferred goal.
     • Infers a commonsense-consistent resolution strategy (e.g., "Wait" for an appliance to free up, or "Clean" a dirty object).
   • Once the constraint is resolved, the agent resumes the original deferred action.

3. Key Contributions

1. DynAfford Benchmark: The first benchmark specifically designed to evaluate embodied agents under dynamic, unspecified affordance constraints and commonsense-driven goal conditions. It exposes a significant performance gap in state-of-the-art methods.
2. ADAPT Module: A unified, architecture-agnostic inference mechanism that enables agents to detect latent preconditions and resolve action applicability failures without modifying the underlying planner or low-level controllers.
3. Domain-Adapted VLMs: Demonstrates that a task-specific, LoRA-finetuned VLM significantly outperforms general-purpose commercial models (like GPT-4o) in grounding affordance states, highlighting the need for specialized visual grounding over general reasoning.

4. Experimental Results

Experiments were conducted on DynAfford using baselines like MOCA, FILM, CAPEAM, SayCan, and LLM-Planner.

• Performance Gains: Integrating ADAPT with strong baselines yielded substantial improvements:
  • FILM + ADAPT: Achieved a 73.2% relative improvement in Success Rate (SR) on the unseen split (from 9.34% to 16.18%) and a 34.7% improvement in Goal Condition (GC) completion.
  • CAPEAM + ADAPT: Showed consistent gains, particularly in path-length weighted metrics, indicating more efficient recovery strategies.
• Comparison with Commercial Models: Replacing the fine-tuned LLaVA backend with GPT-4o resulted in lower performance (approx. 10% lower accuracy in affordance inference), proving that general reasoning is insufficient without task-aligned visual grounding.
• Ablation Studies:
  • Removing LoRA fine-tuning dropped SR from 27.69% to 16.07%.
  • Removing Multimodal In-Context Learning caused a catastrophic drop to 6.15%, proving the necessity of visual reference examples.
• Efficiency: While ADAPT increases total task duration (due to necessary waiting/cleaning steps), the computational overhead is low (approx. 0.2s per inference call) as it is event-driven rather than step-driven.

5. Significance and Conclusion

This work addresses a critical gap in embodied AI: the transition from literal instruction following to commonsense-driven execution.

• Robustness: It demonstrates that agents can handle real-world unpredictability (e.g., dirty objects, occupied appliances) by inferring implicit constraints.
• Future Direction: The paper establishes that effective embodied intelligence requires not just planning what to do, but reasoning when an action is feasible.
• Limitations: Current limitations include sensitivity to partial occlusion (single-view observations) and the lack of real-world physical uncertainty modeling. Future work aims to incorporate multi-view perception and safety-aware control for real-world deployment.

In summary, ADAPT and DynAfford provide a foundational step toward creating embodied agents that are robust, adaptable, and capable of commonsense reasoning in dynamic, unstructured environments.