Environmental Understanding Vision-Language Model for Embodied Agent

1. Problem Statement

While Vision-Language Models (VLMs) have demonstrated strong reasoning and instruction-following capabilities, they face significant limitations in environmental understanding when applied to embodied agents. Current approaches suffer from three main issues:

• Reliance on Metadata: Many existing agents depend on environment metadata (e.g., object IDs, masks) rather than visual perception to execute actions.
• Lack of Explicit Interpretation: End-to-end models often lack explicit capabilities to interpret environmental states, leading to execution errors and task failures.
• Ineffective Recovery: Existing recovery methods rely on external environment feedback, predefined failure types, or textual feedback, which lack the visual grounding necessary for robust agent-environment interaction.

The core challenge is to enable a single VLM to perceive, plan, judge success, and recognize goals within a unified framework without relying on complex modular pipelines or external metadata.

2. Methodology: The EUEA Framework

The authors propose Environmental Understanding Embodied Agent (EUEA), a framework that fine-tunes a single VLM to internalize four core skills. This approach bridges explicit skill modeling with end-to-end learning.

A. Four Core Skills

The framework defines eight specific sub-skills grouped into four categories, trained via Supervised Fine-Tuning (SFT):

1. Object Perception:
   • Object Recognition (OR): Identifies all visible objects in an image.
   • Object Detection (OD): Locates specific objects relevant to the current goal using bounding boxes.
2. Task Planning:
   • Subgoal Task Planning (STP): Breaks down a main instruction into intermediate subgoals.
   • Step-by-Step Action Planning (SAP): Generates the specific action (e.g., PickupObject) and target object for the current step, utilizing past memory ($M$).
3. Action Understanding:
   • Action Success Prediction (ASP): Predicts whether a proposed action will succeed or fail in the current state.
   • Future Situation Captioning (FSC): Describes the expected visual changes in the environment after an action is performed.
   • Action Grounding (AG): Identifies the action and object executed between two consecutive frames.
   • Note: To address the lack of failure data in expert demonstrations, the authors generate "failure-rich" data via random exploration.
4. Goal Recognition:
   • Main Goal Recognition (GRmain): Determines if the overall task is complete.
   • Subgoal Recognition (GRsub): Determines if the current subgoal is complete, enabling the agent to transition to the next step.

B. Recovery Step (Sampling-Based)

To handle interaction failures without external feedback, EUEA introduces a Recovery Step:

• When an action fails (detected by the environment or internal logic), the agent triggers the SAP skill to sample $n$ alternative actions.
• The agent selects the action-object pair with the highest probability (lowest negative log-likelihood) that aligns with the goal.
• If the action-object pair is the same as the failed one, the agent re-samples the bounding box via the OD skill.
• This allows the agent to self-correct based on its learned environmental understanding.

C. GRPO Refinement Stage

To further refine inconsistent predictions, the authors employ Group Relative Policy Optimization (GRPO):

• Reward Function: A rule-based reward system ($R_{total}$) evaluates correctness using Intersection over Union (IoU) for bounding boxes, Jaccard index for object lists, and sequence order for planning.
• Strategy: The model generates multiple responses for ambiguous cases. GRPO optimizes the policy to maximize rewards, specifically targeting cases where the model shows uncertainty, thereby stabilizing the skills.

3. Key Contributions

1. Unified EUEA Framework: A novel architecture that internalizes four core environmental understanding skills (Perception, Planning, Understanding, Recognition) into a single VLM, eliminating the need for separate modular components.
2. Self-Correction Mechanisms: Introduction of a sampling-based Recovery Step (no additional training required) and a GRPO refinement stage that improves performance by resolving inconsistent skill predictions.
3. Comprehensive Datasets & Benchmarks: Construction of large-scale skill datasets (1.24M training samples for ALFRED, 3.7M for LangR) and a new evaluation benchmark covering object grounding, planning, and action prediction.
4. Empirical Validation: Demonstration that explicit skill-level supervision significantly outperforms behavior cloning (BC) baselines and zero-shot VLMs.

4. Results

The framework was evaluated on the ALFRED and LangR benchmarks.

• Task Success Rate (ALFRED):
  • The SFT model achieved an 8.86% improvement in average success rate over the Behavior Cloning (BC) baseline.
  • The addition of the Recovery Step and GRPO stage provided an additional 3.03% gain, reaching a total improvement of 11.89% over the baseline.
  • Final average success rate: 86.48% (compared to 74.59% for BC).
• Skill Evaluation:
  • EUEA significantly outperformed both closed-source (e.g., GPT-4o, Gemini-2.5-Pro) and open-source (e.g., LLaVA, InternVL) VLMs in zero-shot settings, particularly in Action Grounding and Object Perception.
  • While Gemini-2.5-Pro showed strong reasoning, it still struggled with specific environmental constraints (e.g., 14% failure rate in step-by-step planning), whereas EUEA achieved near-perfect accuracy in planning and grounding.
• Ablation Studies:
  • Removing any of the four core skills led to a drop in task success, with Action Understanding (ASP/FSC) having the largest impact (9.32% drop), highlighting the importance of anticipating environmental changes.
  • The method remained robust across different backbone models (InternVL and Qwen series) and scales.

5. Significance

This work addresses a critical gap in embodied AI: the transition from "seeing" to "understanding."

• Robustness: By enabling agents to predict action success and recognize goal completion visually, EUEA reduces reliance on external metadata and predefined failure types.
• Generalization: The framework demonstrates strong Out-of-Distribution (OOD) generalization, transferring skills learned in ALFRED to the photorealistic LangR environment.
• Scalability: The approach proves that end-to-end learning with explicit skill supervision is a viable path toward reliable, instruction-following agents that can operate in complex, dynamic environments without human intervention or complex modular pipelines.

The paper concludes that future work should extend these discrete-action skills to continuous environments using video inputs and integrate explicit reasoning mechanisms to further enhance decision-making.