Technical Summary: PRISM – Perception Reasoning Interleaved for Sequential Decision Making

1. Problem Statement

The integration of Large Language Models (LLMs) and Vision-Language Models (VLMs) into embodied agents for sequential decision-making faces a critical "perception–reasoning–decision" gap. While LLMs excel at logical reasoning and VLMs can process visual inputs, current approaches often decouple these modules. In standard pipelines, a VLM passively generates a generic scene description which is then fed to an LLM for decision-making. This loose coupling leads to two primary issues:

1. Information Loss: VLMs often overlook task-critical, fine-grained visual details because they lack the specific context of the agent's current goal.
2. Hallucination and Noise: Simply conditioning the VLM on the goal can induce hallucinations (describing non-existent objects), while naively concatenating question-answer (QA) pairs with scene descriptions creates long, noisy prompts that hinder downstream reasoning.

Furthermore, existing interactive methods often rely on predefined questions or oracle annotations, limiting their autonomy and applicability to real-world settings where such priors are unavailable.

2. Methodology: The PRISM Framework

PRISM (Perception Reasoning Interleaved for Sequential Decision Making) addresses these challenges by establishing a tight, dynamic coupling between perception and decision-making through a Dynamic Question-Answering (DQA) pipeline. The framework operates in two interleaved phases within a goal-conditioned Partially Observable Markov Decision Process (POMDP):

A. Interactive Goal-Oriented Perception

Instead of a single-pass description, PRISM employs an iterative communication loop between a VLM (perception module, $V_p$) and an LLM (reasoning module, $R$):

1. Initial Description: The VLM generates an initial, goal-agnostic description ($d^t_i$) of the visual observation ($o_t$).
2. Gap Identification & Question Generation: The LLM analyzes $d^t_i$ against the task goal ($g$) to identify missing information. It generates a set of specific, goal-relevant questions ($Q_t$) to query the VLM.
3. Targeted Querying: The VLM answers each question in $Q_t$ based on the current visual observation, producing a set of answers ($\Lambda_t$).
4. Semantic Synthesis: Crucially, the LLM does not simply concatenate the raw QA pairs. Instead, it synthesizes the initial description, questions, and answers into a compact, refined final description ($d^t_f$). This step filters out irrelevant details and hallucinations while emphasizing task-critical elements.

B. Sequential Decision-Making

The refined description ($d^t_f$), along with the goal and interaction history, is fed into the LLM policy ($\pi_\theta$) to generate the next action ($a_t$). The policy is augmented with LoRA (Low-Rank Adaptation) adapters specifically trained for action generation, while the base LLM retains its pre-trained reasoning capabilities.

C. Training Strategy

PRISM utilizes a two-stage training pipeline exclusively within the visual environment, avoiding reliance on privileged textual supervision:

1. Behavioral Cloning (BC): The policy is pre-trained to mimic expert demonstrations (converted to text via the PRISM perception pipeline).
2. Online Reinforcement Learning (RL): The policy is further fine-tuned using Proximal Policy Optimization (PPO). Notably, the DQA perception loop remains active during RL, allowing the agent to learn from online interactions.

3. Key Contributions

• Dynamic Question-Answering (DQA): A fully automatic mechanism that tightly couples perception and decision-making. Unlike prior works, it requires no handcrafted questions or oracle annotations, as the LLM dynamically generates goal-oriented queries.
• Semantic Synthesis: A novel merging strategy where the LLM synthesizes QA interactions into a concise, task-driven description, avoiding the noise and verbosity of naive concatenation.
• Efficient Architecture: The framework separates perception (VLM) from decision-making (LLM with LoRA), enabling the use of established text-based RL recipes while grounding decisions in visual environments.
• Automatic Pipeline: The system eliminates the need for human-in-the-loop clarification or predefined question sets, enhancing autonomy.

4. Experimental Results

The authors evaluated PRISM on two benchmarks: ALFWorld (household tasks) and Room-to-Room (R2R) (navigation).

ALFWorld Results

• Performance: PRISM significantly outperforms state-of-the-art image-based models. The PRISM BC+RL variant achieved an average Success Rate (SR) of 80%, surpassing the Idefics2 8B baseline by 12 points and the VIPERBC+RL baseline by 5 points.
• Comparison to Oracle: PRISM reduced the performance gap with methods using privileged textual supervision (e.g., EMMA, EMAC+) to a negligible margin, achieving comparable performance despite lacking access to object IDs and occluded states.
• Ablation Studies:
  • Architecture: PRISM outperformed monolithic VLMs and static modular VLM-LLM architectures ("Raw perception") by 6–8% on average.
  • Merging Strategy: The LLM synthesis approach outperformed both "QA-Only" and "CONCAT" (naive concatenation) strategies, confirming that synthesizing global context with task-specific details is superior to simple aggregation.
  • Question Quality: PRISM's dynamically generated questions achieved performance within 2% of human-annotated oracle questions, demonstrating that the LLM can effectively identify information needs without predefined priors.
  • Perception Accuracy: Interactive goal-oriented perception significantly improved the accuracy of scene descriptions (measured by RougeL, METEOR, BERTScore, and LLM-as-a-judge) compared to both raw perception and goal-aware perception (which suffered from hallucinations).

R2R Results

• PRISM achieved the highest Success Rate (46.08%) and Success weighted by Path Length (SPL 43.14) among all tested methods, including re-implementations of DiscussNav and LangNav.
• The framework demonstrated superior navigation efficiency, reducing Navigation Error (NE) by over 1.4 points compared to re-implemented baselines, highlighting its ability to filter noise and leverage task-relevant cues for spatial reasoning.

5. Significance and Claims

The paper claims that PRISM represents a significant step toward autonomous embodied agents by effectively bridging the gap between perception and reasoning. The authors argue that:

• Interleaved Reasoning is Essential: The tight coupling of perception and decision-making via DQA compensates for the perceptual limitations of standalone VLMs.
• Autonomy over Priors: The framework proves that high-performance embodied agents can be built without relying on unrealistic, handcrafted questions or privileged textual environments.
• Robustness: The method is robust across different VLM backbones (Idefics2, Qwen3-VL) and task domains (manipulation and navigation).

The authors conclude that while their approach improves interpretability and autonomy, it inherits biases from large-scale pre-trained models. Future work is identified as integrating these interleaved capabilities with lower-level control systems, such as robotics, to bridge the gap between decision-making and physical action.