Technical Summary: Learning POMDP World Models from Observations with Language-Model Priors

Problem Statement

Effective agents in partially observable environments must learn an internal model of how the world responds to their actions. While Partially Observable Markov Decision Processes (POMDPs) provide a flexible formalism for such world models, learning them from observation-action trajectories alone is notoriously difficult. Classical approaches (tabular estimators, spectral methods, deep latent models) typically require extensive environment interaction, strong structural assumptions, or access to ground-truth hidden states.

Existing methods that leverage Large Language Models (LLMs) to induce world models often rely on a critical simplifying assumption: access to privileged hidden states during training. Even the closest precursor, POMDP Coder, assumes post-hoc access to ground-truth states after each episode. This limitation renders such methods inapplicable to real-world scenarios where state supervision is unavailable, such as robots operating in cluttered spaces or agents in imperfect-information games. The central question addressed by this work is whether LLM priors are sufficiently powerful to induce executable POMDP models using only observation-action-reward trajectories, without any access to hidden state labels.

Methodology: Pinductor

The authors introduce Pinductor (POMDP-inductor), a framework that induces executable POMDP world models from observation–action–reward trajectories alone. The method operates on a generate–evaluate–refine–plan loop, avoiding direct comparison between predicted states and ground-truth hidden states.

1. Model Proposal

An LLM is prompted with a natural-language environment description, a small offline dataset of trajectories, and a code API defining the state, action, and observation spaces. The LLM generates a candidate executable model $m = (\rho_0^m, T^m, O^m, R^m)$, defining the initial state distribution, transition dynamics, observation model, and reward function.

2. Belief-Based Model Evaluation

Since hidden states are unobserved, Pinductor evaluates candidates using a belief-based likelihood score.

• Particle Filtering: For a candidate model, a set of particles is sampled from the proposed initial state distribution and propagated through the transition model $T^m$ using the observed actions.
• Soft Observation Likelihood: The LLM-generated observation model $O_{LLM}^m$ is often deterministic. To handle mismatches and provide a gradient-like signal, the method employs a distance kernel. The distance $d_{obs}$ between a predicted observation and the realized observation is converted into a soft likelihood:
  $$O^m(o_{t+1} | s_{t+1}, a_t) \propto \exp(-d_{obs}(o_{t+1}^{pred}, o_{t+1}^{real}) / \kappa)$$
• Scoring: The model is scored by the expected log-likelihood of the observed trajectory under the belief distribution induced by the particle filter:
  $$L(P^m; \mathcal{D}) = \sum_{\tau \in \mathcal{D}} \sum_{t=0}^{H-1} \mathbb{E}_{s_{t+1} \sim Q_{t+1}^m} [\log O^m(o_{t+1} | s_{t+1}, a_t)]$$
  This score evaluates how well the model's own filtering dynamics can explain the observation stream, without privileged state access.

3. Feedback and Refinement

The method refines models iteratively using structured debugging feedback:

• Global Score: The scalar likelihood score indicates overall fit.
• Local Diagnostics: The system identifies specific failure modes, such as execution errors, trajectory segments with high observation distance, or reward mismatches.
• Disagreement Signals: A "committee" of previously generated transition models is used to compute vote entropy. High-entropy contexts (where candidate models disagree on the next state) are highlighted to the LLM, guiding it to resolve ambiguities in the dynamics.
• Refinement Strategy: Pinductor uses a tree-structured refinement process (UCB1 selection) rather than a single chain of edits, exploring multiple candidate families before selecting a near-best model.

4. Planning

The selected model is used for belief-space planning (e.g., via A*-style planners) during online interaction. New trajectories are appended to the dataset, triggering further refinement rounds.

Key Contributions

1. Observation-Only POMDP Induction: The paper demonstrates that LLM priors are sufficient to induce executable POMDP world models from observation–action–reward trajectories alone, eliminating the need for ground-truth latent state supervision at training or inference time.
2. Belief-Based Model Scoring: The authors introduce a kernel-based likelihood objective that scores candidate programs under their filtered belief distributions. This provides a per-step repair signal computable solely from partial observations.
3. End-to-End Validation: The method is validated across five partially observable MiniGrid tasks. Pinductor matches the performance and sample efficiency of a privileged-state LLM baseline (POMDP Coder) while significantly outperforming non-LLM baselines (including tabular POMDPs granted privileged state access).

Results

• Performance: In MiniGrid environments (Empty, Corners, Lava, Unlock, Four Rooms), Pinductor achieves mean episode rewards and win rates comparable to the privileged-state baseline, despite using strictly less information.
• Sample Efficiency: Pinductor reaches strong performance with very few offline trajectories (as few as 2–10), matching the sample efficiency of state-access methods.
• Belief Quality: During planning, Pinductor's belief states become increasingly concentrated on the true hidden state as observations accumulate, demonstrating that the learned models support informative posterior updates.
• Ablations:
  • LLM Capability: Performance scales with LLM capability; weaker models fail to synthesize usable world models, while stronger models (Qwen 3.6 Plus, Claude Opus 4.7) achieve high performance.
  • Semantic Information: Withholding semantic environment descriptions degrades performance, indicating the method relies on LLM priors grounded in textual information.
  • Stochasticity: The method remains effective in stochastic variants of the environments, though performance drops relative to deterministic settings.

Significance and Limitations

The paper positions language-model priors as a practical tool for sample-efficient world-model learning under partial observability. By removing the requirement for privileged state access, Pinductor extends the reach of LLM-guided model induction to settings characteristic of real-world deployments (e.g., robotics, imperfect-information games).

The authors modestly note several limitations:

• Scope: Evaluation is restricted to MiniGrid environments; transfer to other domains or complex real-world settings remains untested.
• Fixed Components: The observation distance function, planner, and demonstration buffer are currently fixed and not optimized by the LLM.
• Variance: Reliance on LLM API calls introduces high variance in execution time and results.
• Data Collection: Offline datasets are manually collected; the impact of automated data collection strategies is not explored.

The work represents a step toward generalist agents capable of learning internal world models from first-person experience in partially observable environments.