Learning Explicit Behavioral Models with Adaptive Questions and World-Model Probes

Technical Summary: Learning Explicit Behavioral Models with Adaptive Questions and World-Model Probes

1. Problem Definition

Interactive agents trained solely on task return (scalar rewards) often achieve high scores by exploiting narrow timing patterns, local shortcuts, or environment-specific quirks without developing a robust representation of the underlying mechanisms. This "high-score, low-understanding" behavior creates a hidden blind spot: the agent's policy appears successful but remains brittle when environment dynamics change, leading to slow recovery and difficult diagnosis.

Current approaches, such as LLM-based reflection or policy-code repair, typically revise behavior only after training or rely on task success as the primary judge of utility. Furthermore, while world-understanding benchmarks exist, they are usually static and applied post-hoc, rarely becoming part of the mechanism that selects, repairs, or rejects a policy during the learning process. The core challenge is to optimize "understanding" as a testable, executable object within the policy-learning loop, rather than as a rhetorical byproduct.

2. Methodology: The Explicit Symbolic Behavioral Model (ESBM)

The paper introduces the Explicit Symbolic Behavioral Model (ESBM), a trainable behavioral model that couples task performance with evidence-grounded question answering and executable mechanism prediction. The ESBM is not a neural network acting as a policy; rather, it is an explicit symbolic structure consisting of:

• Typed Predicates: Map raw observations and memory to semantic facts.
• Weighted Rules: Map facts to actions or option calls.
• Bounded Options: Encapsulate multi-step procedures (e.g., routes, rescue sequences) with defined contracts and termination conditions.
• Mechanism Memory ($K$): An abstract world model that predicts symbolic events, object changes, rewards, and terminal consequences under action interventions.

The Challenger–Optimizer Loop
The learning process is driven by an adversarial yet constructive loop:

1. Challenger (Probe Generation): After each rollout, the challenger analyzes failures, recent symbolic edits, QA errors, and transition-prediction errors to generate two forms of pressure:
   • Adaptive Questions (QA): Grounded questions about entities, hazards, rewards, and counterfactuals. Answers must be supported by the ESBM's explicit context (predicates, rules, mechanisms).
   • Active World-Model Probes: The system branches from a matched checkpoint into two paths: the real environment and the ESBM's abstract model. Both execute the same action sequence (observed, counterfactual, or mixed). The ESBM's mechanism memory must predict the resulting symbolic events, score changes, and object states.
2. Optimizer (Edit Proposal): An LLM acts as a backend to propose local, typed edits to the ESBM (e.g., revising a rule condition, updating a mechanism entry) based on the challenger's feedback.
3. Verifier Gate (Selection): A multi-criterion acceptance rule evaluates candidate models. A candidate is accepted only if it improves at least one validated dimension (Task Score, QA Accuracy, Passive/Active World-Model Consistency) while staying within regression tolerances on protected dimensions. This prevents ungrounded explanations from being accepted even if they improve the score.

3. Key Contributions

• ESBM Definition: The authors define the ESBM as an explicit optimization object combining state abstraction, symbolic action selection, bounded options, and executable mechanism memory.
• Adaptive Training Mechanism: They introduce policy-adaptive, evidence-grounded questions and active world-model probes that convert model failures into reusable understanding constraints, moving QA from post-hoc evaluation to the core learning loop.
• Multi-Criterion Acceptance: A novel acceptance rule is formulated that jointly evaluates task performance, QA accuracy, and active interventional world-model consistency, ensuring that updates improve both behavior and mechanistic understanding.

4. Experimental Results

The approach was evaluated on three Atari-style JAXAtari environments (Kangaroo, Seaquest, Kingkong), which require route planning, resource management, and hazard avoidance.

• Task Performance (RQ1): ESBM achieved competitive or superior scores compared to model-free baselines (DQN, PPO), neuro-symbolic learners (NUDGE, BlendRL), and a score-only symbolic reference (EvoSymbol). Notably, ESBM outperformed EvoSymbol (which lacks QA and world-model validation) across all three games.
• Explicit Understanding (RQ2): The learned ESBMs demonstrated high evidence-supported answerability on held-out game-understanding benchmarks. The models could answer questions about world structure, mechanics, and counterfactuals, with the supporting evidence (predicates, rules) explicitly identifiable.
• Executable World-Model Prediction (RQ3): The mechanism layer ($K$) successfully predicted symbolic dynamics under active counterfactual branches, outperforming persistence, constant-dynamics, and decision-tree world-model baselines.
• Recovery After Mechanism Changes (RQ4): When environment dynamics were altered (e.g., faster monkeys, faster oxygen decay, faster bombs), ESBM recovered a significantly larger fraction of the original score compared to PPO. This indicates that explicit mechanistic understanding facilitates faster adaptation to new dynamics.

5. Significance and Claims

The paper claims that adaptive questions and active world-model probes can serve as both training pressure and reusable benchmarks for mechanistic policy learning. By making mechanism understanding an executable object tied to policy updates, the framework successfully separates high-score behavior from symbolically verifiable mechanism knowledge.

The authors argue that this approach addresses the brittleness of score-only policies. Instead of relying on scalar rewards that can hide missing or incorrect mechanisms, ESBM forces the agent to maintain a model that is auditable, queryable, and repairable. The results suggest that optimizing for explicit behavioral models allows agents to not only perform tasks but also to understand why their actions succeed, leading to more robust adaptation when environment dynamics change.

The paper remains modest regarding its scope, noting that the current setting uses Atari-style games as a controlled testbed and that transferring the method to robotics or open-ended environments would require richer state abstraction and stronger verifiers. It also acknowledges dependencies on the quality of generated probes and the limitations of the symbolic vocabulary.