🤖 AI

Do Enterprise Systems Need Learned World Models? The Importance of Context to Infer Dynamics

This paper argues that in configurable enterprise systems, agents should prioritize runtime discovery of active business logic over relying solely on brittle, offline-trained world models, demonstrating through the new CascadeBench benchmark that this approach significantly improves robustness against deployment shifts.

Original authors: Jishnu Sethumadhavan Nair, Patrice Bechard, Rishabh Maheshwary, Surajit Dasgupta, Sravan Ramachandran, Aakash Bhagat, Shruthan Radhakrishna, Pulkit Pattnaik, Johan Obando-Ceron, Shiva Krishna Reddy Ma

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jishnu Sethumadhavan Nair, Patrice Bechard, Rishabh Maheshwary, Surajit Dasgupta, Sravan Ramachandran, Aakash Bhagat, Shruthan Radhakrishna, Pulkit Pattnaik, Johan Obando-Ceron, Shiva Krishna Reddy Malay, Sagar Davasam, Seganrasan Subramanian, Vipul Mittal, Sridhar Krishna Nemala, Christopher Pal, Srinivas Sunkara, Sai Rajeswar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Do We Need to Memorize the Rules?

Imagine you are a new employee at a massive, complex office building (an Enterprise System). Every time you do something—like filing a report or approving a request—different things happen automatically. Maybe a notification gets sent, a budget is updated, or a manager gets an alert.

In a normal office, these rules are fixed. But in this specific type of enterprise system, every office branch has its own unique set of rules.

The New York branch might send an email when a file is saved.
The London branch might just log it in a spreadsheet.
The Tokyo branch might trigger a meeting invite.

Furthermore, the managers can change these rules at any time. Today, saving a file sends an email; tomorrow, they might change it to send a text message.

The paper asks a simple question: To predict what will happen next, does an AI agent need to "memorize" (learn) all these rules beforehand, or can it just look them up when it needs them?

The Two Approaches

The researchers tested two different ways for an AI to handle this:

1. The "Memorizer" (Learned World Model)

This is like a student who tries to study every single rulebook for every branch of the company before starting work. They try to internalize the logic into their brain.

How it works: The AI is trained on historical data (past actions and results) to build a mental model of how the system works.
The Problem: If the company changes a rule (a "deployment shift"), the student's brain is stuck with the old rules. They might predict an email will be sent, but the new rule says a text will be sent. Because they relied on memory, they get it wrong. They are "brittle" (easy to break) when things change.

2. The "Detective" (Enterprise Discovery Agent)

This is like a new employee who doesn't try to memorize the rulebooks. Instead, they keep a phone handy.

How it works: When the employee needs to know what will happen if they file a report, they don't guess based on memory. They immediately call the office manager or check the current digital rulebook on their screen to see exactly what the current rules say.
The Advantage: Even if the rules change tomorrow, the detective just checks the new rulebook. They are "robust" because they are grounded in the reality of the moment, not a memory of the past.

The Experiment: "CascadeBench"

To test this, the researchers built a playground called CascadeBench.

The Setup: They created thousands of fake office scenarios with different, complex rules. Some rules were simple (like "if X, then Y"). Others were complex chains (like "if X, then Y, which triggers Z, which triggers a meeting").
The Test: They asked different AIs to predict the outcome of an action.
- Some AIs had to rely on their training (Memorizers).
- Some AIs were allowed to look up the rules in real-time (Detectives).
- Some AIs were given the rulebook right in front of them (The "Oracle").

What They Found

The results were clear and surprising:

Memorizers are good at home, but bad away: The "Memorizer" AIs did very well when tested on the exact same rules they studied. But as soon as the rules changed slightly (a new branch, a new configuration), their performance crashed. They couldn't adapt.
The Detective wins: The "Discovery Agent" (the one that looked up rules in real-time) performed much better when the rules changed. Even though they didn't "know" the rules by heart, they knew how to find them instantly.
The "Oracle" limit: When the AIs were given the rulebook directly (without having to search for it), they did the best. This proved that the information was there; the problem was just that the "Memorizers" couldn't access the right information when the context changed.

The Three Levels of Difficulty

The researchers also found that some tasks are easier than others:

Level 1 (The Basics): Simple rules based on the database structure (e.g., "If you create a user, they get a default ID"). Even the "Memorizers" could guess these okay because they are standard.
Level 2 (The Chains): Complex chains of rules (e.g., "If you change the priority, it triggers a workflow, which triggers a notification"). Here, the "Memorizers" failed badly. The "Detectives" succeeded because they could trace the chain in the current rulebook.
Level 3 (The Hidden Stuff): Sometimes, the outcome depends on the computer's internal timing or hidden engine behaviors that aren't written in the rulebook. Even the "Detectives" struggled here because the answer literally wasn't in the book they could read.

The Conclusion

The paper concludes that for enterprise systems, you shouldn't rely solely on an AI that has "memorized" the world.

Because business rules are constantly changing and specific to each customer, the best strategy is a hybrid one:

Don't just train the AI on past data.
Give the AI the ability to look up the current rules at the moment it needs to make a decision.

Think of it like driving a car. A "Memorizer" tries to remember every traffic light and stop sign on the route. If a construction crew moves a sign, they crash. A "Discovery Agent" looks at the road signs in real-time. If the sign moves, they see it and adjust immediately.

In short: In a world where the rules change often, it's better to be a detective who can read the signs than a student who tries to memorize them all.

Technical Summary: Do Enterprise Systems Need Learned World Models?

Problem Statement

Enterprise systems differ fundamentally from traditional environments (e.g., robotics or video games) because their transition dynamics are not fixed but are defined by tenant-specific configuration artifacts (business rules, workflows, approval policies) that vary across deployments and evolve over time. In such settings, the same action $a_t$ applied to a state $s_t$ can yield different outcomes $s_{t+1}$ depending on the active instance configuration $c$ .

The paper addresses a critical question in the world-models literature: When the rules governing the environment are readable and accessible at inference time, does an agent still need to learn them internally?

Existing approaches rely on learned world models that internalize dynamics from historical transition data. However, the authors argue that models trained solely on historical transitions become brittle under deployment shifts (changes in configuration or tenant logic). If the rules are explicitly available in the system's configuration, relying on a static, internalized representation may introduce unnecessary approximation errors and reduce robustness.

Methodology

1. Formalization: Contextual Transition Models

The authors formalize enterprise dynamics as a contextual transition model:
$s_{t+1} \sim P(s_{t+1} | s_t, a_t, c)$
where $c$ is the explicit, readable configuration of the specific instance. Unlike standard world models where $c$ is fixed and unknown, enterprise $c$ is mutable and inspectable. The paper distinguishes three levels of transition complexity:

Tier 1 (Schema-Deterministic): Effects determined by data dictionary defaults and constraints.
Tier 2 (Rule-Composable): Effects determined by chains of business rules (cascades).
Tier 3 (Execution-Inferred): Effects dependent on engine-internal behaviors (e.g., execution order, race conditions) not fully exposed in static artifacts.

2. Benchmark: CascadeBench

To evaluate agents under controlled configuration and deployment shifts, the authors introduce CascadeBench.

Construction: Built on synthetic schemas (to prevent memorization of real-world table structures) generated from a catalog of 1,596 business rule patterns across 6 industries.
Ground Truth: Data is collected by executing actions on live ServiceNow instances (not simulated), capturing real audit logs ( $sys\_audit$ ) to record state diffs and cascade paths.
Evaluation: Models must predict field-level state changes ( $s_{t+1}$ ) given a current state ( $s_t$ ) and action ( $a_t$ ). The benchmark isolates reasoning from parametric memorization by providing context (schemas, rules) or withholding it.

3. Proposed Approach: Enterprise Discovery Agents

The paper proposes Enterprise Discovery Agents, which do not attempt to internalize dynamics during training. Instead, they:

Retrieve: At inference time, query the live instance configuration $c$ (e.g., business rules, current record states, SLA definitions) to recover relevant transition logic.
Reason: Use a frozen language model to reason over the retrieved subset $\tilde{c}$ to predict the next state.
Iterate: For multi-step rollouts, the agent re-queries the system at each step to ground predictions in the current state, mitigating error compounding.

4. Comparative Baselines

The study compares three approaches:

Prompted Baseline: Frozen LLMs predicting transitions from static context (with or without provided business rules).
Learned Enterprise World Model: Models fine-tuned (via LoRA) on historical $(s_t, a_t, s_{t+1})$ tuples to internalize dynamics.
Discovery Agent: The retrieval-based approach described above.

Key Results

1. Internalization vs. Retrieval

In-Distribution Performance: Fine-tuned world models perform exceptionally well on data drawn from the same distribution as their training set (e.g., 91.6% IoU for Gemma-4-31B in-distribution).
Out-of-Distribution Degradation: Under deployment shift (synthetic schemas and unseen configurations), fine-tuned models degrade significantly, dropping to performance levels comparable to prompted baselines (~40% IoU).
Discovery Robustness: Discovery agents, which retrieve rules at runtime, maintain robust performance across shifts. They consistently outperform prompted baselines and match the performance of "Oracle" models (which are given all rules in the prompt) on Tier 1 and Tier 2 transitions.

2. Impact of Business Rules

Rule Dependency: Removing business rules from the prompt causes a uniform collapse in performance across all model classes (from ~40% IoU to ~10% IoU). This confirms that enterprise dynamics are driven by rules, not pre-trained knowledge.
Retrieval vs. Memorization: For frontier models, runtime retrieval recovers a large fraction of the "Oracle" signal (reaching mid-20s to low-30s IoU without training on the target instance), whereas internalization alone fails to generalize.

3. Multi-Step Prediction

Error Compounding: In multi-step rollouts (horizons $k=1$ to $5$), performance degrades for all methods as depth increases.
Grounding Advantage: Discovery agents consistently outperform prompted baselines at all horizons. The ability to re-query the live instance at each step refreshes the model's view of the configuration, reducing the accumulation of errors compared to relying solely on prior predicted states.

4. Tier-Stratified Analysis

Tier 1 & 2: Discovery agents achieve near-Oracle performance, effectively recovering hidden rules and cascades.
Tier 3: A performance gap remains for Tier 3 (execution-inferred dynamics), where outcomes depend on engine-internal behaviors not fully exposed in configuration. This suggests a structural limit to what can be recovered from configuration alone.

Significance and Claims

The paper argues that enterprise world modeling should not rely solely on fixed, internalized dynamics. Instead, in configurable environments where transition logic is readable, agents should incorporate mechanisms for runtime discovery.

Complementarity: Discovery agents are not a replacement for learned world models but a necessary complement. The authors suggest a hybrid future where agents combine learned priors (for efficiency and pattern recognition) with runtime retrieval (for robustness against configuration shifts).
Shift in Paradigm: The work challenges the assumption that environment dynamics must be "learned" in the traditional sense. In enterprise settings, the "world model" can be partially externalized, with the agent acting as a dynamic reasoner over live system configurations rather than a static simulator.
Limitations: The authors note that discovery relies on the inspectability of rules (access controls can degrade performance to the prompted baseline) and the tool-use capabilities of the underlying model. Additionally, Tier 3 dynamics remain a challenge, indicating that some aspects of enterprise behavior are not fully recoverable from static configuration artifacts.

In conclusion, the paper demonstrates that for enterprise systems, grounding predictions in the active system instance via runtime discovery is more robust to deployment shifts than relying on offline-trained internalized world models.