Technical Summary: The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspective

1. Problem Statement

Foundation Model (FM) agents are increasingly deployed for real-world decision-making tasks, ranging from tool-augmented assistants to robotic control. However, these agents suffer from a significant "sim-to-real" gap: they perform well on clean, curated benchmarks but fail when exposed to the distribution shifts, noisy inputs, and stochastic constraints of production environments.

The paper argues that the foundation model community is currently treating agent robustness as a novel phenomenon, effectively "reinventing the wheel" by ignoring mature frameworks established in robotics and classical reinforcement learning (RL). The core problem is the lack of a unified vocabulary and standardized stress-testing benchmarks to diagnose and mitigate these deployment failures. The authors posit that these failures are not fundamentally new but mirror classical discrepancies found in Markov Decision Processes (MDPs).

2. Methodology: A Unified MDP Perspective

The paper proposes formalizing the FM agent evaluation and training gap as a classical sim-to-real problem structured entirely around the four elements of a Markov Decision Process: Observation (O), Action (A), Transition (T), and Reward (R).

The authors operationalize this framework by:

1. Mapping Classical Gaps to FM Agents: They translate traditional RL sim-to-real discrepancies (e.g., sensor noise, actuation delay) into the specific context of FM-controlled agents (e.g., multilingual input, API latency, tool schema mismatches).
2. Defining Benchmarking Paradigms: For each MDP component, the paper proposes specific perturbation strategies to systematically expose vulnerabilities:
   • Observation: Injecting typos, paraphrasing, and multilingual shifts to test robustness against noisy or non-standard inputs.
   • Action: Expanding the action space with "distractor" tools (tools with identical names but incorrect parameters) to test semantic understanding versus surface-level cue reliance.
   • Transition: Simulating stochastic environment responses, such as timeouts, partial API failures, and incomplete JSON payloads, to test autonomous recovery capabilities.
   • Reward: Shifting evaluation metrics from binary accuracy to operational constraints like latency, financial cost, and user satisfaction, forcing agents to optimize for real-world utility rather than just semantic alignment.
3. Empirical Demonstration: The paper cites recent studies on multilingual tool calling to provide concrete evidence of these gaps. Specifically, it highlights the "parameter value language mismatch," where models correctly identify the intent and tool but fail operationally because parameter values are generated in the user's language (e.g., Chinese) while the execution environment expects a fixed convention (e.g., English).

3. Key Contributions

• Unified Formalization: The primary contribution is the proposal to view FM agent robustness through the lens of the four MDP elements (O, A, T, R), providing a structured vocabulary that aligns FM research with established RL and robotics literature.
• Translation of Classical Solutions: The paper advocates for adopting established sim-to-real mitigation techniques—such as domain randomization, grounded action transformation, and delay-aware control—specifically tailored for foundation model agents.
• Concrete Benchmarking Protocols: It outlines a research agenda for designing stress tests that go beyond clean benchmarks. This includes protocols for injecting multilingual noise, distractor tools, and realistic execution failures (timeouts/retries).
• Empirical Evidence of Observation Gaps: By analyzing data from multilingual tool-calling studies (Table 2 in the paper), the authors demonstrate that error rates for non-English languages can be significantly higher (e.g., rising from ~13.5% to ~28.5% for GPT-5 and from ~5.5% to ~46.5% for Qwen-Next-80B) due to mismatches between observation language and execution conventions.

4. Results and Findings

The paper does not present new experimental results generated by the authors but synthesizes existing literature and recent studies to validate their framework:

• Multilingual Discrepancies: Data from recent studies [37] shows that even state-of-the-art models (DeepSeek, GPT-5, Llama 3.1, Qwen, Granite) exhibit significant performance degradation when moving from English-only queries to multilingual inputs. The failure mode is often not a lack of semantic understanding but an operational invalidity caused by language-execution mismatches in parameter values.
• Vulnerability to Distractors: Theoretical analysis suggests that agents trained on clean tool sets are highly susceptible to "distractor" tools in real-world deployments, where overlapping tool names and noisy functional landscapes exist.
• Reward Misalignment: The paper notes that current benchmarks overfit to binary accuracy, ignoring critical real-world objectives like latency and cost, which leads to severe performance degradation when agents are deployed in environments where these constraints are active.

5. Significance and Claims

The paper claims that adopting this unified MDP perspective is critical for preventing fragmented research efforts and accurately measuring deployment readiness. Its significance lies in:

• Paradigm Shift: It aims to shift the field from treating robustness as a novel problem to recognizing it as a known deployment gap with established solutions.
• Standardization: It advocates for a unified vocabulary and standardized stress test benchmarks, which are currently lacking in the foundation model community.
• Trustworthy AI: By bridging the gap between simulation and reality, the framework aims to foster a new generation of highly trustworthy agents capable of maintaining performance despite severe multilingual inputs, transition perturbations, and cost-aware reward constraints.
• Alignment with KDD: The agenda directly advances KDD thematic topics by establishing a rigorous foundation for "Trustworthy and Responsible Data Science" and shaping "Modern AI for reliable real-world Applications."

The authors conclude that while the specific challenges of FM agents are unique, the underlying mechanisms of the sim-to-real gap are classical. Therefore, the path forward involves operationalizing these classical MDP-based solutions to harden agents against the inevitable noise and unpredictability of real-world production environments.