🤖 AI

PACE-Bench: Benchmarking Physics Adaptation via Code Evolution in Dynamic Environments

The paper introduces PACE-Bench, a simulator-grounded benchmark designed to evaluate self-evolving agents' ability to adapt code-based behaviors when physics environments change, revealing that current methods struggle with mechanism redesign and that simulator-grounded reflection outperforms unverified self-revision.

Original authors: Yuhao Zhan, Bingxiang He, Zecong Tang, Chaojun Xiao

Published 2026-08-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuhao Zhan, Bingxiang He, Zecong Tang, Chaojun Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to build a bridge. You show it how to build one that works perfectly on dry, solid ground. The robot learns the rules, writes the code, and builds a sturdy structure. Now, imagine you secretly swap the ground for a sheet of ice. The bridge that worked perfectly yesterday collapses instantly today. This is the daily reality for many smart computer programs: they are great at following instructions in a fixed world, but they often freeze when the rules of the game suddenly change.

Scientists are trying to build "self-evolving agents"—AI systems that can learn from their mistakes and fix themselves without a human needing to hit the "reset" button. Think of these agents like a video game character that doesn't just memorize a map, but actually rewrites its own character sheet after falling into a pit. The big question is: if the physics of the world change (like gravity getting stronger or friction disappearing), can these agents figure out why their old solution failed and invent a brand new one? It's not just about knowing the answer; it's about knowing how to rebuild the engine when the fuel changes.

This is exactly what the new paper, PACE-Bench, sets out to test. The researchers created a giant digital playground called a "benchmark" to see how well these self-fixing robots handle sudden changes in physics. They didn't just ask the AI to solve a puzzle; they gave it a puzzle it had already solved, then secretly changed the rules of the universe and asked, "Can you fix your solution?"

The Game: PACE-Bench

The researchers built 144 different challenges across six different "physics worlds," ranging from simple static structures (like a bridge) to complex, wobbly dynamics (like a swinging pendulum or a vehicle driving through mud).

Here is how the game works:

The Source: An AI agent is given a task and a "source environment" (e.g., build a bridge over a 10-meter gap). It writes a computer program to build a bridge that works.
The Twist: The environment is mutated. The gap might suddenly become 20 meters, the wind might start blowing sideways, or the materials might become brittle. The old bridge code now fails.
The Challenge: The agent has 20 attempts to look at the failure, guess what changed, and rewrite its code to build a new bridge that works in the new conditions. Crucially, the agent isn't told exactly what changed. It has to infer the new physics from the feedback it gets (like "your bridge broke because the joint snapped under too much force").

The Results: Who Passed the Test?

The researchers tested 10 different self-evolving methods (different strategies for how the AI tries to fix itself) using various AI models. The results were eye-opening:

The Benchmark is Hard: Even the smartest AI models struggled. The best combination (a method called Reflexion paired with a large model) only succeeded on about 35.9% of the challenges. Even a top-tier model called GPT-5.5 only solved 66.7% of the easier "Statics" subset. This means the benchmark is far from "solved"; there is still a huge gap in how well AI can adapt to new physical realities.
Reflection Wins, Blind Revision Loses: The most successful strategy was Reflexion. This method forces the AI to pause and think: "I failed because X, so I should try Y." It's like a student reviewing a test to see why they got a question wrong before trying again. In contrast, methods that just kept rewriting code without checking if the new idea made sense (called Self-Refine) often made things worse, compounding errors.
Memory Can Be a Trap: Some methods tried to remember past successes to help with new problems. However, the paper found that this often "anchored" the AI to its old, broken ideas. Instead of exploring new solutions, the AI kept trying to tweak the old bridge that was already doomed, a behavior the authors call Design Fixation.
Searching Without Converging: Other methods tried to explore many different ideas at once (like a tree growing many branches). While this was good at finding some solutions, it often failed to settle on the best one within the 20-attempt limit, getting stuck in a loop of endless experimentation.

The Big Discovery: "Know What" vs. "Know How"

Perhaps the most surprising finding came from a specific experiment. The researchers tried to help the AI by telling it exactly what changed (e.g., "The friction is now 0.5"). You might think this would make the task easy.

It didn't. Even when the AI knew the exact numbers of the new physics, it still couldn't solve the problem. This suggests that the real bottleneck isn't parameter inference (figuring out what the numbers are); it's mechanism redesign (figuring out how to rebuild the structure to work with those numbers). Knowing the wind speed is 50 mph doesn't help if you don't know how to reshape the bridge to withstand it.

Why This Matters

The paper concludes that while AI is getting better at solving static problems, it is still very clumsy at adapting when the world changes. The current "self-evolving" tools are often too rigid, too stuck on their first ideas, or too chaotic to find a stable solution.

The authors suggest that for AI to truly become a lifelong learner, it needs to get better at redesigning mechanisms, not just tweaking parameters. PACE-Bench provides a reproducible way to test this, acting as a "stress test" for the next generation of AI that hopes to operate in our messy, ever-changing real world. For now, the paper suggests we are still far from having an AI that can look at a collapsed bridge on ice and instantly know how to build a better one.

Technical Summary: PACE-Bench

Problem Statement

Existing benchmarks for self-evolving agents primarily evaluate performance under fixed execution conditions or measure the transfer of learned skills to new, held-out tasks. However, they fail to test an agent's ability to adapt a previously successful executable design when the underlying physical environment changes while the goal and interface remain constant. In real-world deployment, operating conditions (e.g., friction, material strength, dynamics) frequently shift, rendering prior solutions obsolete. Current agents struggle to identify obsolete assumptions, preserve useful components, and redesign mechanisms to function under these new physical constraints without explicit instruction on what changed.

Methodology

The authors introduce PACE-Bench (Physics Adaptation via Code Evolution), a simulator-grounded benchmark designed to evaluate self-evolving adaptation in dynamic physical environments.

Benchmark Construction

Scope: The benchmark comprises 144 source-to-target adaptation pairs across six physics domains: Statics, Kinematics, Dynamics, Granular/Fluid Interaction, Cybernetics/Control, and Exotic Physics.
Task Structure: Each base task defines a fixed context (goal, constraints, and primitive APIs) and five environments ( $E_0$ $E_{0}$ to $E_4$ $E_{4}$ ).
- $E_0$ is the source environment where a reference solution succeeds.
- $E_1$ through $E_4$ are mutated target environments where physical parameters (e.g., gravity, friction, joint limits, wind forces) are systematically altered.
- The source solution is guaranteed to fail in all target environments, while a verified reference design exists for each target.
Evaluation Protocol: Agents are given a source solution ( $x_0$ $x_{0}$ ) and a target environment ( $E_k$ $E_{k}$ ). They have a budget of 20 attempts to iteratively revise the code-driven design.
- Agents receive diagnostic sandbox feedback (e.g., peak joint forces, failure timestamps, constraint margins) but are not told which specific physical parameters changed (Change-Hidden setting).
- The agent must infer the hidden changes and redesign the mechanism to pass the target environment.
Dataset Pipeline: The dataset is constructed via a three-phase process involving automated auditing for cross-module consistency, difficulty escalation (ensuring Qwen3-4B fails two independent runs), and human verification by experts in physics/engineering.

Experimental Setup

The authors evaluated ten self-evolving methods across four paradigms using various model sizes (Qwen3-4B/8B/14B/32B, DeepSeek-V4-Pro, GPT-5.5, etc.):

Context-based: Reflexion, Self-Refine.
Memory-augmented: ACE, ExpeL, ReasoningBank.
Inference-time Search: Tree-of-Thoughts (ToT), CodeEvolve.
Parameter-based: SEAL, RAGEN, TTT-Discover.

Metrics: Pass@2 (fraction of pairs where at least one of two runs succeeds) and Score@2 (mean of the best scores from two runs).
Analysis: The study includes code similarity analysis (to detect fixation vs. exploration), error taxonomy (categorizing failures like catastrophic collapse or stagnation), and ablation studies on information disclosure (Change-Exposed) and visual feedback (VLM video).

Key Results

Benchmark Saturation: The benchmark remains far from saturated. The strongest configuration, Reflexion + Qwen3-14B, succeeds on only 35.9% of the full benchmark pairs. Even GPT-5.5 solves only 66.7% of the Statics subset under the full budget.
Paradigm Performance:
- Reflexion (Context-based) consistently outperforms Vanilla and other methods, demonstrating that simulator-grounded reflection is more reliable than unverified self-revision.
- Tree-of-Thoughts (ToT) shows high efficiency (Score/Hr) but struggles to converge within the 20-attempt budget, often exploring without finding a solution.
- Memory-augmented methods (e.g., ACE) often anchor agents to early designs, leading to "Design Fixation" where retrieved experiences constrain stronger models from exploring necessary redesigns.
- Self-Refine performs poorly, as its unverified inner-loop revisions compound errors.
Failure Modes: Agents predominantly fail due to Design Fixation (locking onto a narrow design family) or Stagnation/Exploration (making undirected changes without progress). Model scaling improves reasoning capabilities more than it solves the combinatorial search required for complex physical redesigns.
Information Asymmetry: Revealing the exact changed physical parameters (Change-Exposed) does not significantly raise the performance ceiling for top models. This suggests that the primary bottleneck is mechanism redesign ("know how") rather than parameter inference ("know what").
Visual Feedback: Adding video feedback from a VLM helps memory-based methods (by countering stale memories) but can hinder context-based methods (by scattering focused reasoning).

Significance and Claims

The paper positions PACE-Bench as a reproducible testbed for diagnosing and improving self-evolving agents in changing environments. Its primary contributions are:

A New Evaluation Paradigm: It shifts the focus from generalization to new tasks to the adaptation of existing executable designs under physical shifts, a critical capability for real-world deployment.
Diagnostic Insights: The benchmark reveals that current self-evolving agents struggle with the tension between exploration and exploitation. It highlights that "knowing what changed" is insufficient without the ability to "know how" to redesign the mechanism.
Methodological Guidance: The results suggest that simulator-grounded reflection is a robust strategy, while memory mechanisms can inadvertently constrain stronger models. It also indicates that simply scaling model size does not solve the fundamental difficulty of physical mechanism redesign.

The authors conclude that PACE-Bench provides a necessary foundation for developing agents capable of lifelong adaptation in dynamic physical worlds, noting that current frontier models still face significant challenges in this domain.