Technical Summary: RevengeBench

Problem Definition

The paper addresses a computational analogue to the scientific inverse problem: inferring hidden mechanisms from observable behavior. Specifically, it asks whether a learner can reconstruct an executable decision program (a policy) given only the behavioral traces of an opaque target agent in a game environment. Unlike standard inverse reinforcement learning or policy imitation which often assume access to a fixed corpus of demonstrations or direct query interfaces, this setting restricts the learner to an intermediate epistemic regime. The learner cannot query arbitrary target states directly; instead, it must infer the target's logic through passive observation of the target playing against sampled opponents and, optionally, by designing active "probes"—custom opponent policies intended to elicit specific, informative behaviors from the target. The goal is to output a single, runnable policy that faithfully reproduces the target's actions across held-out trajectories.

Methodology

Benchmark Construction (REVENGEBENCH)

The authors introduce REVENGEBENCH, a benchmark comprising 75 hidden target policies generated by Large Language Models (LLMs) for the CodeClash tournament. These policies span five distinct game environments:

1. BattleSnake (Discrete, grid-based, sequential)
2. Halite (Continuous control, multi-unit)
3. Poker (Partial observability, stochastic)
4. RoboCode (Java-based, continuous)
5. RobotRumble (JavaScript, multi-unit)

The policies vary in complexity (up to ~500 lines of code) and are written in Python, C, Java, and JavaScript. For each environment, 15 high-Elo policies are designated as "targets," while the remaining policies form an opponent pool.

Evaluation Protocol

The evaluation follows a closed-loop inference process over $R=5$ refinement rounds:

1. Observation: The learner observes trajectories of the target policy playing against sampled opponents.
2. Hypothesis Generation: The learner submits an executable policy (hypothesis) starting from a simplistic baseline.
3. Active Probing (Optional): In the "Active" regime, the learner can design up to $B=5$ probe opponents per round to force the target into specific states or behaviors, aiming to reduce uncertainty in the hypothesis.
4. Refinement: The learner iteratively edits its code based on the observed mismatches between its hypothesis and the target's actual actions.

Metrics

Recovery quality is measured using a per-state action distance ($d_g$), which quantifies the dissimilarity between the learner's action and the target's action. This metric is graded (e.g., betting 90 chips is closer to 100 than folding) rather than binary. The primary score is the distance reduction ($\Delta$), defined as the fraction of the initial behavioral distance (from a random policy) that is closed by the final recovered policy.

Experimental Setup

The study evaluates 12 frontier LLMs (including GPT-5.5, GPT-5, DeepSeek v4 Pro, Gemma 4, and Qwen3.6) using the mini-SWE-agent scaffold, a minimal bash-based interface that forces models to interact with the environment via file operations and script execution. The study compares passive observation against active probing and evaluates the downstream utility of recovered policies in Player-vs-Player (PvP) tournaments.

Key Contributions

1. REVENGEBENCH Benchmark: A novel benchmark of 75 executable policies across five diverse game environments, specifically designed to test the reverse engineering of programmatic strategies from behavioral traces.
2. Constrained-Intervention Recovery Protocol: A formalization of the inverse problem where intervention is restricted to designing opponent policies (probes) rather than direct state queries, modeling a realistic scientific inference scenario.
3. Comprehensive Empirical Evaluation: An evaluation of 12 frontier LLMs demonstrating that:
   • Recovery quality varies significantly (34% to 72% of initial distance closed).
   • Active probing yields benefits primarily for stronger models capable of designing effective experiments.
   • Recovered code carries actionable signal, providing a measurable competitive advantage in downstream PvP tournaments, particularly for weaker models that struggle to devise counter-strategies from scratch.

Results

Strategy Recovery Performance

• Variability: Recovery performance varies substantially across models. The best model (GPT-5.5) closes approximately 72% of the initial behavioral distance, while the weakest (GPT-oss-120b) closes only ~34%.
• Environment Differences: Performance is not uniform across games. Poker is the easiest environment (most models exceed 60% recovery), while RobotRumble shows the widest performance spread (5–72%).
• Cost-Quality Trade-off: There is no strong correlation between API cost and recovery quality; some lower-cost models (e.g., Gemma-4 31B) achieve competitive recovery rates (~51%) at a fraction of the cost of top-tier models.

Observation vs. Intervention

• Active Probing: Active probing improves recovery in 16 of 20 model-game pairs, but the gains are concentrated in stronger models (e.g., DeepSeek v4 Pro sees a mean boost of +0.052). Weaker models often show near-zero gains, suggesting they lack the capability to design informative probes.
• Baselines: Raw behavioral traces are the most effective input channel. Replacing traces with natural language summaries (Description-only) generally degrades performance, and Bayesian Program Inference (BPI) underperforms iterative refinement for most models.

Downstream Utility

In PvP tournaments, agents armed with recovered code consistently outperform "blind" agents (those with only game rules) and underperform "oracle" agents (those with the true source code). Crucially, weaker challenger models derive the largest relative gains from recovered intelligence, suggesting that even imperfect reconstructions capture structural information necessary for effective counter-strategies.

Robustness and Reliability

• Arena Reliability: Poker and BattleSnake yield stable rankings across runs and evaluators (high intra-class consistency). Halite, RoboCode, and RobotRumble exhibit higher variance, requiring multi-run aggregation for reliable comparison.
• Failure Modes: Common failure patterns include exhausting the step budget due to repetitive debugging loops, submitting invalid code that crashes evaluation, and "regression" where later edits worsen the policy compared to earlier rounds.

Significance and Claims

The paper positions the behavioral recovery of programmatic policies as a tractable inverse problem in code-space. Its significance lies in three areas:

1. Opponent Modeling: It demonstrates that LLM agents can infer hidden decision procedures from observation, a capability with direct value for multi-agent workflows where anticipating opaque agent behavior is critical.
2. Policy Interpretability: The benchmark provides a testbed for understanding when behavioral observation is sufficient for mechanism recovery and where active intervention is required.
3. Scientific Analogue: By framing the task as a computational analogue to scientific discovery (inferring latent mechanisms from perturbed observations), the work bridges the gap between agent benchmarks and the broader question of inferring latent structures from data.

The authors conclude that while exact source recovery is not always necessary for downstream utility, the ability to reconstruct executable policies from behavior represents a significant step toward interpretable and strategic AI agents. They note limitations, including the synthetic nature of the targets (which do not adapt or conceal strategies) and the fact that behavioral equivalence does not guarantee structural identity.