Reinforcement Learning for Execution under Dynamic Fees in a Closed-Loop DEX Simulator

Technical Summary: Reinforcement Learning for Execution under Dynamic Fees in a Closed-Loop DEX Simulator

1. Problem Statement

The paper addresses a fundamental limitation in evaluating trader-facing dynamic fees on Automated Market Makers (AMMs). While dynamic fees are proposed to protect Liquidity Providers (LPs), historical data cannot determine how order flow responds to such fees because:

1. Static Fees: Historical fees have not varied significantly.
2. Latent Types: Trader types and routing decisions are unobserved.
3. Non-Sequential Nature: Replayed historical tapes do not constitute a sequential decision environment where actions influence future states.

Consequently, counterfactual questions—such as "what would order flow do if fees moved against it?"—cannot be answered by historical event studies. Previous attempts to apply Reinforcement Learning (RL) to historical trading tapes failed because the tape lacks an action-dependent transition kernel; an agent's actions in a replay cannot alter the subsequent state of the market.

2. Methodology

To overcome the lack of counterfactual signals in historical data, the authors construct a minimal closed-loop simulator where the signal exists by construction. The methodology is defined by a rigorous evaluation discipline rather than a specific learning algorithm.

The Market Simulator:

• Structure: Two constant-product pools (Uniswap lineage) and an external Centralized Exchange (CEX).
• Agents:
  • Execution Agent: Trades a target order over a finite horizon.
  • Noise Traders: Route flow based on effective execution costs (softmax over cost).
  • Arbitrageur: Closes CEX-AMM gaps in closed-form whenever profitable (minus gas).
• Dynamic Fee Rule: Fees are repriced every step based on an equilibrium-inspired closure (linearization of Nash fee structures). Fees depend on:
  • Own-pool inventory imbalance.
  • Rival-pool state.
  • Oracle price gap.
• Transition Dynamics: The environment is a finite-horizon Markov Decision Process (MDP). The transition function composes the agent's action with market operators (fee update, oracle update, arbitrage, noise flow) in a fixed, declared order. Crucially, the agent's intra-step position (e.g., acting before or after arbitrage) is an explicit design variable.

Evaluation Protocol:

• Objective: Minimize Implementation Shortfall (IS) under a forced-terminal completion rule (all inventory must be executed, with residual inventory liquidated immediately at the final step).
• Benchmark Ladder: Instead of a single baseline, the study compares learners against a "ladder" of tuned heuristics to rule out alternative explanations for any performance edge:
  • Schedule Baselines: TWAP, fastest liquidation, random routing.
  • Lookahead: Tuned one-step control with a mark-to-market urgency term (proxy for future costs).
  • Planners: Deterministic K-step certainty-equivalent planners and stochastic rollout planners.
  • Tabular Learners: On-policy Monte Carlo control with discretized states.
  • DQN: Deep Q-Network with function approximation.
• Rigorous Freeze: All hyperparameters, baseline tuning, and design choices are frozen on a validation block. The final headline results are derived exclusively from a reserved block of 1,000 seeds never seen during development.

3. Key Contributions

1. Closed-Loop Simulator Construction: The paper provides a minimal, reproducible simulator where execution actions directly influence pool inventory, quotes, and dynamic fees, creating a valid sequential decision environment for RL.
2. Evaluation Discipline: It establishes a protocol for evaluating model-free control in closed-loop markets, emphasizing paired comparisons against tuned heuristics, strict completion semantics, and separation of training/validation/final evaluation blocks.
3. Counterfactual Evidence: It generates model-conditioned evidence regarding execution control in AMMs, specifically isolating the impact of dynamic fees on execution strategies.

4. Results

The study evaluates a small DQN against the benchmark ladder on the reserved final block (1,000 seeds) under forced completion.

• Performance Edge: The DQN is the only evaluated policy whose paired improvement over a tuned one-step lookahead benchmark excludes zero.
  • Agent-First Ordering: DQN reduces implementation shortfall by 13.3 bps (basis points) of order notional compared to the lookahead benchmark.
  • Agent-Last Ordering: The edge is 14.9 bps.
  • Randomized Ordering: The edge is 5.6 bps (the smallest but still significant).
• Failure of Alternatives:
  • Planning: Deterministic and stochastic shallow planners fail to close the gap with the lookahead benchmark. Their performance is statistically indistinguishable from or worse than the tuned one-step lookahead.
  • Tabular Learning: A 450-state tabular learner performs worse than lookahead. A 1,440-state tabular learner shows a paired difference of -0.42 bps, but the 95% confidence interval contains zero.
• Attribution of Edge:
  • The advantage is concentrated in dynamic-fee environments. Under constant fees, the paired difference between the DQN and the heuristic is indistinguishable from zero.
  • The DQN learns to time trades into low-fee states and route based on fee gaps, exploiting the correlation between cheap quotes and low fees.
• Robustness:
  • The edge persists across variations in gas costs, arbitrage speed, and noise intensity.
  • Transfer vs. Re-solving: A frozen policy trained under one ordering (e.g., agent-first) loses its edge when evaluated under a different ordering (e.g., agent-last). However, retraining the DQN under the new ordering restores the edge, indicating the loss was due to policy-environment mismatch rather than a lack of exploitable structure.

5. Significance and Claims

The paper is modest in its scope, explicitly defining what the results do and do not support:

• What it Supports:

  • In the specific constructed environment, model-free execution control (DQN) can outperform strong, tuned one-step routing heuristics when strict completion is enforced.
  • The performance advantage is learned specifically from the structure of dynamic fees; it does not exist in constant-fee environments.
  • Shallow planning (deterministic or stochastic) is insufficient to capture the value of dynamic fee timing in this setting.

• What it Does Not Support:

  • Historical Behavior: The results are not evidence about how historical traders would behave, as the data is synthetic.
  • Deployable Profit: The results do not prove that a deployed strategy would be profitable in a live market, as the simulator is a minimal model.
  • Equilibrium Play: The dynamic fee rule is used as a closure principle to make the environment "defend itself," not as a certification that the fee rule is a Nash equilibrium.
  • LP Welfare: The paper does not claim that dynamic fees are optimal for Liquidity Providers.

Conclusion:
The paper demonstrates that in a closed-loop market with dynamic fees, a model-free agent can learn execution strategies that outperform heuristic lookaheads by exploiting fee-state variations. However, this advantage is contingent on the specific market dynamics (dynamic fees) and the alignment between the agent's training environment and the execution ordering. The work serves as a proof-of-concept for using closed-loop simulation to study execution control in AMMs, rather than a blueprint for immediate deployment.