Technical Summary: DRIVE (Distributional and Retrieval-Augmented Bidding with Value Evaluation)

1. Problem Statement

The paper addresses critical limitations in applying offline reinforcement learning (RL) and Transformer-based sequence modeling (e.g., Decision Transformer) to real-world auto-bidding systems. While offline RL is necessary due to the prohibitive risks and costs of online exploration in advertising auctions, existing methods face two primary challenges:

1. The "Average Action" Trap: Standard Transformer-based approaches typically employ unimodal regression objectives (e.g., Mean Squared Error). In bidding environments, similar market states often admit multiple distinct yet effective strategies (e.g., aggressive high bids vs. conservative low bids). Unimodal modeling collapses these diverse behaviors into a single suboptimal averaged action, which is neither aggressive enough to win auctions nor conservative enough to control costs.
2. Unreliability in Sparse/Long-Tail Data: Purely parametric models lack explicit mechanisms to retain high-quality historical decisions. Consequently, they generate unreliable actions in low-density regions or under long-tail traffic conditions, despite the presence of high-quality decisions in the logged dataset.

2. Methodology: The DRIVE Framework

The authors propose DRIVE (Distributional and Retrieval-Augmented Bidding with Value Evaluation), a unified Transformer-based framework that decouples candidate action generation from decision-making. The framework integrates three core components:

A. Distributional Action Modeling (GMM Head)

Instead of a deterministic regression head, DRIVE employs a Gaussian Mixture Model (GMM) to model the action space.

• Mechanism: The policy predicts a set of $M$ mixture components, each defined by mixing coefficients ($\alpha_m$), means ($\mu_m$), and variances ($\sigma^2_m$).
• Objective: The model is trained by maximizing the log-likelihood of historical actions. This allows the policy to represent a multi-peaked density, capturing diverse bidding modes simultaneously rather than collapsing them into a single point estimate.
• Inference: During inference, the model stochastically samples a set of candidate actions ($A_{gen}$) from the learned mixture distribution, preserving multimodal bidding patterns.

B. Retrieval-Augmented Candidate Generation

To mitigate the unreliability of parametric models in sparse data regimes, DRIVE incorporates a retrieval mechanism grounded in high-quality historical trajectories.

• Index Construction: A retrieval index ($I$) is built from the offline dataset using contextual state embeddings generated by a lightweight Transformer encoder.
• Retrieval Strategy: At inference time, the current state is encoded, and the system retrieves the $K_{pool}$ nearest neighbors based on cosine similarity. These candidates are then filtered and ranked by their stored Return-to-Go (RTG) values to select the top-$K$ high-quality actions ($A_{ret}$).
• Role: This provides explicit non-parametric support, anchoring decisions to proven high-performing behaviors from the dataset.

C. Value-Based Action Evaluation

To robustly select the final bid from the combined pool of generated and retrieved candidates, DRIVE utilizes a value-based critic.

• Critic Architecture: The framework adopts the Implicit Q-Learning (IQL) paradigm. It learns two Q-functions and a state value function ($V(s)$) strictly within the support of the offline dataset using expectile regression.
• Constraint Awareness: For constrained bidding tasks (e.g., Cost Per Action - CPA), the reward function is shaped to include penalty terms. This ensures the learned value landscape inherently reflects safety constraints, guiding the agent toward feasible regions.
• Selection: The final action ($a^*$) is selected by evaluating all candidates ($A_{cand} = A_{gen} \cup A_{ret}$) using the critic:
  $$a^* = \arg \max_{a \in A_{cand}} \min_{i=1,2} Q_i(s, a)$$

3. Key Contributions

1. Unified Framework: The proposal of DRIVE, which synergizes distributional GMM modeling, retrieval-augmented context, and value-based evaluation to address the "Average Action" trap and data sparsity in offline auto-bidding.
2. Empirical Validation on AuctionNet: Extensive experiments on AuctionNet, a large-scale industrial benchmark featuring stochastic dynamics and strict budget/CPA constraints, demonstrate that DRIVE consistently outperforms state-of-the-art baselines (including CQL, IQL, BCQ, DT, and diffusion-based methods).
3. Generalizability: The framework is shown to be broadly applicable, seamlessly integrating into multiple Transformer-based architectures (DT, CDT, PDiT, BC) and consistently improving performance across standard offline RL benchmarks (D4RL Gym-MuJoCo and Maze2D).

4. Experimental Results

• AuctionNet Performance: DRIVE achieves superior performance across all budget constraints (50% to 150%). Notably, on the AuctionNet-Sparse dataset, where purely generative baselines (like DT and DiffBid) degrade significantly due to data sparsity, DRIVE maintains robust performance by leveraging retrieval to anchor decisions.
• D4RL Benchmarks: DRIVE attains the highest average normalized scores on Gym locomotion tasks and demonstrates significant advantages in the Maze2D domain (e.g., a score of 136.8 on maze2d-medium, outperforming all competitors).
• Ablation Studies:
  • Removing the GMM head (reverting to deterministic) or the retrieval module leads to substantial performance drops.
  • Comparing GMM vs. Diffusion heads shows GMM achieves comparable or better performance with significantly lower inference latency (11ms vs. 223ms), making it more suitable for real-time bidding.
  • Analysis of Q-function multimodality reveals that 17.6% of states are multimodal, where standard DT policies exhibit a 54.7% suboptimal rate, confirming the necessity of distributional modeling.
• Efficiency: While DRIVE incurs additional memory usage (primarily for the FAISS retrieval index), its total inference latency remains under 50ms, satisfying the strict requirements of industrial real-time bidding systems.

5. Significance and Claims

The paper claims that DRIVE provides a robust paradigm for deploying offline RL in bidding environments. By decoupling candidate generation from decision-making, it effectively mitigates the "Average Action" trap and ensures reliable decision-making under sparse and long-tail data regimes. The authors emphasize that their approach does not rely on complex environment modeling or online exploration, making it a practical solution for real-world advertising systems where safety and budget constraints are paramount. The framework's ability to generalize across different Transformer backbones suggests it offers a general solution to mode collapse and sampling instability in sequence modeling for offline RL.