🤖 machine learning

AIGB-R1: Self-Evolving Generative Auto-Bidding via Hierarchical Planner-Executor Optimization

This paper introduces AIGB-R1, a hierarchical self-evolving auto-bidding framework that leverages Large Language Models with a planner-executor architecture and a novel Decoupled Group Relative Policy Optimization (D-GRPO) algorithm to overcome the limitations of existing AI-generated bidding methods by enhancing strategic reasoning, numerical precision, and autonomous optimization through an experience-driven loop.

Original authors: Yuejia Dou, Hesong Wang, Xinyu Zhang, Tianyu Wang, Zhilin Zhang, Chuan Yu, Jian Xu, Bo Zheng, Qi Qi

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuejia Dou, Hesong Wang, Xinyu Zhang, Tianyu Wang, Zhilin Zhang, Chuan Yu, Jian Xu, Bo Zheng, Qi Qi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where every time you scroll through your phone, a silent, high-speed auction is happening. Advertisers are fighting for your attention, throwing digital money at the screen to show you their ads. This is the wild frontier of online advertising, a place where billions of opportunities flash by in the blink of an eye. To win these battles, companies used to rely on humans manually tweaking bids, but that's like trying to catch a speeding bullet with a net made of spaghetti—it's just too slow and messy. So, they turned to computers. First came "Auto-bidding," where algorithms automatically adjust prices to get the best deal. Then came "Generative AI," which tries to learn from past auctions to predict the perfect move. But here's the catch: these AI models are like students who only studied from a single, outdated textbook. They get confused when the real world changes, and they often struggle with the math, sometimes "hallucinating" (making up) numbers that don't make sense.

Enter the new star of the show: Large Language Models (LLMs). You might know them as the chatbots that can write poems or solve riddles. They are incredibly smart at understanding context and planning ahead, but they are terrible at doing quick, precise math and they are too slow for real-time auctions. The big question scientists are asking is: Can we combine the brainpower of a smart planner with the speed of a fast calculator to win these digital auctions? This is exactly what a team of researchers set out to solve, aiming to build an AI that doesn't just guess, but actually thinks its way to victory.

The Brain and the Muscle: AIGB-R1

The researchers propose a new system called AIGB-R1. Think of it as a high-stakes trading team where the roles are perfectly split. In the past, people tried to use one giant AI to do everything—think, calculate, and act. But the authors argue that this is a bad idea because AI is bad at precise numbers and slow at reacting. Instead, AIGB-R1 uses a hierarchical approach, splitting the job into two distinct parts: a Planner and an Executor.

The Planner is the "Big Brain." It's a powerful Large Language Model that acts like a seasoned general or a chess grandmaster. Before the auction even starts, this general looks at the advertiser's budget, their goals, and past performance. It doesn't worry about the tiny details of the next second; instead, it formulates a macro-level strategy. It might say, "Today, we are going to be aggressive and spend fast," or "We need to be conservative and save our money." It writes this strategy down as a clear, structured prompt.

The Executor is the "Muscle." This is a lightweight, super-fast AI model (specifically a Prompt Decision Transformer) that does the actual bidding. It receives the general's strategy prompt and then zooms in on the immediate moment. It looks at the current state of the auction, the remaining budget, and the competition, and it instantly calculates the exact number to bid. Because the Executor is a specialized math model, it doesn't hallucinate numbers, and it reacts in milliseconds. The Planner gives the "what" and "why," and the Executor handles the "how" and "when."

Learning from Experience, Not Just Books

The paper also introduces a clever way for this system to get smarter over time, which they call a self-evolving loop. Most AI systems are trained on old, static data—like studying for a test using a textbook from ten years ago. If the market changes, the AI gets confused. AIGB-R1, however, builds a simulation environment called AuctionNetEnv. Imagine a video game where the AI can play against thousands of other bots, trying out different strategies over and over again.

Here is where it gets really cool. The system doesn't just guess randomly. It keeps a memory bank of its best past moves. If a certain strategy worked well yesterday, the Planner remembers it and tries to refine it today. If a strategy failed, it learns from that mistake. This is the "self-evolving" part: the system learns from its own accumulated experience, constantly tweaking its approach to get closer to the perfect bid.

The Secret Sauce: D-GRPO

One of the biggest challenges in this setup is figuring out who gets the credit (or the blame) when the team wins or loses. Did the General (Planner) give a bad order, or did the Soldier (Executor) fail to follow it? To solve this, the authors invented a new math trick called Decoupled Group Relative Policy Optimization (D-GRPO).

Imagine a coach watching a relay race. If the team loses, the coach needs to know if the first runner was slow or if the second runner dropped the baton. D-GRPO acts like a super-smart coach who can separate the performance of the Planner from the Executor. It looks at the results and says, "The strategy was good, but the execution was off," or "The execution was perfect, but the strategy was wrong." By separating these two signals, the system can train both parts simultaneously without them getting confused, leading to a much more stable and effective learning process.

What They Found

The researchers tested AIGB-R1 on a massive, real-world dataset from Alibaba containing about 480,000 bidding trajectories. They compared their system against many other top-tier AI methods, including those based on Reinforcement Learning and other Generative models.

The results were clear: AIGB-R1 won. It consistently outperformed all the other methods, achieving the highest scores in both standard and challenging "sparse" auction scenarios. The paper suggests that simply using a Large Language Model as a decision-maker isn't enough because of its math limitations, but using it as a strategic planner while letting a specialized model handle the numbers is a game-changer.

The study also showed that every part of their system was necessary. When they removed the "self-evolving" training (letting the AI learn from its own simulations), the performance dropped. When they removed the Planner and just let the Executor guess, it did worse. And when they froze the Planner so it couldn't learn from new experiences, the system wasn't as good as when it could adapt. This proves that the combination of a smart planner, a fast executor, and a loop that learns from experience is the key to unlocking the next level of auto-bidding.

In short, AIGB-R1 suggests that the future of automated advertising isn't about building one giant, all-knowing robot. It's about building a team: a wise strategist who plans the route and a nimble driver who steers the car, both learning together from every mile they travel.

Technical Summary: AIGB-R1

Problem Statement

Auto-bidding is critical for online advertising, where agents must automatically adjust bids to maximize advertiser goals (e.g., conversions) under constraints like budget and Cost-Per-Action (CPA). While the emerging AI-Generated Bidding (AIGB) paradigm utilizes generative models (e.g., Decision Transformers) to learn from offline data, these methods face three primary limitations:

Data Coverage: Performance degrades in out-of-distribution (OOD) states due to the limited mode coverage of offline datasets.
State Understanding: Existing methods condition policies on "return-to-go" without fully understanding specific task states or action values, hindering optimal strategy planning.
Adaptability: Fixed preference distributions in training make it difficult to adapt to diverse advertiser preferences without costly retraining.

While Large Language Models (LLMs) offer superior reasoning and world knowledge to address these gaps, their direct application to auto-bidding is hindered by:

Numerical Precision: LLMs struggle with the precise continuous numerical control required for bidding parameters, often suffering from hallucinations and insufficient precision.
Latency: Online bidding requires millisecond-level responses, whereas large LLMs face a trade-off between decision quality and inference speed.
Credit Assignment: Existing hierarchical attempts often fail to optimize both high-level reasoning and low-level execution end-to-end, relying on offline value functions that suffer from extrapolation errors.

Methodology: AIGB-R1 Framework

The authors propose AIGB-R1, a hierarchical, self-evolving auto-bidding framework that decouples macro-level strategy planning from fine-grained decision-making.

1. Hierarchical Planner-Executor Architecture

High-Level Planner: Utilizes a large LLM to reason about macro-level strategies (e.g., risk preference, budget pacing) for a bidding period based on advertiser context and historical performance. It outputs a structured strategy prompt containing enumerated strategy labels and semantic reasoning. This planning occurs asynchronously before the bidding period begins.
Low-Level Executor: Employs a Prompt Decision Transformer (PDT), a lightweight causal transformer. The Executor takes the Planner's strategy prompt as a conditional prefix and generates precise bidding parameters at each timestep based on observed numerical sequences (states, actions, return-to-go). This design leverages the LLM's reasoning while using a specialized model for precise numerical control and low-latency execution.

2. Two-Stage Training Pipeline

Offline Pre-training: The Executor is pre-trained on offline bidding trajectories. The strategy prompts are annotated by an advanced LLM. The Executor learns a stochastic policy (modeled via log-normal distribution) to maximize the likelihood of the dataset, conditioned on both the prompt and historical sequences.
Post-training Alignment (Self-Evolving Loop): To overcome offline data limitations, the framework employs an interactive bidding simulation environment (AuctionNetEnv) for online strategy rollout.
- Experience-Guided Rollout: For each task, the Planner generates $M$ diverse strategy prompts. For each prompt, the Executor performs $K$ rollouts in the simulation.
- Experience Buffer: A buffer maintains a "best-strategy anchor" and a sliding memory window of recent trajectories for each task, allowing the Planner to refine strategies based on accumulated experience rather than random sampling.

3. Decoupled Group Relative Policy Optimization (D-GRPO)

To enable end-to-end optimization of both the Planner and Executor, the authors propose D-GRPO. Standard Group Relative Policy Optimization (GRPO) fails here because the reward signal couples the contributions of both modules. D-GRPO addresses this via advantage decoupling:

The reward matrix ( $M$ $M$ strategies $\times$ $\times$ $K$ $K$ rollouts) is decomposed into two orthogonal components:
- Inter-group component: The deviation of a strategy's mean reward from the global mean. This signal is used to update the Planner, reflecting the quality of strategy selection.
- Intra-group component: The deviation of a specific trajectory's reward from its strategy's mean. This signal is used to update the Executor, reflecting decision-making performance under a fixed strategy.
This decomposition allows for stable, end-to-end credit assignment without misattributing rewards between layers.

Key Contributions

AIGB-R1 Framework: A hierarchical architecture that fully leverages LLM reasoning for macro-planning while circumventing numerical and latency limitations via a dedicated PDT Executor.
Experience-Driven Self-Evolving Loop: A mechanism that enables autonomous strategy exploration and optimization from accumulated rollout experience, moving beyond blind random sampling.
D-GRPO Algorithm: A novel optimization method that achieves inter-layer credit assignment through advantage decoupling, enabling stable end-to-end training of the hierarchical system.
Interactive Simulation Environment: The construction of AuctionNetEnv to facilitate reliable online strategy rollout and mitigate offline extrapolation errors.

Experimental Results

The framework was evaluated on large-scale public datasets (AuctionNet and AuctionNet-Sparse) containing approximately 480K bidding trajectories.

Performance: AIGB-R1 significantly outperformed state-of-the-art baselines, including RL methods (USCB, CQL, IQL), generative models (Decision Transformers, Diffusion models), and other LLM-based approaches (LBM).
- On AuctionNet, AIGB-R1 achieved a score of 385, surpassing the next best baseline (GAS at 359).
- On AuctionNet-Sparse, it achieved 39.0, outperforming the next best (GAVE at 37.2).
Generalization: The model demonstrated robust performance across varying budget settings (50% to 150% of baseline), consistently outperforming advanced generative baselines.
Ablation Studies:
- Removing the Planner or the post-training stage resulted in significant performance drops, confirming the necessity of both components.
- End-to-end optimization (updating both Planner and Executor) via D-GRPO yielded superior results compared to freezing the Planner, validating the importance of joint training.

Significance

The paper claims that AIGB-R1 provides a promising solution for LLM-empowered auto-bidding by effectively bridging the gap between the reasoning capabilities of LLMs and the precision/latency requirements of real-time bidding systems. By introducing a self-evolving loop and a decoupled optimization algorithm, the framework addresses the critical bottlenecks of offline data coverage, numerical precision, and credit assignment in hierarchical decision-making. The results suggest that decoupling strategy planning from execution, combined with experience-driven self-improvement, is a viable path toward more adaptive and optimal automated bidding systems.