🤖 AI

HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising

The paper proposes HOBA, a hierarchical reinforcement learning framework that integrates a large language model for adaptive hyperparameter tuning, a bias-corrected SARSA agent for dynamic expert model selection, and a diverse expert pool for bid execution, thereby overcoming the limitations of offline-trained bidding systems through significant online adaptability and proven business value in large-scale deployments.

Original authors: Ji Wu, Yunshan Peng, Wentao Bai, Yunke Bai, Wenzheng Shu, Jinan Pang, Yanxiang Zeng, Xialong Liu

Published 2026-07-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ji Wu, Yunshan Peng, Wentao Bai, Yunke Bai, Wenzheng Shu, Jinan Pang, Yanxiang Zeng, Xialong Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a giant, high-speed auction house where millions of people are trying to buy tiny slices of attention every second. This is the world of online advertising. Advertisers want to show their ads to the right people without spending their entire budget in a single minute. To do this, they use "bidding agents"—computer programs that decide how much to pay for an ad slot. Think of these agents like race car drivers. Some drivers are very careful and follow a strict rulebook (like a PID controller), while others are like grandmasters who have studied thousands of past races to learn the best moves (like offline Reinforcement Learning).

The problem is that the race track changes constantly. The weather shifts, other drivers get faster, and the rules of the road evolve. A driver who only follows a static rulebook might crash when the track gets slippery. A driver who tries to learn new moves while racing at 200 miles per hour might spin out of control and run out of gas (budget) in seconds. Scientists have been trying to build a driver that is both safe and smart enough to adapt in real-time, but it's a tricky balancing act. If you let the computer learn too freely, it might make a disastrous mistake. If you don't let it learn at all, it gets left behind when the market changes.

This is where the paper "HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising" steps in. The researchers from Kuaishou Technology propose a new way to manage these racing drivers called HOBA. Instead of trying to teach one single driver how to do everything, they build a three-tiered team that works together like a pit crew, a strategist, and a driver.

At the top of the team is a "Strategist" powered by a Large Language Model (LLM). Imagine this as a wise coach who looks at the big picture once every hour. This coach doesn't decide the exact speed for every turn; instead, they look at the weather, the fuel levels, and the competition, then write a new set of instructions for the team. They might say, "Today, be a bit more aggressive," or "Keep the spending slow and steady." This coach learns from a memory bank of past races, using a "Think-Act-Observe-Reflect" loop to get smarter over time.

In the middle is a "Tactical Manager," a smart agent that checks in every two minutes. Its job is to pick the best "driver" from a garage of pre-trained experts. The garage contains different types of drivers: one is great at quick corrections (PID), another is good at long-term planning (MPC), and others are experts who learned from massive datasets (like IQL or Decision Transformers). The Tactical Manager doesn't guess randomly; it uses a special "causal adjustment" tool to make sure it isn't fooled by luck. For example, if a driver happened to win a race just because the weather was perfect, the manager knows not to blame the driver for that win. It picks the driver that is truly the best fit for the current moment.

At the bottom are the "Drivers" themselves. These are the experts who actually place the bids for every single ad auction, which happens in milliseconds. They follow the rules set by the Strategist and the choice made by the Tactical Manager. Crucially, these drivers don't change their own brains while racing. They are safe, pre-tested tools. The "learning" happens only in the middle layer, where the team decides which driver to use. This keeps the system safe from making wild, budget-busting mistakes while still allowing it to adapt quickly to a changing market.

The researchers tested this system in two ways. First, they ran it in a simulated environment called AuctionNet, which is like a video game version of the ad market. In these tests, HOBA consistently beat the best existing methods, improving performance by up to 12% in difficult, unpredictable scenarios. Second, and most importantly, they tried it in the real world. They ran a massive test on a live advertising platform with thousands of campaigns and millions of dollars in budget.

The results were impressive. In the real-world test, HOBA helped advertisers hit their cost targets 3.6% more often than the old system. This means they got more value for their money without overspending. The system also increased the total value of conversions by 8.1% and improved the return on investment by 3.3%. Perhaps most importantly, it did all this without crashing the budget or causing chaos. The system proved that by splitting the job into a strategic coach, a tactical manager, and a team of specialized drivers, you can create an advertising system that is both safe and incredibly adaptable. It's a reminder that sometimes, the smartest way to win a race isn't to have one super-driver, but to have a perfect team working together.

Technical Summary: HOBA – Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising

1. Problem Statement

Online advertising bidding systems operate in non-stationary auction markets where competitor strategies, user behaviors, and platform dynamics evolve continuously. Current industry practices rely on offline-trained expert models (e.g., PID controllers, Model Predictive Control, and offline Reinforcement Learning policies) but face two critical limitations:

Lack of Online Adaptability: Expert parameters remain fixed post-deployment, preventing response to market shifts without costly retraining.
Manual Hyperparameter Tuning: Systems require labor-intensive, campaign-specific tuning of bid bounds, budget pacing rates, and exploration coefficients.

Enabling safe online learning is challenging. On-policy methods (e.g., PPO, A3C) risk catastrophic exploration that can exhaust budgets rapidly, while off-policy methods suffer from action collapse and distributional shift when deployed in environments different from their training data. Furthermore, existing approaches often conflate strategic planning with tactical execution, lacking interpretability.

2. Methodology: HOBA Framework

The authors propose HOBA (Hierarchical On-Policy Bidding Agents), a framework that decomposes the bidding problem into three hierarchical tiers operating at distinct time scales. This design confines online learning to discrete expert selection rather than continuous bid optimization, significantly reducing exploration risk while maintaining adaptability.

2.1 High-Level Layer: Strategic Reasoning (Hourly Cycle)

Agent: A Large Language Model (LLM) agent.
Function: Infers optimal hyperparameters ( $\theta_t$ ) from campaign context via a Think-Act-Observe-Reflect loop.
Mechanism:
- Think: The LLM retrieves similar historical experiences from a memory bank using semantic similarity and performs structured reasoning to identify root causes of performance deviations.
- Act: Generates hyperparameters including bid bounds, response modes (conservative/aggressive), control targets (CPA, pacing rate), and exploration bonuses.
- Observe & Reflect: After a one-hour execution window, the system aggregates observations, computes deviations from predictions, and the LLM evaluates decision quality to generate structured lessons for future retrieval.
Safety: Outputs are strictly clipped to predefined safe ranges, and JSON schema enforcement ensures valid outputs.

2.2 Mid-Level Layer: Tactical Selection (2-Minute Cycle)

Agent: A SARSA (State-Action-Reward-State-Action) agent.
Function: Dynamically selects the best expert model from a pool of pre-validated experts based on current market context.
Causal Adjustment: To address confounding bias in historical data (where experts were preferentially selected in favorable contexts), the agent employs a Doubly Robust Estimator. This combines an outcome model (XGBoost) and a propensity model (Logistic Regression) to estimate the causal effect of selecting a specific expert, ensuring the Q-learning process does not inherit spurious correlations.
Exploration: Utilizes Upper Confidence Bound (UCB) exploration modulated by the high-level LLM's exploration bonus.

2.3 Low-Level Layer: Operational Execution (Per-Auction)

Components: A dynamic expert pool containing offline-trained models:
- PID: For fast error correction.
- MPC: For explicit budget planning over a future horizon.
- Offline RL Models: CQL (Conservative Q-Learning), IQL (Implicit Q-Learning), and DT (Decision Transformer) for learning from logged data.
Function: Executes bids per auction under constraints imposed by the high-level layer.
Safety Mechanism: Bids are subject to double-clipping: first against strategic bounds (from the LLM) and second against volatility limits (e.g., $\pm 20\%$ of the previous bid). This guarantees all bids satisfy budget and cost-per-acquisition (CPA) constraints regardless of the selected expert's behavior.
Adaptation: The pool dynamically expands with fine-tuned variants of top-performing experts and prunes underperforming ones, though the core online learning remains in the selection layer.

3. Key Contributions

Hierarchical Multi-Agent Framework: The first framework to decouple strategic reasoning, expert selection, and bid execution across time scales (hourly, minute, per-auction), treating each layer as an autonomous agent within a shared revenue constraint.
Safe Online Adaptation: Introduces LLM-guided hyperparameter optimization with natural language reasoning and causal-adjusted on-policy expert selection. This approach achieves safety by restricting online learning to discrete model selection (5–10 validated models) rather than continuous bid optimization.
Causal Debiasing: Incorporates doubly robust causal adjustment in the mid-level agent to eliminate selection bias from historical data, ensuring robust expert selection in non-stationary markets.
Empirical Validation: Demonstrates practical viability through large-scale A/B tests, achieving significant business value improvements.

4. Experimental Results

The framework was evaluated on the AuctionNet benchmark and via a large-scale online A/B test on a major advertising platform.

4.1 Offline Benchmarks (AuctionNet)

Performance: HOBA consistently outperformed state-of-the-art baselines (including PID, MPC, BCQ, CQL, IQL, DT, and HRL-Bid) across both standard and sparse settings.
Improvements: Achieved score improvements ranging from +3.0% (standard, 50% budget) to +12.1% (sparse, 100% budget).
Ablation Studies:
- Removing SARSA-based expert selection caused the largest performance drop (-10.7%), confirming that no single expert dominates all market conditions.
- Removing the high-level LLM degraded performance by 7.1%, highlighting the value of adaptive hyperparameter inference.
- Replacing SARSA with off-policy DQN resulted in a 6.4% loss and increased switching instability, validating the stability of on-policy learning in non-stationary environments.
- Removing causal adjustment led to a 4.9% loss, demonstrating the necessity of debiasing historical data.

4.2 Online A/B Testing

Deployed across 100,000+ campaigns with a phased rollout (5% $\to$ 20% $\to$ 100% traffic), HOBA demonstrated:

Target Cost Achievement: +3.6% improvement (90.0% $\to$ 93.2%), indicating better adherence to advertiser CPA targets.
Conversion Value: +8.1% increase ($1.50M additional value over 7 days).
ROI: +3.3% improvement (2.15 $\to$ 2.22).
Budget Completion: +3.8% improvement (94.2% $\to$ 97.8%), reducing underspend waste.
Advertiser Satisfaction: Increased by 7.6%.
Efficiency: The system maintained P99 latency of 38ms (well within the 100ms requirement) and reduced operational costs by 96% compared to manual hyperparameter tuning.

5. Significance and Claims

The paper claims that HOBA successfully addresses the fundamental safety-adaptability tradeoff in online advertising. By decomposing the problem into a hierarchical structure, the system restricts online learning to discrete, safe decisions (selecting an expert) rather than risky continuous optimization (generating a bid).

The authors emphasize that this design allows for:

Interpretability: Strategic oversight is provided through LLM reasoning traces, and expert contributions are quantified via causal analysis.
Robustness: The system maintains performance across varying traffic volumes, user demographics, and competitive landscapes, including cold-start scenarios where it outperformed baselines by 48% in ROI improvement.
Scalability: The framework has been validated in production serving billions of daily auctions, proving that hierarchical multi-agent bidding is a viable paradigm for adaptive online advertising.

The work concludes that while future directions include meta-learning for rare campaign types and multi-objective optimization, the current framework provides a robust, interpretable, and highly effective solution for real-world bidding challenges.