Knowledge-informed Bidding with Dual-process Control for Online Advertising

This paper proposes KBD, a novel bid optimization method that integrates human expertise as inductive biases and employs a dual-process control mechanism combining a fast rule-based PID system with a Decision Transformer to overcome the limitations of black-box models in data-sparse, long-term, and out-of-distribution scenarios.

The Gist

Imagine you are running a lemonade stand in a very busy, chaotic city square. You have a limited amount of money (your budget) and you want to sell as many cups as possible (GMV) without spending more than you earn per cup (tCPA).

Every time a new customer walks by, you have to decide instantly: Do I offer them a discount? Do I raise my price? Do I even bother trying to sell to them?

In the past, companies used simple rules (like "always offer 10% off") or complex computer programs that just looked at yesterday's sales to guess what to do today. But these methods often failed when the weather changed, a festival started, or a new competitor opened next door. They were either too rigid or too short-sighted.

This paper introduces a new, smarter way to make these decisions called KBD. Think of KBD as a super-lemonade-stand manager that combines three powerful tools: Human Wisdom, Long-Term Vision, and Two Brains.

Here is how it works, broken down into simple parts:

1. The Two-Stage Strategy: The Daily Plan vs. The Hourly Hustle

KBD doesn't just make one decision; it thinks in two timeframes:

• The Macro Stage (The Daily Plan):
  Imagine you are the Head Manager who wakes up every morning. You look at the forecast, check your bank account, and decide, "Today, we should aim to sell 500 cups at an average price of $2."

  • The Innovation: Instead of just guessing, this manager uses Human Wisdom (Expert Knowledge). It knows that if you lower the price too much, you might sell a lot but lose money. It uses a special "Price-Volume Map" that is built with rules humans have learned over years (e.g., "price goes down, volume goes up, but not forever"). This ensures the daily plan is realistic and safe, even if there isn't much data yet.

• The Micro Stage (The Hourly Hustle):
  Now, imagine the Floor Manager who is actually standing at the stand. Every hour, they look at the crowd. Is it raining? Is a bus stopping nearby? Are people rushing? They tweak the price slightly up or down based on what's happening right now.

  • The Innovation: This manager uses a Decision Transformer (DT). Think of this as a "Time-Traveler." Instead of just looking at the last 5 minutes, it simulates the next 24 hours. It asks, "If I lower the price now, will I run out of money by 5 PM?" It optimizes for the whole day, not just the next minute.

2. The Secret Sauce: The "Two-Brain" System (Dual-Process Control)

This is the most creative part of the paper. The authors realized that sometimes the "Time-Traveler" (the AI) gets confused. Maybe a surprise parade happens, or a new product launches, and the AI's past data doesn't match the present. It might make a crazy mistake.

To fix this, they added a Second Brain based on how humans think:

• System 1 (The Reflex Brain - PID Controller):
  This is your knee-jerk reaction. If you see a fire, you run. If your spending is getting too fast, this "Reflex Brain" immediately hits the brakes. It's a simple, rule-based calculator that says, "We are spending too fast, slow down!" It's fast, reliable, and never panics.
• System 2 (The Thinking Brain - The AI):
  This is the slow, deep thinker. It analyzes complex patterns and plans for the future. It's smart but can be slow or make mistakes when things are weird.

How they work together:
The system fuses these two.

• When things are normal, the Thinking Brain (AI) takes the lead, making smart, long-term bets.
• When things get weird (like a sudden storm or a viral trend), the Reflex Brain steps in. It acts as a safety net. If the AI gets too excited or confused, the Reflex Brain gently pulls the reins back to keep you safe.

3. Why is this better than the old way?

• Old Way (Black Box): The computer just memorized the past. If the past changed, the computer crashed or made bad bets. It was like driving a car with your eyes closed, hoping the road doesn't change.
• KBD (Knowledge-Informed):
  1. It respects human rules: It knows that "more sales usually mean lower prices" and builds that rule into its brain from day one.
  2. It thinks ahead: It doesn't just win the next auction; it plans the whole day's strategy.
  3. It has a safety net: By combining the smart AI with a simple, reliable rule-based system, it can handle sudden changes (like a holiday sale) without losing money.

The Result

In their tests (simulated and real-world), this "Two-Brain" manager made more money and stayed within budget much better than the old methods. It was especially good at handling "surprises" where other systems failed.

In short: KBD is like hiring a brilliant strategist who plans the whole week, paired with a vigilant guard who keeps an eye on the clock and the wallet, ensuring you never overspend even when the world gets crazy.

Technical Summary

1. Problem Statement

In online advertising, auto-bidding systems are critical for purchasing ad impressions, with the industry projected to exceed $200 billion in spend by 2026. Advertisers typically set high-level performance goals (e.g., target CPA or ROI), and the platform automatically determines bid prices.

Current state-of-the-art approaches rely on black-box machine learning models trained on historical data. However, these models face three significant limitations:

1. Data Sparsity: They generalize poorly in scenarios with limited historical data (e.g., new products or campaigns).
2. Myopic Optimization: Most models optimize single-step rewards, ignoring long-term interdependencies and future ad dynamics.
3. Distribution Shifts: They struggle to adapt to abrupt changes in data distribution (e.g., sales promotions, new product launches) where human experts typically succeed.

The core challenge is to develop a bidding system that combines the scalability of machine learning with the adaptability, robustness, and long-term reasoning capabilities of human experts.

2. Methodology: KBD Framework

The authors propose KBD (Knowledge-informed Bidding with Dual-process Control), a two-stage framework that integrates human expertise into the learning process and employs a dual-process control mechanism.

A. Macro Stage: IEFormer (Informed Machine Learning)

The macro stage operates at a daily level to establish a robust baseline for the target CPA ($C_{ma}$). It utilizes the Informed Machine Learning (IML) paradigm to embed human expertise at three levels:

• Hypothesis Level (Hybrid Cognitive Architecture):
  • Connectionist Module: A Transformer encoder extracts features from historical bidding data to create dense ad embeddings.
  • Symbolic Module: A price-volume interpreter based on monotonic piecewise-linear reasoning. It maps cost to tCPA using interpretable weights ($W$) over cost segments.
  • Interaction: The Transformer personalizes the symbolic weights, while the symbolic structure imposes inductive biases (monotonicity) to constrain the black-box learning.
  • Adaptive Partitioning: To handle sparse data, an entropy-driven Generalized Lloyd Algorithm (GLA) is used to dynamically partition cost segments, ensuring uniform sample distribution.
• Algorithm Level (Inductive Biases):
  • The model enforces monotonicity (tCPA increases with cost) and smoothness via regularization.
  • Crucially, it introduces a diminishing marginal returns constraint, ensuring the rate of tCPA increase slows as cost grows, mimicking expert economic intuition.
• Data Level (Knowledge Transfer):
  • To mitigate data scarcity, the model transfers knowledge from other bidding strategies (e.g., tROI, tCPC) by converting them into equivalent tCPA values using a unified eCPM formula, effectively augmenting the training data.

B. Micro Stage: Dual-Process Control (PID + Decision Transformer)

The micro stage operates at an hourly level to adjust the baseline ($C_{mi}$) for real-time optimization. It addresses the "myopia" of single-step optimization and the instability of distribution shifts using Dual-Process Theory:

• System 2 (Deliberative): A Decision Transformer (DT) models the bidding problem as a 24-step Markov Decision Process (MDP). It optimizes long-horizon cumulative rewards (GMV) by considering state transitions and future dynamics.
• System 1 (Reactive): A rule-based PID Controller acts as a fast, robust heuristic. It adjusts bids based on the deviation between actual spending and the ideal budget trajectory.
• Fusion Mechanism:
  • Training: The DT is regularized using a Minimum Description Length (MDL) prior to stay close to the PID baseline, preventing over-aggressive behavior.
  • Inference: An uncertainty-weighted fusion dynamically blends the outputs. If the DT's prediction error (uncertainty) is high (e.g., during a sales promotion), the system shifts weight toward the robust PID controller. If uncertainty is low, the DT's long-term optimization dominates.

3. Key Contributions

1. KBD Framework: A novel two-stage approach coupling expert-driven daily calibration (IEFormer) with sequential hourly control (DT + PID) to optimize long-term bidding rewards.
2. Dual-Process Control: A robust mechanism where a rule-based PID controller (System 1) guides and fuses with a complex Decision Transformer (System 2). This handles out-of-distribution scenarios effectively, ensuring stability when data distributions shift.
3. Informed Machine Learning (IML): The successful integration of human expertise (monotonicity, smoothness, diminishing returns) into deep learning architectures via hybrid cognitive models, significantly improving performance in data-sparse regimes.

4. Experimental Results

The authors evaluated KBD on two datasets: the public iPinYou dataset and a private E-Commerce Advertising (ECA) dataset from 200 advertisers.

• Performance vs. Baselines (RQ1):
  • On the iPinYou dataset, KBD achieved the highest normalized return ($R/R^*$) of 0.730 and constraint satisfaction of 82.78%, outperforming state-of-the-art methods like PUROS, GCB-safe, and ARTEO.
  • Ablation studies showed that removing the DT or PID components degraded performance, confirming the necessity of both long-term planning and reactive control.
• Impact of IML (RQ2):
  • Removing the IE module (replacing it with end-to-end prediction) caused the largest performance drop, highlighting the critical role of the interpretable price-volume interpreter.
  • Enforcing the "diminishing marginal returns" loss ($L_{margin}$) prevented overfitting to noisy data and produced curves aligning with expert expectations.
• Robustness (RQ3):
  • IEFormer demonstrated strong robustness to the number of cost segments ($N$), consistently outperforming black-box models (like XGBoost and GP) across varying $N$ values (6 to 15).
• Real-World Impact (Table 2):
  • In online A/B tests on the ECA platform, the full KBD system (using both IEFormer and DT) improved the cost-exhaust ratio by 8.44% and GMV by 6.14% compared to a control group using only PID.
  • The system showed significant gains during distribution shifts (e.g., sales promotions), validating the effectiveness of the dual-process control.

5. Significance

This paper bridges the gap between data-driven machine learning and human expert intuition in online advertising.

• Practical Impact: It provides a deployable solution that not only maximizes revenue (GMV) but also ensures strict adherence to budget and CPA constraints, a critical requirement for advertisers.
• Theoretical Advancement: It demonstrates the efficacy of Dual-Process Theory in industrial AI, showing that combining fast, rule-based heuristics with slow, deliberative planning creates systems that are both efficient and robust to real-world volatility.
• Generalizability: The approach of embedding domain knowledge into deep learning (IML) and using uncertainty-aware fusion offers a blueprint for solving other sequential decision-making problems in dynamic environments where data scarcity and distribution shifts are common.