Into the Unknown: Accounting for Missing Demographic Data when Mitigating Ad Delivery Skew

Technical Summary: Into the Unknown: Accounting for Missing Demographic Data when Mitigating Ad Delivery Skew

Problem Statement
Online advertising platforms utilize algorithmic systems to match ads with users, a process known as ad delivery. Prior audits have established that these systems often introduce demographic skew, where ads are systematically under-delivered to specific groups despite an advertiser's intent to reach a proportional audience. This is particularly critical in public sector advertising (e.g., public services, grants), where skewed delivery can prevent marginalized groups from accessing essential information, thereby exacerbating social disparities.

While advertisers can theoretically counteract skew through direct demographic targeting, major platforms like Google Ads do not provide a mechanism to target users with "unknown" demographic labels (users for whom the platform cannot infer attributes like gender). Excluding these "unknown users" is problematic for public interest campaigns because:

1. It contradicts the goal of reaching the entire eligible population.
2. Data suggests unknown users are disproportionately likely to be socioeconomically disadvantaged or hold non-binary gender identities.
3. Excluding them may reify heteropatriarchal systems and marginalize non-conforming individuals.

Furthermore, existing solutions like Meta's Variance Reduction System (VRS) offer limited advertiser control, increase costs significantly, and do not apply to Google's search ecosystem. Traditional "Direct Budget Split" interventions (splitting a campaign into separate gender-specific campaigns) are effective at reducing skew but often exclude unknown users and incur high costs due to increased competition for granular audiences.

Methodology
The authors conducted a case study in collaboration with a state-level government agency running a Google Ads campaign to promote small business resources. The campaign historically showed consistent skew toward users labeled "male" by Google's inference algorithms.

To address this, the authors designed and implemented a novel "Budget Split with Unknown Users" intervention. The methodology involved:

1. Campaign Restructuring: Replacing the original single campaign (targeting "All Users") with a set of four new campaigns running in alternating cycles (Cycle A and Cycle B) over a six-week period.
2. Targeting Strategy: Since Google Ads does not allow targeting "Unknown" as a standalone group, the authors leveraged the platform's ability to combine groups. The intervention utilized:
   • Single Gender Campaigns: Targeting only "Male" or only "Female."
   • Combined Campaigns: Targeting "Male + Unknown" or "Female + Unknown" (by excluding the opposite binary gender).
3. Budget Allocation: Budgets were split to enforce a desired parity ratio (0.5) between male and female delivery, while ensuring unknown users were included in the "Combined" campaigns.
4. Evaluation Metrics: The study measured ad delivery skew (ratio of impressions/spend to male users relative to total male/female impressions/spend), cost-effectiveness (Cost Per Mille - CPM), and engagement (Click-Through Rate and Conversion Rate).
5. Simulation: To account for the uncertainty of unknown users' actual identities, the authors simulated various gender distributions for the "unknown" pool (Symmetric, Informative, and SimilarWeb priors) to test the robustness of their skew estimates.

Key Contributions

1. Empirical Evidence of Skew: The study identifies persistent gender-based skew in a real-world government advertising campaign, specifically showing that "Max Clicks" bidding strategies consistently under-delivered to female-labeled users.
2. Novel Intervention Design: The paper proposes and implements a "Budget Split with Unknown Users" strategy. This design explicitly includes users with missing demographic data, addressing a gap in prior literature which often excluded them.
3. Trade-off Analysis: The work demonstrates that this intervention serves as a "middle ground" in the trade-off between the high costs of granular targeting (Direct Budget Split) and the high skew of untargeted campaigns (Original Campaign). It reduces skew without incurring the full cost penalty of excluding unknown users.

Results

• Skew Reduction: The intervention successfully reduced ad delivery skew. In the "Max Clicks" campaign, the original skew (0.563) was reduced to near-parity (0.499) in Single Gender campaigns and to 0.532 in Single Gender + Unknown campaigns. In the "Max Conversions" campaign, the intervention shifted skew from 0.486 to 0.494 (Single Gender + Unknown), closer to the desired 0.5 than the over-corrected Single Gender approach (0.526).
• Cost-Effectiveness: While granular targeting generally increases CPM, the "Single Gender + Unknown" strategy significantly reduced the cost differential between male and female users compared to pure "Single Gender" targeting. This suggests that including unknown users in the targeting pool lowers the competition premium for female-labeled users, making equitable delivery more affordable.
• Simulation Robustness: Simulations of unknown user distributions indicated that the intervention's ability to reduce skew holds true across various assumptions about the gender identity of unknown users, providing directional guidance for advertisers even without perfect data.

Significance and Claims
The paper claims that the "Budget Split with Unknown Users" is a valuable, practical approach for resource-constrained government advertisers committed to equitable public service outreach. It argues that:

• Inclusion is Necessary: Excluding unknown users in public interest advertising systematically disadvantages marginalized groups and undermines the goal of universal resource access.
• Practicality over Purity: While perfect demographic knowledge is unavailable, imperfect platform-inferred labels combined with inclusive targeting strategies can effectively mitigate skew.
• Systemic Recommendations: The authors conclude that online advertising platforms are currently ill-equipped for public interest goals. They recommend that government advertisers establish technical ownership of data during procurement, consider offline outreach as a complement to online ads, and that platforms provide better tools for measuring and addressing skew without imposing restrictive bidding constraints (like CPC caps) that exacerbate inequality.

The study emphasizes that while online advertising systems are designed for profit, they can be adapted to serve public interest goals through careful, inclusive intervention design that acknowledges the reality of missing demographic data.