🤖 machine learning

Building a User Foundation Model for the Open Web

This paper introduces a user foundation model for the open web that overcomes fragmented user identity and privacy constraints by leveraging self-supervised learning on browsing histories and an LLM-in-the-loop optimization pipeline, ultimately achieving significant improvements in production bid win rates, click-through rates, and cost-per-click metrics.

Original authors: Solal Vernier, Ivan Can Arisoy, Merwan Barlier, Blaž Škrlj

Published 2026-07-31

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Solal Vernier, Ivan Can Arisoy, Merwan Barlier, Blaž Škrlj

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a massive, bustling city where every person leaves a trail of digital footprints as they walk from shop to shop. For years, the companies that show us ads have tried to build a "profile" of each person based on these footprints to guess what they might want to buy next. This is the world of recommender systems, a branch of computer science that uses math to predict human behavior. The secret sauce in modern systems is machine learning, where computers learn patterns from huge piles of data, and foundation models, which are like giant, pre-trained brains that have already learned how to understand sequences of events before being taught a specific job.

But here's the tricky part: on the "open web" (the part of the internet you visit without logging into a specific account), these footprints are messy. You might visit a site on your phone, then later on a laptop, and the computer has no way of knowing it's the same person. Your history is often short, broken into tiny pieces, or missing entirely because of privacy settings. This paper tackles a big question: Can we build a smart system that understands these messy, fragmented footprints just as well as it understands the clean, long histories of logged-in users? The answer matters because if we can't, we miss out on understanding a huge chunk of the internet, leading to ads that feel random or irrelevant.

The Puzzle of the Broken Trail

The authors, a team from Teads, faced a unique challenge. In most recommendation systems (like those on social media or shopping apps), you have a long, continuous movie of a user's life: they logged in, watched a video, bought a shoe, and liked a photo. The computer can watch this whole movie to understand the user.

But on the open web, the "movie" is often just a few scattered frames, or sometimes just a single frame. The user might be anonymous, and their history might be cut off by privacy choices. The old way of handling this was to count things: "This user clicked 3 times on sports ads." But this is like trying to understand a person's personality by only knowing how many times they blinked. It misses the story—the order of events, the timing, and the flow.

The team asked: Can we teach a computer to understand these short, broken stories? And if we do, will it help predict what a user will click on, even when we don't know who they are?

The "Time Traveler" Brain

To solve this, the team built a User Foundation Model (UFM). Think of this model as a super-smart time traveler who has read millions of short, fragmented stories about people browsing the web.

Instead of just counting clicks, the model looks at the sequence of events. It treats a user's browsing history like a sentence. Just as a sentence has a subject, verb, and object, a user's history has a Publisher (where they were), an Advertiser (what they saw), and an Event (what they did, like clicking or just looking).

The model was trained using a game called Self-Supervised Learning. Imagine you have a sentence with some words hidden (masked), and you have to guess what they are based on the context. The model practiced this by looking at a user's history, hiding parts of it, and trying to fill in the blanks. It also played a matching game: it took two different parts of the same user's history (like the morning and the afternoon) and learned that they belong to the same person, even if they look different.

The "Robot Coach" that Tweaks the Brain

Here is where it gets really cool. The team didn't just build the model; they used an AI to build the AI. They called this "LLM-as-Optimizer."

Imagine you are trying to build the perfect race car. Instead of guessing which engine or tire is best, you have a robot coach (a Large Language Model) that reads a catalog of 150 different "tuning tips" from science books. The robot coach suggests changes: "Let's try making the engine slightly bigger," or "Let's change the fuel mixture." The team tests these changes, and if a change makes the car faster, the robot coach remembers it and tries to build on that success.

In this paper, the robot coach tweaked the "User Foundation Model" by changing things like how many layers of thinking it has, how it handles time, and how it learns. This process, called Neural Architecture Search (NAS), found a version of the model that was significantly better than the original.

The Results: From "Maybe" to "Yes"

The team tested their new, robot-coached model in two ways: first in a simulation (offline), and then in the real world (online).

1. The Simulation (Offline):
They tested the model on a massive dataset of real ad requests. They compared it to the system Teads was already using.

The Good News: The new model improved the prediction of what users would click on by +1.354%. In the world of ads, this is a huge jump.
The "Broken Trail" Test: They specifically tested users with very short or missing histories. Even for users with almost no history, the model improved predictions by +0.99%. This proves the model works even when the "movie" is just a single frame.
The Generalization Test: They tried the model on different types of prediction tasks (like guessing if an ad would win a bid, not just if it would be clicked). It worked there too, improving those predictions by +1.197%. This suggests the model learned a deep understanding of human behavior, not just a trick for one specific task.

2. The Real World (Online):
The team ran a live test for 7 days, showing the new model to half of their users and the old model to the other half.

The Outcome: The group with the new model clicked on ads 2.13% more often.
The Cost: Because the ads were more relevant, the cost per click went down by 1.13%.
The Confidence: The team is very sure of these numbers. They calculated an 80% confidence interval, which means they are statistically confident that the improvement is real and not just luck.

Why This Matters

This paper shows that we don't need a perfect, long history to understand a user. Even with fragmented, short, and anonymous data, a smart model can piece together the story.

The authors suggest that this approach opens a "new axis" for improvement. It's not just about making the existing math slightly better; it's about adding a whole new way of thinking about user data. By treating browsing history as a sequence to be understood rather than just a list of counts, they found a way to make ads more relevant, even for the "ghosts" of the internet who don't log in.

The team admits there are limits. They only tested one specific type of model architecture and one specific set of data. They also note that their "robot coach" might have found a good solution, but we don't know if it's the absolute best possible solution. However, the live test confirms that this approach works in the real world, turning a theoretical idea into a practical tool that makes the open web feel a little less chaotic and a little more personal.

Technical Summary: Building a User Foundation Model for the Open Web

Problem Statement
User foundation models (UFMs) have achieved significant success in e-commerce and social recommendation, but these deployments rely on environments with stable, persistent user identities and long, contiguous action histories. In contrast, Open-Web Real-Time Bidding (RTB) operates under structurally different constraints:

Fragmented Identity: User identity is non-persistent and fragmented across sessions and devices.
Sparse History: A significant portion of traffic lacks historical data entirely (due to privacy opt-outs), and available histories are short, bounded, and disjointed.
Latency Constraints: Predictions must be served at sub-millisecond latency on the bid path.

Existing deep learning models for RTB rely on discrete, hand-crafted feature vectors (e.g., recency/frequency buckets), failing to exploit the sequential structure, temporal ordering, and behavioral trajectories of user actions. The authors investigate whether self-supervised learning on these fragmented sequences is viable and can provide a new, decoupled improvement axis for production models.

Methodology
The authors propose a User Foundation Model (UFM) trained and deployed under the constraints of Open-Web RTB. The pipeline consists of four main stages:

Data Construction & Tokenization:
- User actions are modeled as sequences of triplets: (Publisher, Advertiser, Interaction Type).
- Two-View Construction: For contrastive learning, a user's history is partitioned at a random cutpoint into a past subsequence ( $A$ ) and a future subsequence ( $B$ ). This strategy ensures the model encounters "history-absent" regimes (where $A$ contains only a single event) during pre-training.
- Time Embedding: Events are augmented with continuous time embeddings encoding the delay relative to the most recent event in the subsequence, using a log-scaled parameterization.
Self-Supervised Pre-training:
- Architecture: A bidirectional Transformer encoder (BERT-style) processes tokenized subsequences.
- Objectives: The model is trained with a joint objective:
  - Masked Language Modeling (MLM): Reconstructs masked tokens within each view ( $A$ and $B$ ).
  - Sequence-Level Contrastive (NT-Xent): Aligns the pooled [CLS] embeddings of the past ( $c_A$ ) and future ( $c_B$ ) views for the same user, treating other users' embeddings as negatives.
- Input: Triplets are tokenized into fixed vocabulary tokens (Publisher, Advertiser, Event Type) plus a [CLS] token.
Supervised Fine-Tuning:
- The pre-trained encoder is fine-tuned on a click prediction task (CTR).
- Only the past subsequence ( $A$ ) is used as input.
- A dense layer with a sigmoid activation is attached to the [CLS] embedding to predict the binary click outcome.
- At inference, the click head is discarded, and the pooled [CLS] embedding is exported as the user representation.
LLM-as-Optimizer (NAS):
- The authors implement a Neural Architecture Search (NAS) workflow where an LLM (Claude Opus 4.7) acts as the optimizer.
- Search Atoms: Instead of random hyperparameter sampling, the LLM proposes "lifters"—reviewable, code-level edits to the training pipeline (e.g., normalization placement, activation families, masking ratios) anchored in literature.
- Process: The LLM iteratively proposes lifters based on a catalog of ~150 options and the history of trial outcomes. Candidates are evaluated via a proxy classifier and pre-training loss.

Key Contributions

A Deployed UFM for Open Web: A user foundation model pre-trained on fragmented browsing sequences using MLM and contrastive objectives, successfully deployed under strict latency and sparse-history constraints.
LLM-in-the-Loop Optimization: An industrial instantiation of the "LLM-as-optimizer" paradigm, using a curated catalog of code-level edits to systematically improve the encoder's pre-training pipeline.
Generalization: Demonstration that the learned representation improves diverse downstream tasks (CTR and bid win-rate) and architectures (GDCN and DCN2) without re-fine-tuning the encoder for each specific task.

Results

Offline Performance:
- The NAS-optimized encoder (UFM-NAS) achieved a +1.354% Relative Information Gain (RIG) on the production GDCN CTR ranker compared to the unextended baseline.
- Gains were consistent across history regimes: +0.99% RIG for users with no history (opt-out dominated) and +1.62% RIG for users with history.
- The representation generalized to a structurally different task (bid win-rate prediction), yielding +1.197% RIG without task-specific fine-tuning.
- It also improved a second CTR architecture (DCN2) by +0.894% RIG.
Online A/B Test:
- A 7-day live test on production traffic showed a +2.13% increase in CTR and a -1.13% decrease in eCPC (effective cost per click).
- Visit-optimized traffic saw a +2.37% increase in visit rate and -1.47% decrease in eCPV.
- All metrics had 80% confidence intervals excluding zero.
Optimization Efficiency: The NAS process reduced total pre-training loss by 25.2% while converging in fewer epochs (6 vs. 10), resulting in a negligible increase in wall-clock training time (+2.6%).

Significance and Claims
The paper claims to demonstrate the viability of sequential modeling in the Open-Web RTB domain, a setting previously considered intractable for such approaches due to identity fragmentation and latency constraints. The authors position the UFM not merely as a model improvement but as a new, decoupled improvement axis for production systems.

The work serves as an existence proof for:

Applying foundation models to sparse, non-persistent user data.
Using LLMs to optimize industrial training pipelines via code-level edits ("lifters").
The transferability of sequential user representations across different ranker architectures and prediction targets (CTR vs. Win-Rate).

The authors remain modest regarding the specific contribution of the LLM versus the curated catalog of lifters, noting that isolating these factors requires future controlled studies. They also acknowledge limitations in single-seed point estimates and the potential for "winner's curse" bias in the selection process, relying on the live A/B test to validate the offline gains.