💻 computer science

Scaling Laws for Behavioral Foundation Models over User Event Sequences

This paper establishes scaling laws for behavioral foundation models trained on user event sequences, revealing that a small feature-based embedder is consistently compute-optimal, that optimal training strategies shift with budget and evaluation metrics, and that negative sampling constraints eventually transition from FLOPs to memory limits.

Original authors: Rickard Brüel Gabrielsson

Published 2026-06-05

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Rickard Brüel Gabrielsson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-speed recommendation engine for a giant online store. Every time a customer clicks, buys, or searches, it leaves a digital footprint. Your goal is to build an AI that predicts what a customer will do next based on their history.

This paper is like a massive, scientific "recipe book" for building that AI. The authors ran about 600 different experiments, tweaking the AI's brain in every possible way, to figure out the most efficient way to spend their computing power (which costs a lot of money and electricity).

Here is the breakdown of their findings, using simple analogies:

1. The Two-Part Brain: The Librarian and the Storyteller

Most of these AI models have two distinct parts working together:

The Librarian (The Embedder): This part looks at a specific item (like a pair of shoes or a transaction) and figures out what it is. It reads the text, the price, the color, and the category.
The Storyteller (The Transformer): This part looks at the sequence of events. It remembers that you looked at shoes, then a hat, then a jacket, and tries to guess what you'll buy next.

The Big Discovery: The authors found that the "Librarian" should be tiny.

The Analogy: Imagine you are hiring a team to write a novel. You have a limited budget. You might think you need a huge team of researchers (the Librarian) to look up facts for every word. But the authors found that you only need a tiny team of researchers (about 2% of your total staff).
Why? The "Librarian" sees the same popular items (like "Nike sneakers") thousands of times. It gets bored and learns them quickly. The "Storyteller," however, sees unique combinations of events that rarely repeat. The Storyteller needs the big brain; the Librarian just needs a small, efficient one.

2. The "Batch Size" Dilemma: How Many Students in the Classroom?

When training the AI, you don't show it one example at a time; you show it a "batch" of examples.

The Analogy: Think of a teacher teaching a class. If the class is too small (batch size 64), the teacher gets distracted by individual quirks. If the class is huge (batch size 2048), the teacher might get overwhelmed or the lesson becomes too generic.
The Discovery: The "sweet spot" depends on what you are measuring.
- If you care about accuracy (did we get the right answer?), the sweet spot is a medium-sized class (around 570 students).
- If you care about ranking (is the top result the best one?), the sweet spot is a smaller class (around 200–275 students).
- Takeaway: You can't just pick one batch size for everything. The "best" size changes depending on what you are trying to optimize.

3. The Data vs. Model Trade-off: Big Brain or Big Library?

You have a fixed amount of money (Compute Budget). Do you buy a super-smart AI (more parameters) and feed it less data? Or a slightly less smart AI and feed it a massive library of data?

The Discovery:
- At small budgets: You should feed the AI massive amounts of data. The AI is like a sponge; it needs to soak up as much information as possible because it's not smart enough to generalize on its own yet.
- At huge budgets: As you get richer (more computing power), the balance shifts. You start needing a smarter AI and less data, eventually reaching a point where the ratio looks similar to how Large Language Models (like the ones that write text) are trained.
- The Trend: It starts as a "data-heavy" strategy and slowly moves toward a "balanced" strategy as you get more powerful computers.

4. The "Negative" Sampling: How Many Wrong Answers to Show?

When the AI learns, it doesn't just see the correct answer; it also sees "distractors" (wrong answers) to learn what not to pick.

The Analogy: Imagine a multiple-choice test. If you only show the student 3 wrong answers, they might get lucky. If you show them 1,000,000 wrong answers, they learn the pattern perfectly.
The Discovery:
- Small budgets: You need a moderate number of wrong answers (hundreds of thousands).
- Huge budgets: You want as many wrong answers as possible. In fact, at the very largest budgets tested, the limit wasn't computing power anymore; it was memory. The system wanted to show 2 million wrong answers, but the computer ran out of space to hold them.
- The Catch: The "best" number of wrong answers depends on whether you are measuring simple accuracy or complex ranking. They don't agree on the number.

5. The Most Important Lesson: The Goalpost Moves

The paper's most critical warning is about metrics (how you measure success).

The Analogy: Imagine you are training a race car. If you measure success by "top speed," you tune the engine one way. If you measure success by "fuel efficiency," you tune it a completely different way.
The Discovery: In these behavioral models, the metric you choose changes the recipe.
- If you train the AI to minimize "loss" (mathematical error), you get one set of settings.
- If you train it to maximize "ranking" (putting the best item first), you get a different set of settings.
- Crucial Point: The AI that is best at minimizing math errors is not always the AI that is best at ranking items correctly. You have to decide exactly what you want to optimize before you start building the model, because the "optimal" model changes based on that choice.

Summary

To build the most efficient "Behavioral Foundation Model" (an AI that understands human actions):

Keep the "item understanding" part (Embedder) very small (about 2% of the total size).
Use a medium batch size, but adjust it based on whether you care about accuracy or ranking.
Start with huge amounts of data, but as you get more powerful computers, shift toward a smarter model.
Use as many "wrong answer" examples as your computer's memory can hold.
Most importantly: Decide exactly what "success" looks like (the metric) before you start, because that decision dictates the entire architecture of your AI.

Technical Summary: Scaling Laws for Behavioral Foundation Models over User Event Sequences

Problem Statement

Foundation models are increasingly applied to sequences of user actions in domains such as recommendations, payments, fraud detection, and commerce. However, unlike Large Language Models (LLMs), these behavioral models lack established scaling laws to guide compute allocation. A critical architectural distinction exists: while LLM tokens are typically opaque vocabulary IDs, behavioral events are feature-rich (containing text, metadata, visual features, timestamps, etc.). Consequently, modern systems employ a two-part architecture: a feature-based event embedder that maps raw item features to dense vectors, and a decoder-only transformer contextualizer that models sequences of these embeddings.

This architecture introduces unique scaling challenges not present in standard text LMs. Specifically, the embedder must be re-computed for every candidate item during inference, which is prohibitively expensive for large catalogs. Standard practice involves a two-stage training recipe: joint training with in-batch sampled softmax, followed by freezing the embedder and caching embeddings to train the contextualizer with a larger pool of sampled negatives. This setup creates four coupled hyperparameters (knobs) that require calibration:

Embedder Share: The proportion of parameters allocated to the embedder versus the contextualizer.
Critical Batch Size: The batch size that maximizes data efficiency.
Model/Data Allocation: The optimal split of compute budget between model size ( $N$ ) and data tokens ( $D$ ).
Negative Sampling: The number of sampled negatives ( $K$ ) used after the embedder is frozen.

Methodology

The study conducts a comprehensive empirical investigation across approximately 600 training runs on a single real-world retail interaction corpus (combining searches, views, clicks, and purchases). The experiments span a compute budget range of $C \in [10^{15}, 10^{19}]$ FLOPs.

The experimental design isolates four axes while holding others constant or sweeping them systematically:

Architecture: Varying the embedder parameter share ( $s = p_e/N$ ) from 0% to 50% and contextualizer depth.
Batch Size: Sweeping global batch sizes ( $B$ ) from 64 to 2048 to identify critical batch sizes ( $B_{crit}$ ) for different metrics.
Model/Data Allocation: Training on iso-FLOP frontiers to determine the optimal $N$ and $D$ for various budgets, comparing against the "Chinchilla" heuristic ( $D/N \approx 20$ ).
Negative Sampling: Varying the number of extra sampled negatives ( $K$ ) from 0 to 2 million in the second training stage.

Evaluation Protocol: The study distinguishes between two evaluation regimes:

Stage 1 (In-batch): Evaluation against unique targets within the batch, matching the training distribution.
Stage 2 (Full-Catalogue): Evaluation against the full cached product catalogue, simulating the deployed retrieval setting.
Metrics include cross-entropy loss, perplexity, recall@k, NDCG@k, MRR@10, and coverage.

Key Contributions and Results

1. The Compute-Optimal Embedder is Small

Contrary to intuition that feature-rich inputs require large encoders, the study finds the optimal embedder share is remarkably small and stable across four decades of compute.

Finding: The optimal parameter share is $s^\star \approx 2\%$ (specifically $1.1\% - 3.7\%$ depending on the metric and budget).
Mechanism: This is driven by two asymmetries:
1. Compute Asymmetry: Embedder parameters are touched $\approx 24$ times more often per item than contextualizer parameters due to the short context length of the embedder versus the sequence length of the transformer.
2. Data Distribution Asymmetry: Popular items in the catalog are repeated hundreds to thousands of times, exposing the embedder to high-frequency memorization, whereas the contextualizer rarely sees the same event window twice.
Implication: A small embedder is compute-optimal because its parameters are more expensive per step and see a more repetitive data distribution.

2. Scaling is Initially Data-Heavy, Converging to Chinchilla

Behavioral models exhibit different scaling dynamics compared to text LMs, particularly at lower compute budgets.

Finding: The optimal data-to-parameter ratio ( $D/N$ ) is initially very high, starting at $\approx 344$ at $10^{15}$ FLOPs and decreasing to $\approx 36$ at $10^{19}$ FLOPs.
Trend: As compute increases, the ratio approaches the text-LM Chinchilla heuristic ( $D/N \approx 20$ ).
Exponents: The fitted model-size exponent is $N^\star \propto C^{0.617 \pm 0.025}$ , consistent across various ranking metrics (Recall, NDCG).

3. Evaluation Metrics Define the Scaling Law

A central finding is that the choice of evaluation metric fundamentally alters the "compute-optimal recipe."

Critical Batch Size ( $B_{crit}$ ): Metrics disagree on the optimal batch size. Loss and Recall@10 share a $B_{crit} \approx 570$ , while position-weighted metrics (NDCG@10, MRR@10) saturate earlier at $B_{crit} \approx 200\text{--}275$ .
Negative Sampling ( $K$ ): The optimal number of negatives is metric-dependent. Loss-like metrics prefer larger $K$ (often $>10^6$ ), while ranking metrics prefer smaller $K$ (often $10^5$ ).
Loss vs. Ranking Disagreement: The sampled-softmax training loss is not always a reliable proxy for full-catalogue ranking quality. In Stage 2, the correlation between training loss and ranking metrics can flip sign depending on the compute budget and the specific metric used.

4. Negative Sampling Shifts from Compute to Memory Constraints

Small Budgets: Optimal $K$ is in the low hundreds of thousands and is metric-dependent.
Large Budgets ( $10^{19}$ FLOPs): At the largest scale, every headline metric continues to improve as $K$ increases up to the maximum tested ( $2M$ ). The active constraint shifts from FLOPs to candidate-axis memory (the memory footprint of the softmax logits).
Implication: At scale, negative sampling becomes a memory engineering problem rather than a pure compute allocation problem.

Significance and Claims

The paper claims that for behavioral foundation models, the evaluation metric is part of the scaling law. Unlike text LMs where a single loss function often suffices to guide architecture and training recipes, behavioral models require practitioners to select their target metric before fitting scaling laws. Changing the metric can change the optimal embedder share, batch size, and negative sampling strategy.

The study provides a calibrated "recipe" for practitioners training standard event-embedder $\to$ transformer stacks:

Architecture: Use a small embedder ( $\approx 2\%$ of parameters).
Data: Use a data-heavy allocation ( $D/N$ ) at low compute, converging toward Chinchilla at high compute.
Batch Size: Tune based on the specific target metric (lower for ranking, higher for loss).
Negative Sampling: Treat large-scale negative sampling as a memory-bound problem; use $K$ in the hundreds of thousands for standard deployments, but recognize that at $10^{19}$ FLOPs, memory constraints, not compute, limit further gains.

The authors emphasize that while the scaling exponents (how $N$ and $D$ grow with $C$ ) are relatively robust across metrics, the actionable recipe (the specific cell to deploy) is highly sensitive to the chosen evaluation metric.