🤖 machine learning

CADET: Context-Conditioned Ads CTR Prediction With a Decoder-Only Transformer

This paper introduces CADET, a context-conditioned decoder-only transformer deployed at LinkedIn that overcomes key challenges in ads CTR prediction through innovations like multi-tower post-scoring modeling and self-gated attention, achieving an 11.04% CTR lift over the production baseline.

Original authors: David Pardoe, Neil Daftary, Miro Furtado, Aditya Aiyer, Yu Wang, Liuqing Li, Tao Song, Lars Hertel, Young Jin Yun, Senthil Radhakrishnan, Zhiwei Wang, Tommy Li, Khai Tran, Ananth Nagarajan, Ali Naqvi

Published 2026-08-12

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: David Pardoe, Neil Daftary, Miro Furtado, Aditya Aiyer, Yu Wang, Liuqing Li, Tao Song, Lars Hertel, Young Jin Yun, Senthil Radhakrishnan, Zhiwei Wang, Tommy Li, Khai Tran, Ananth Nagarajan, Ali Naqvi, Yue Zhang, Renpeng Fang, Avi Romascanu, Arjun Kulothungun, Deepak Kumar, Praneeth Boda, Fedor Borisyuk, Ruoyan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

=== SUMMARY ===
Imagine you are walking through a massive, bustling digital marketplace. Every second, thousands of digital signs flash up, trying to catch your eye. Some signs are for shoes, others for job postings, and some for coffee. The people running this marketplace have a tricky job: they need to guess which sign you will actually stop and look at before you even see it. This guessing game is called "Click-Through Rate" (CTR) prediction. For a long time, the best guessers were like super-smart accountants who looked at a list of your past habits and the details of the sign to make a calculation. But recently, a new kind of "generative" AI has started showing up in other areas, like recommending movies or songs. These new AIs are like storytellers; instead of just crunching numbers, they read a whole story of what you've done before and predict what happens next. The big question for the marketplace owners was: Can we use this new "storyteller" AI to guess which ads you'll click, even when the rules of the game change after the guess is made?

This is exactly what the team at LinkedIn set out to solve with their new system, called CADET. They built a "decoder-only transformer," which is a fancy way of saying they created an AI that reads a sequence of events (like your past clicks and views) and predicts the future in one smooth pass. But ads are tricky. Unlike recommending a movie, an ad's success often depends on things that aren't known until after the AI makes its guess—like exactly where on the screen the ad will end up. If the AI guesses wrong because it didn't know the ad would be at the bottom of the page instead of the top, the whole system gets confused.

The researchers found that by giving their AI a special set of tools, they could solve this puzzle. They taught the model to imagine different "what-if" scenarios for where the ad might land, and they built a "self-gating" mechanism that acts like a bouncer, stopping the AI from getting distracted by noisy or confusing information. They also gave the AI a special sense of time, allowing it to understand that a click from five minutes ago is different from a click five months ago. Most importantly, they made sure the AI didn't use information it wouldn't have in the real world during its training. When they tested this new system in the real LinkedIn marketplace, it didn't just work; it performed significantly better than the old, complex systems they replaced. The new AI suggested that users were 11.04% more likely to click on ads, proving that a unified, story-telling approach can outperform the old, multi-part machine.

The Story of CADET: A Storyteller for Ads

So, how did they build this? Let's break down the magic of CADET (Context-Conditioned Ads Decoder-Only Transformer) into simple, everyday concepts.

The "Chicken-and-Egg" Problem
Imagine you are a chef trying to guess how much a customer will enjoy a meal. But here's the catch: you have to guess the enjoyment before you know where the table is sitting. If the table is right next to the kitchen, the customer might be hungry and excited. If it's in a dark corner, they might be grumpy. In the world of ads, the "table" is the ad's position on the screen. The AI has to predict if you'll click an ad, but it doesn't know yet if that ad will be at the very top (where everyone sees it) or way down at the bottom. This is the "chicken-and-egg" problem: the prediction depends on the position, but the position depends on the prediction.

CADET solves this by being a master of "what-if" scenarios. Instead of making one single guess, it has multiple "prediction heads" (think of them as different chefs in the kitchen). One chef guesses, "If this ad is at the top, will you click?" Another chef guesses, "If it's at the bottom, will you click?" The model learns to produce all these answers at once. When the ad actually gets placed, the system just picks the answer from the chef who was right about the location. This way, the AI never gets confused by the mystery of where the ad will end up.

The Bouncer (Self-Gated Attention)
In a crowded room, sometimes one loud person drowns out everyone else. In AI, this is called "attention sink," where the model focuses too much on one token (a piece of data) and ignores the rest, leading to bad guesses. The researchers gave CADET a "bouncer" called a self-gated attention mechanism.

Think of the AI's brain as a busy hallway. Every time a piece of information tries to pass through, the bouncer checks it. If the information is noisy or not very useful, the bouncer dims its lights (gates it down) so it doesn't distract the model. If the information is important, the bouncer lets it shine. This happens twice: once when the information first arrives (representation level) and again when the AI tries to connect different pieces of information (interaction level). This keeps the training smooth and stable, preventing the AI from going crazy or getting stuck on bad patterns.

The Time Machine (Timestamp RoPE)
Most AI models count steps: "Step 1, Step 2, Step 3." But in the real world, time isn't just about steps; it's about when things happened. A click that happened 10 seconds ago is very different from one that happened 10 months ago.

CADET uses a special kind of "time machine" called a timestamp-based Rotary Positional Embedding (RoPE). Instead of counting steps, the AI looks at the actual clock time (Unix timestamps) of every event. It can understand that the gap between two events is short (like seconds) or long (like months). This helps the model realize that your behavior changes over time. If you clicked on a job ad yesterday, you might still be interested today, but if you clicked on it six months ago, you probably aren't anymore. This time-sense allows the AI to learn patterns across different scales, from the immediate moment to long-term trends.

The "No Cheating" Rule (Session Masking)
When you train an AI, you want it to learn like a student, not a cheater. In the real world, when an ad is shown, the system doesn't immediately know if you clicked it; there's a tiny delay (like a few seconds or minutes) while the data travels. If the AI is trained on data where it can "see" the click before it's supposed to happen, it learns to use information it shouldn't have. It will think, "Oh, I knew you clicked because I saw the result!" but in the real world, it won't have that info.

To stop this, the researchers used session masking. Imagine a teacher covering the answer key on a test. During training, the AI is forced to look only at the information that would have been available at that exact moment. If a click happens 30 seconds later, the AI is blind to it during the training of the current moment. This ensures that when the AI goes to the real world (online serving), it doesn't get confused or make bad guesses because it's relying on information it can't actually see.

The Speed Boost (Production Engineering)
Finally, building a smart AI is one thing; making it fast enough to handle billions of ads every day is another. The team used some clever engineering tricks. They "packed" the data tightly, like organizing a suitcase so there's no wasted space, which made the training much faster. They also built custom "FlashAttention" engines (specialized software parts) that let the AI score hundreds of ads at once in a single, lightning-fast pass, rather than checking them one by one.

The Results: A Big Win for LinkedIn

When the team put CADET to the test on LinkedIn's main feed, the results were impressive. They compared it against their old, highly optimized system (a mix of different models working together). The new CADET model didn't just keep up; it crushed the competition.

In a large-scale online test, CADET achieved an 11.04% increase in click-through rates compared to the old system. This means that for every 100 ads shown, significantly more people stopped to look at them. Interestingly, because advertisers on LinkedIn usually spend their full budgets automatically, the total money spent didn't change much, but the value of those clicks went up. The cost per click dropped by 10.9%, meaning advertisers got a better return on their investment.

The paper suggests that this success comes from combining a unified "storytelling" AI with smart fixes for the specific problems of ads (like position and time). It shows that a single, well-designed model can beat a complex team of older models. While the researchers note that there is still work to be done—like handling even more types of "what-if" scenarios—their work proves that decoder-only transformers are ready to take the lead in the world of online advertising.

Technical Summary: CADET (Context-Conditioned Ads CTR Prediction With a Decoder-Only Transformer)

Problem Statement

Click-through rate (CTR) prediction is the core mechanism driving ad delivery and user experience on LinkedIn's homefeed. While Deep Learning Recommendation Models (DLRMs) like DCN and DIN have historically dominated this domain through explicit feature interactions, adapting generative, decoder-only transformer architectures to industrial ad ranking presents unique challenges:

The "Chicken-and-Egg" Problem: Post-scoring contextual signals (e.g., final ad position) significantly influence user behavior but are unavailable at the time of scoring. Existing methods often treat position as a debiasing feature or use secondary models, creating a dependency loop between predicted CTR and final ranking.
Offline-Online Consistency: Training data often contains information (e.g., same-session interactions) that is unavailable during inference due to system delays or tracking latencies, leading to train-serve skew.
Scalability and Stability: Industrial workloads require training on billions of daily impressions with strict latency budgets (p99 < 50ms) and stable convergence, which can be difficult with standard transformer architectures prone to pathological attention behaviors.

Methodology

The authors present CADET (Context-Conditioned Ads Decoder-Only Transformer), an end-to-end decoder-only transformer designed specifically for ads CTR prediction. The architecture processes interleaved sequences of user static features, ad impressions, and contextualized actions.

Core Architectural Innovations

1. Context-Conditioned Decoding Architecture
To resolve the dependency between predicted CTR and post-scoring context (like ad position), CADET employs a multi-tower decoding block.

Mechanism: The model partitions the context space into $K$ discrete buckets (e.g., positions 1–4 vs. 5+). For each impression, the shared transformer output is fed into $K$ parallel prediction heads.
Training: Only the head corresponding to the ground-truth context bucket contributes to the loss.
Inference: All $K$ heads generate predictions in a single forward pass. The serving system selects the appropriate prediction based on the actual rendering context, eliminating the need for iterative re-scoring.

2. Self-Gated Attention Mechanism
To stabilize training and mitigate pathological attention behaviors (e.g., attention sinks), the authors introduce a self-gated attention module that regulates information flow at two levels:

Representation-Level Gating: A lightweight module applies a sigmoid gate to the input token representations ( $X$ ) before projection, acting as adaptive feature selection to suppress noisy dimensions.
Interaction-Level Gating: The projected queries ( $Q$ ) and keys ( $K$ ) are individually gated before the dot-product attention calculation. This constrains the magnitude of interactions, preventing dominant tokens from monopolizing attention.

3. Temporal Encoding via Modified RoPE
Standard Rotary Positional Embeddings (RoPE) rely on sequence position. For ads, temporal differences are more critical.

Modification: The authors replace sequence position $p$ with Unix timestamps $t$ .
Adaptation: The rotation angle is defined as $\alpha_i(t) = t \cdot \theta_i$ , where $\theta_i$ is scaled to handle time differences ranging from seconds to months. This allows the model to capture temporal relationships across diverse timescales without discarding causal ordering (enforced by the causal mask).

4. Session-Aware Masking
To ensure offline-online consistency, the model employs customized masking strategies during training.

Problem: Standard training might allow tokens to attend to future events within the same session that are not yet observable at inference time due to latency.
Solution: A mask $M$ is applied where $M_{i,j} = -\infty$ if $t_j > t_i - \Delta_{delay}$ . This prevents the model from learning dependencies on unavailable in-session events, aligning training conditions with serving constraints.

5. Production Engineering

Tensor Packing: To handle the sparsity of user sequences, the system compacts batches into a contiguous token buffer, eliminating padding tokens and increasing effective batch size by ~4x.
Custom FlashAttention: A custom kernel integrates masking logic directly into the tiled attention computation. This enables efficient multi-item scoring (ranking hundreds of candidates in one pass) with linear complexity relative to the number of candidates, rather than quadratic.

Key Contributions

Context-Conditioned Decoding: A novel multi-tower architecture that explicitly models post-scoring signals (e.g., ad position) in a single pass, resolving the ranking dependency loop.
Self-Gated Attention: An improved attention design with dual-level gating that stabilizes training and ensures reliable convergence at scale.
Timestamp-Based RoPE: A temporal encoding variant that captures time differences from seconds to months, replacing sequence position with event timestamps.
Session Masking: Strategies that prevent train-serve skew by masking unavailable in-session events during training.
Scalable Production System: Optimizations including tensor packing and custom FlashAttention kernels that enable efficient training on large-scale datasets and low-latency serving.

Results

Offline Experiments

Ablation studies on LinkedIn's sponsored updates dataset (1 year of data) demonstrated the contribution of each component:

Context-Conditioned Modeling: Removing this reduced AUC by 0.61%, highlighting its critical role.
Session Masking: Removing this reduced AUC by 0.31%, confirming the importance of offline-online consistency.
Temporal RoPE: Removing this reduced AUC by 0.13%.
Self-Gated Attention: The combined representation and interaction-level gating provided the most stable convergence compared to baselines without gating or single-level gating.
Backbone Comparison: CADET variants (using either HSTU or Transformer backbones) performed nearly identically (within 0.0001 AUC), suggesting the gains stem from the modeling components rather than the backbone choice. The full CADET model achieved an AUC of 0.8554.

Online A/B Testing

CADET was deployed on LinkedIn's homefeed sponsored updates, replacing the production LiRank baseline (a hybrid ensemble of DCNv2 and TransAct).

CTR Lift: CADET achieved a statistically significant 11.04% lift in Click-Through Rate.
Revenue & ROI: While total revenue remained approximately neutral (due to advertisers spending full budgets under auto-bidding), the relevance gains resulted in a 10.9% reduction in Cost-Per-Click (CPC), indicating stronger advertiser ROI.
Efficiency: The system serves main traffic with a p99 latency under 50ms, processing ~330 requests per second per GPU.

Significance

The paper claims that CADET demonstrates that a unified decoder-only transformer, when combined with targeted modeling innovations (specifically context-conditioned decoding and self-gated attention) and production optimizations, can effectively outperform highly optimized multi-component hybrid DLRM systems in industrial ad ranking. It addresses the specific challenges of applying autoregressive transformers to advertising, particularly the handling of post-scoring signals and the maintenance of offline-online consistency, marking a successful transition from traditional DLRMs to generative paradigms in large-scale ad systems.