Ad Headline Generation using Self-Critical Masked Language Model

Technical Summary: Ad Headline Generation using Self-Critical Masked Language Model

Problem Statement
Generating enduring, high-quality advertisements for e-commerce platforms is a non-trivial challenge, particularly at scale. Existing manual methods often rely on programmatic keyword padding or templated creatives (e.g., "Save Now on..."), which fail to create excitement or brand identity. Furthermore, retail product data presents unique difficulties: titles often suffer from poor structure, grammatical issues, or partial phrases, and campaigns frequently involve multiple related products that must be generalized into a single, cohesive headline. The goal is to build a Natural Language Generation (NLG) system that can instantly generate attractive, grammatically correct headlines that capture shared characteristics across multiple products without being specific to a single item.

Methodology
The authors propose a novel approach combining Masked Language Models (MLM) with Reinforcement Learning (RL) using a Self-Critical policy gradient method.

1. Architecture and Input Encoding:

   • The system utilizes a Transformer-based BERT model (specifically BERT Large).
   • Unlike standard auto-regressive models, this approach conditions the generation on multiple products simultaneously. The model concatenates tokens from multiple product titles using a special separator token ([P_SEP]) and prepends a start token ([SOS]).
   • Only the tokens corresponding to the target headline are randomly masked during training; source product tokens remain unmasked to preserve their information.
   • To enable auto-regressive generation (predicting one word at a time) with a bidirectional model, the authors employ masked attention. This modifies the standard self-attention mechanism to prevent the model from attending to future tokens during the inference step, effectively restricting attention to accessible tokens only.

2. Training via Self-Critical Sequence Training (SCST):

   • Standard MLM training optimizes for log-likelihood, which can lead to exposure bias and does not directly optimize for non-differentiable quality metrics like BLEU or ROUGE.
   • To address this, the authors frame the generation task as an RL problem. The model acts as an agent that predicts masked words (actions) to maximize a reward based on headline quality.
   • They employ Self-Critical Sequence Training (SCST). In this setup, two headlines are generated for the same input:
     • $\hat{x}_h$: Generated by sampling from the model's vocabulary distribution.
     • $\hat{z}_h$: Generated using a greedy inference strategy (selecting the token with the highest probability at each step).
   • The gradient is updated based on the difference in rewards between the sampled headline and the greedy baseline: $\nabla \theta L_{SC\_MLM} \approx -(r(\hat{x}_h) - r(\hat{z}_h)) \nabla_\theta P$. This allows the model to directly optimize for metrics like ROUGE-L without altering the inference setup or latency.

3. Inference:

   • During inference, headlines are generated auto-regressively using Beam Search.
   • A modified Length Normalization technique is applied to the beam search scores to prevent the bias against longer headlines, which typically have lower raw log-probabilities.
   • Post-processing includes Regular Expression-based cleaning to remove extra spaces around symbols.

Key Contributions

• Multi-Product Conditioning: The model is the first to utilize MLMs for generating headlines conditioned on multiple products simultaneously, allowing it to generalize shared attributes effectively.
• RL-Optimized MLM: The paper demonstrates a novel application of Self-Critical policy gradients to train MLMs. This allows for the direct optimization of headline quality metrics (e.g., ROUGE, BLEU) during training while maintaining the standard inference latency of the model.
• Performance Over Baselines: The proposed method outperforms existing baselines, including LSTM-based Pointer Networks with RL and standard Transformer models, in both overlap metrics and human quality audits.
• Efficiency: The approach significantly reduces the manual effort and time required to create headlines while maintaining low inference latency.

Results
The system was evaluated on a dataset of over 500,000 ad campaigns from Amazon.

• Overlap Metrics: The proposed Self-Critical MLM (SC-MLM) achieved the highest scores across all overlap metrics (ROUGE-L, CIDEr, BLEU-4, METEOR) compared to baselines and ablation studies. For instance, it achieved a ROUGE-L score of 6.33, significantly outperforming the baseline bi-LSTM (0.62) and the standard MLM without RL (5.08).
• Quality Audits: In a large-scale, double-blind crowd-sourced evaluation (N ≈ 5,000), the SC-MLM model outperformed human-submitted headlines. It achieved a 2.07% improvement in mean rating over human headlines and generated 6.53% more headlines rated as a perfect "3 out of 3" (Correct and Attractive).
• Grammar: The model demonstrated superior grammatical correctness, with 98.13% of SC-MLM headlines judged as grammatically correct, compared to 93.14% for human-submitted headlines.
• Ablation Studies: Experiments confirmed that pre-training, using BERT Large, masking only the headline (not source titles), and utilizing multiple products were all critical to performance. Using only a single product resulted in lower quality, and masking source titles reduced performance.

Significance and Claims
The paper claims that this work represents a state-of-the-art (SOTA) solution for programmatic ad headline generation. The authors emphasize that their method successfully bridges the gap between the expressive power of pre-trained Transformers and the specific requirements of advertising (generalizing multiple products, optimizing for quality metrics).

Crucially, the authors note that the Reinforcement Learning component only modifies the training procedure. It does not change the inference setup or introduce additional latency, making the solution practical for real-world, large-scale deployment. The paper concludes that this approach is not limited to advertising but can be applied to other NLG tasks such as summarization and translation where optimizing for non-differentiable metrics is desired.