Technical Summary: LLM-Based Intelligent Notification Composition

Problem Statement

Push notifications serve as a critical channel for user re-engagement on digital platforms, yet current optimization efforts have disproportionately focused on who to notify, when to notify, and what to recommend. The final stage of the pipeline—how to communicate the recommendation (message composition)—remains the least optimized. Existing systems rely heavily on template-based generation with slot-filling personalization. While efficient, this approach hits a "structural ceiling": it can insert relevant values but cannot naturally compose context, explain relevance, or frame novelty compellingly. Consequently, users experience habituation, where repeated exposure to structurally identical messages leads to channel invisibility.

Furthermore, there is significant analytical confusion in the industry regarding the attribution of engagement gains. Reported improvements in Click-Through Rates (CTR) are often broadly attributed to "AI notifications," even when the uplift may stem from upstream improvements in targeting, ranking, or timing rather than the message text itself. This conflation obscures the specific value of language generation and misleads engineering investment.

Methodology

The paper employs a PRISMA-guided systematic survey of academic and industrial literature (January 2018 – March 2026), screening 142 sources to select 28 high-quality references (18 Tier 1 peer-reviewed, 7 Tier 2 engineering reports, 3 Tier 3 contextual).

The analysis proceeds through three main phases:

1. Definition and Decomposition: The authors define "message quality" not merely as a proxy for CTR, but as a multi-dimensional construct. They disentangle the message generation layer from adjacent pipeline components (audience selection, content ranking, send-time optimization) to isolate the specific contribution of language generation.
2. Comparative Analysis: The paper contrasts template-based capabilities against Large Language Model (LLM) capabilities across six defined dimensions, supported by production evidence from domains including social media, food delivery, and e-commerce.
3. Architectural Synthesis: Based on the survey, the authors propose a unified, end-to-end architectural framework that integrates retrieval, generation, ranking, and governance, including a decision framework for when to deploy LLMs versus templates.

Key Contributions

1. A Six-Dimension Framework for Message Quality

The paper moves beyond CTR as the sole metric, defining message quality along six concrete dimensions where LLMs offer distinct advantages over templates:

• Contextual Relevance: LLMs synthesize multiple signals (time, location, weather, history) into a coherent narrative rather than inserting static slot values.
• Clarity: LLMs optimize phrasing for brevity and natural grammar on constrained lock-screen surfaces, avoiding the awkward syntax common in slot-filled templates.
• Actionability: LLMs tailor calls-to-action to the user's specific situation (e.g., reframing an abandoned cart from an obligation to an opportunity with specific incentives).
• Novelty Handling: LLMs bridge the gap between known user preferences and new items, reducing the perceived risk of discovery.
• Linguistic Freshness: LLMs generate semantic variety across exposures, breaking the habituation cycle that plagues static templates.
• Persuasive Appropriateness: LLMs can dynamically select persuasive frames (urgency, social proof, value) based on user segments, though this requires guardrails to prevent manipulation.

2. Architectural Attribution Analysis

The paper provides a rigorous disentanglement of the notification pipeline. It argues that observed engagement gains are frequently misattributed to text generation when they arise from interaction effects across the full stack.

• Targeting, Ranking, and Timing are identified as distinct problems with their own optimization levers.
• Message Generation is the specific layer where LLMs directly impact quality.
• Evidence: The paper cites controlled deployments (e.g., DoorDash holding item recommendations constant) where framing changes alone yielded a +1.8% engagement lift, isolating the language contribution.

3. The Binding-Constraint Decision Framework

The paper introduces a three-criterion framework to determine when LLM generation is the "binding constraint" (i.e., when it is worth the compute cost) and when templates remain superior:

1. Framing Variance: Does the content admit multiple plausible, meaningfully different framings?
2. Linguistic Sensitivity: Is user response sensitive to how the message is phrased, or only to what is recommended?
3. Context Richness: Does the system possess sufficient grounded context to support non-trivial composition?

The authors argue that for transactional alerts (e.g., password resets, delivery confirmations), templates are superior due to low variance and high clarity requirements, whereas LLMs excel in re-engagement, discovery, and recommendation scenarios.

Results and Empirical Evidence

The paper synthesizes production data from 28 sources, revealing a consistent pattern: LLM gains scale inversely with baseline sophistication.

• High Gains: Systems replacing static, under-optimized templates see CTR improvements ranging from +8% to +14.5%. For example, the PushGen system reported a +14.08% CTR lift and 3.2× greater lexical diversity.
• Moderate Gains: Systems where LLMs compete against already-mature slot-filling or optimized baselines see more modest gains of +1% to +2.5%.
• Specific Findings:
  • Freshness: LLM-generated messages maintained engagement levels 18% higher than templates after 4 weeks of sustained delivery.
  • Conversion: E-commerce deployments reported +12.5% CTR and +8.3% Conversion Rate (CVR) improvements, attributed largely to improved action framing and persuasive appropriateness.
  • Diversity: Instagram's implementation of a diversity penalty layer showed that fewer, more diverse notifications outperformed homogeneous ones in CTR.

Proposed Unified Architectural Framework

The paper outlines a unified pipeline for LLM-based notification composition:

1. Inputs: User profile, content inventory, and contextual signals.
2. Budget-Aware Router: Applies the three-criterion framework to decide whether to invoke the LLM path or fall back to templates.
3. Generation Path:
   • RAG Retrieval: Injects grounded context (user history, item metadata) into the prompt.
   • Generation: Produces 4–8 candidates using LoRA adapters for style control.
   • Guardrails: Factuality and policy filters remove unsupported claims or prohibited content.
   • Ranking: A pairwise reward model selects the best candidate based on predicted preference.
   • Diversity Control: Prevents fatigue by managing frequency and semantic variety.
   • Send-Time Optimization: A contextual bandit determines the optimal delivery window.
4. Feedback Loop: Counterfactual logging and online learning update the models.

Significance and Claims

The paper claims its primary significance lies in shifting the industry focus from "AI as a magic bullet" to a nuanced understanding of message quality as an independent optimization target.

• Scientific Clarity: By disentangling message generation from targeting and timing, the paper provides a framework for rigorous causal inference in notification systems, preventing the misattribution of gains.
• Practical Guidance: The binding-constraint framework offers a principled basis for engineering investment, explicitly defining when not to use LLMs to avoid unnecessary latency and cost.
• Governance: The paper emphasizes that the increased persuasive power of LLMs introduces new risks of manipulation and requires robust governance (factuality checks, fairness auditing, and regulatory compliance) to prevent "dark patterns."

The authors conclude that while LLMs offer significant untapped potential in the message layer, realizing this value requires moving beyond binary claims of "LLMs work" toward component-isolated evaluation, stronger experimental design, and open benchmarks that disentangle generation from the broader recommender stack.