Sustained Impact of Agentic Personalisation in Marketing: A Longitudinal Case Study

This longitudinal case study demonstrates that while human-in-the-loop management yields the highest initial engagement lift, agentic personalization systems can autonomously sustain performance gains over an 11-month period, supporting a symbiotic model where humans drive strategic discovery and agents ensure scalable retention.

The Gist

The Big Idea: The "Smart Chef" vs. The "Head Chef"

Imagine you run a massive restaurant that serves millions of customers every day. Your goal is to send out flyers (marketing messages) to bring people back in.

The Old Way (Manual CRM):
In the past, you had a team of chefs manually writing a different flyer for every single customer. They would guess who liked spicy food, who liked dessert, and when they were likely to be hungry.

• The Problem: As the restaurant grew to millions of customers, the chefs got overwhelmed. They couldn't write enough unique flyers, so they started sending the same generic "We miss you!" flyer to everyone. It worked okay for a bit, but people got bored and stopped coming.

The New Way (Agentic Personalization):
The researchers in this paper tried something different. They built a Smart Robot Chef (an AI Agent).

• Instead of writing whole flyers from scratch, the human chefs created a "pantry" of ingredients: different greetings, different offers, different pictures, and different times to send them.
• The Robot Chef's job is to mix and match these ingredients to create the perfect, unique flyer for each customer, instantly.

The Experiment: The 11-Month Test

The researchers wanted to answer a big question: Can the Robot Chef keep the restaurant busy forever on its own, or does it need the Human Chef to keep checking in?

They ran a massive 11-month experiment with 8.8 million users, split into two distinct phases:

1. Phase 1: The "Active" Phase (Months 1–4)

   • What happened: The Human Chef was in the kitchen, actively guiding the Robot. They added new ingredients, tweaked the recipes, and told the Robot, "Hey, try this new greeting for people who like pizza."
   • The Result: This was the best time! The restaurant saw a huge spike in customers. The combination of human creativity and robot speed was a powerhouse.

2. Phase 2: The "Passive" Phase (Months 5–11)

   • What happened: The Human Chef went on vacation. They stopped adding new ingredients or giving new instructions. The Robot Chef was left alone with the pantry it had built during Phase 1.
   • The Fear: Most people thought, "Oh no, without the human, the Robot will get confused, stop learning, and the customers will leave."
   • The Reality: The Robot didn't crash. It didn't stop working. It actually kept the restaurant busy for another 7 months. While it wasn't quite as busy as when the Human Chef was there, it was still doing much, much better than the old "generic flyer" method.

The Key Takeaways (In Plain English)

1. Humans are the "Spark Plugs"
When humans are involved, they act like a spark plug. They provide the initial energy, the new ideas, and the strategic direction. This creates the highest possible boost in performance.

2. Robots are the "Engine"
Once the engine is running, the Robot can keep the car moving for a long time without needing a mechanic to touch it every second. The study showed that the AI didn't just "coast"; it kept learning and adapting to what customers liked, maintaining a high level of success even without human help.

3. The "Symbiotic" Model (The Best of Both Worlds)
The paper suggests we shouldn't choose between "Humans only" or "Robots only." Instead, we should use a partnership:

• Humans do the heavy lifting at the start: They set the strategy, create the creative "ingredients," and teach the system what matters.
• Robots take over the daily grind: They scale that strategy to millions of people, ensuring the performance stays high even when humans are busy doing other things.

The Bottom Line

Think of it like training a dog.

• The Human is the trainer who teaches the dog new tricks and sets the rules.
• The Agent is the dog.
• The Study proved that if you train the dog well for a few months, the dog can actually keep performing those tricks and even figure out small variations on its own for a long time, even if you aren't standing right there with a treat in your hand every second.

Conclusion: You don't need a human to micromanage every single message forever. You need a human to set the stage, and then you can let the AI do the show, keeping the lights on and the customers happy for the long haul.

Technical Summary

1. Problem Statement

Traditional Customer Relationship Management (CRM) in consumer applications relies on static, rule-based messaging strategies managed manually by marketing teams. As user bases scale to millions and content catalogs expand, this manual approach faces a scalability bottleneck:

• Combinatorial Complexity: Matching the right user to the right message at the optimal time exceeds human processing capacity.
• Diminishing Returns: Manual strategies often devolve into generic "broadcast" campaigns that fail to deliver true personalization.
• The "Human-in-the-Loop" Dilemma: While autonomous agents and reinforcement learning (RL) promise scalable personalization, it is unclear whether continuous human oversight is required to prevent performance degradation (model drift, feedback loops) or if a more episodic intervention model is sufficient. Existing literature largely focuses on short-term A/B tests, lacking long-term longitudinal data.

2. Methodology

The authors conducted an 11-month longitudinal case study (Randomized Controlled Trial) with a large-scale consumer delivery application (anonymized) serving 8.8 million users.

Experimental Design

The study was divided into two distinct phases to compare Active Human Management vs. Passive Autonomous Operation:

1. Active Phase (Months 0–4): Marketers acted as "humans-in-the-loop," actively curating content, defining audiences, and adjusting strategies.
2. Passive Phase (Months 5–11): Human intervention ceased. Agents operated autonomously using a fixed library of content components, relying solely on their learned policies to optimize messaging.
3. Control Group: A 90/10 split was used, where 90% of users received the agentic treatment and 10% received standard "Business-As-Usual" (BAU) messaging.

System Architecture

The system utilizes a sequential decision-making framework (based on Abboud et al.) where the agent optimizes a cumulative reward signal (e.g., app opens, conversions).

• State Space ($s_{u,t}$): Encapsulates user demographics, past interactions, and app usage history.
• Action Space ($A$): The agent constructs dynamic message tuples comprising:
  • Content Assembly: Selecting atomic components (value props, greetings, visuals).
  • Timing Strategy: Determining optimal send times based on user habits.
  • Channel Selection: Choosing the delivery medium (Push, Email, SMS).
• Optimization Algorithm: The system uses Thompson Sampling to balance exploration (new strategies) and exploitation (known high-performers).
• Human-Agent Interaction: Humans provide "atomic components" (tones, offers, greetings) rather than full messages. The agent autonomously permutes these atoms to generate variants.

Metrics & Statistical Analysis

• Primary Metrics: Days with Engagement, Direct Opens (push), Intent (upper funnel), and Conversions (lower funnel).
• Economic Metric: Gross Merchandise Value (GMV).
• Statistical Method: Relative lift ($\Delta$) was calculated between treatment and control groups. To handle high variance in sparse conversion data, the authors employed ML-RATE (Machine Learning-based Regression Adjustment for Treatment Effects) to reduce estimation variance while maintaining unbiasedness.

3. Key Contributions

1. Empirical Evidence of Sustained Autonomy: The study provides the first longitudinal evidence that autonomous agents can sustain positive performance lifts over long periods (7+ months) without continuous human content injection.
2. Symbiotic Operational Model: The authors propose a new framework for industry practitioners defining distinct roles:
   • Humans for Strategic Initialisation (driving initial lift and discovery).
   • Agents for Scalable Maintenance (retaining gains and preventing decay).

4. Results

The study yielded significant findings regarding the interplay between human strategy and agent execution:

Performance During Active Phase

Human-guided agents achieved the highest relative lift across all metrics:

• Direct App Opens: +65.3% increase.
• Engagement Days: +2.8% increase.
• Intent & Conversion Days: +0.3% increase.
• Key Vertical Conversions: +1.2% increase.

Performance During Passive Phase (Autonomy)

Crucially, when human intervention stopped, performance did not revert to baseline. The agents successfully sustained significant gains:

• Notification Clicks: +57% lift sustained.
• Engagement Days: +2.4% lift sustained.
• Intent/Conversion Lift: +0.2% sustained.
• GMV: Lifts translated proportionally to economic value with no adverse effects.

Comparative Analysis (The "Delta")

While the system remained robust in the passive phase, the marginal value of human oversight was quantified by the drop in performance when humans stepped away:

• Overall Engagement: Dropped by ~15% compared to the active phase.
• Notification Opens: Dropped by ~12%.
• Intent/Conversion Days: Dropped by ~26%.
• Trend Observation: Performance rose during the active phase, plateaued for the first 4 months of the passive phase, and then showed a slight decline toward the end of the trial, suggesting that while agents are robust, they eventually benefit from human "creative refreshment."

5. Significance & Conclusion

This paper challenges the binary view that CRM must be either fully manual or fully autonomous. Instead, it advocates for a Symbiotic Model:

• Agents as the Baseline: Autonomous systems are capable of preventing the "cliff-edge" decay typical of static rule-based campaigns, maintaining a high-fidelity retention layer even without human input.
• Humans as the Multiplier: Human intervention provides a critical "performance premium" (an additional 12–26% lift), essential for strategic novelty, creative discovery, and setting up the initial component library.

Future Outlook:
The authors identify the manual production of "atomic content" as the current bottleneck. They propose that Generative AI (LLMs) could automate the creation of diverse message variants, potentially closing the performance gap between the "active" and "passive" phases. This would allow agents not just to select the best existing components, but to dynamically generate optimal messages for every user context, further enhancing the scalability of agentic personalization.