🤖 AI

Controlled Personalization in Legacy Media Online Services: A Case Study in News Recommendation

This industry article demonstrates through an A/B test on a major Norwegian news outlet that "controlled personalization," a hybrid approach combining editorial curation with algorithmic selection, successfully enhances user engagement and content diversity while upholding journalistic values.

Original authors: Marlene Holzleitner, Stephan Leitner, Hanna Lind Jorgensen, Christoph Schmitz, Jacob Welander, Dietmar Jannach

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Marlene Holzleitner, Stephan Leitner, Hanna Lind Jorgensen, Christoph Schmitz, Jacob Welander, Dietmar Jannach

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a traditional newspaper, like a well-respected local librarian who has been curating stories for over 160 years. This librarian (the news organization) has a strict set of values: they want to make sure everyone reads a little bit of everything, from serious politics to local sports, so that the whole community stays informed and connected. They are wary of "algorithms" (robots) taking over because they fear the robot might only show you the same three stories you already like, creating an "echo chamber" where you never hear new ideas.

On the other hand, modern tech companies (like Google News) act like a super-fast vending machine. They know exactly what you want to buy and hand it to you instantly, but they don't care about the "big picture" of your education or the community's health.

The Experiment: The "Controlled Personalization" Recipe

The researchers in this paper asked: Can we mix these two worlds? Can we keep the librarian's values while using a little bit of the vending machine's speed?

They tested this at Aftenposten, a major Norwegian newspaper. They didn't let the robot take the wheel completely. Instead, they created a "controlled" recipe:

The Librarian's Hand (80%): The robot still had to follow the librarian's rules. It prioritized fresh news, popular stories, and articles that editors manually selected.
The Robot's Nudge (20%): The robot was allowed to whisper a suggestion to the system: "Hey, this specific user actually liked sports articles yesterday, maybe show them this one too."

They ran a month-long test (an A/B test) where half the readers saw the old way (100% librarian rules), and the other half saw the new way (80% librarian + 20% robot).

What Happened? The Results

The results were surprisingly positive, even with that small 20% robot nudge. Here is what they found, using simple analogies:

1. Finding the Needle in the Haystack became easier

The Old Way: Readers had to scroll through a long list of news (like flipping through a massive phone book) to find something interesting.
The New Way: Readers scrolled less. Because the robot helped surface relevant articles faster, users found what they wanted with less effort.
The Result: People clicked on more articles (a 14% increase) but actually scrolled fewer times to find them. It was like the librarian started handing you the book you were looking for before you even had to ask.

2. No "Clickbait" Traps

Sometimes, when you trick someone into clicking a link with a flashy headline, they realize the article is boring and leave immediately. This is called a "canceled click."
The Result: Even though people clicked more often, they didn't leave faster. They actually spent a tiny bit more time reading the articles. This proves the robot wasn't tricking them; it was actually helping them find things they genuinely wanted to read.

3. Breaking the "Echo Chamber"

The Fear: The librarians worried the robot would only show users the same popular stories over and over, ignoring niche topics.
The Result: The opposite happened! The robot actually helped users discover a wider variety of topics.
- Diversity: Users clicked on articles from more different sections (like culture, politics, and sports) rather than just sticking to one.
- Popularity: The robot helped less popular, "niche" articles get noticed. It reduced the bias toward only the "blockbuster" hits.
- Analogy: Instead of everyone in the town square only talking about the same three popular songs, the robot helped people discover some great jazz and classical music they wouldn't have heard otherwise.

4. It Worked for Everyone

Whether a user was a "news junkie" (reading 50 articles a day) or a "casual reader" (reading just a few), the robot helped them find better content. Even the casual readers, who usually just skim the headlines, started exploring more diverse topics.

The Big Takeaway

The paper concludes that you don't need to let the robot take over completely to get the benefits of personalization. By keeping the "human editor" in charge of the big decisions (80%) and letting the robot handle the small, personal touches (20%), the newspaper achieved a "sweet spot."

They managed to:

Make users happier and more engaged.
Help users find relevant content faster.
Crucially: Still uphold their journalistic values by ensuring users saw a diverse range of news, avoiding the trap of only showing them what they already agree with.

In short, the study shows that legacy media (traditional newspapers) can safely adopt personalization technology without losing their soul, as long as they keep the "human hand" firmly on the steering wheel.

Technical Summary: Controlled Personalization in Legacy Media Online Services

Problem Statement
Legacy news media organizations face a unique challenge in adopting recommender systems: they must balance the technological benefits of personalization (e.g., increased engagement, retention) with core journalistic values and editorial missions (e.g., promoting diverse viewpoints, avoiding filter bubbles, and ensuring democratic engagement). Unlike news aggregators (e.g., Google News) which often prioritize algorithmic optimization, legacy outlets like Aftenposten typically rely on editorial judgment for prominent content placement. While hybrid approaches exist, there is limited empirical evidence regarding the effectiveness of "controlled personalization"—a strategy where algorithmic recommendations are heavily constrained by editorial rules and weighted conservatively against traditional ranking factors. The paper addresses the gap in understanding how such a constrained, controlled approach impacts both user engagement metrics and journalistic value dimensions in a real-world legacy media environment.

Methodology
The study presents a large-scale A/B test conducted over 34 days (November 30, 2023 – January 2, 2024) on the mobile platform of Aftenposten, a major Norwegian legacy news provider. The experiment involved approximately 58,000 paying subscribers, randomly assigned to a control group (non-personalized) or a treatment group (controlled personalization).

Experimental Setup:
- Control Group: Articles were ranked using a composite score ( $cs_i$ ) based on three non-personalized components: Popularity (clicks in a time window), Recency (publication time and initial news value), and Performance (historical CTR).
- Treatment Group: A fourth component, a personalized relevance score ( $s_{4,u}$ ), was added to the ranking equation. This score was generated by a collaborative filtering model (matrix factorization) trained on user-article interactions from the last 21 days.
- Weighting Strategy: The personalization component was capped at a maximum influence of 20% ( $w_4 = 0.20$ ) of the final score. The remaining 80% was distributed among the non-personalized factors (Popularity, Recency, Performance), ensuring that popular, fresh, and newsworthy content remained dominant. This "conservative" strategy was designed to align with editorial oversight.
Data Collection and Metrics:
The study utilized fine-grained logging, capturing impressions, clicks, scroll depth (reading percentage), and activity duration. The analysis focused on two categories of Key Performance Indicators (KPIs):
1. User Engagement: Click-Through-Rate (CTR), Canceled Click Rate (CCR), Impressions Per User (IPU), Clicks Per User (CPU), Reading Percentage (RP), and Activity Duration (AD).
2. Journalistic Values:
  - Diversity: Measured via Gini Index and distribution of impressions/clicks across article sections.
  - Coverage: Click coverage (fraction of displayed articles receiving at least one click).
  - Popularity Bias: Average Recommendation Popularity (ARP), Average Click Popularity (ACP), and Percentage of Popular Items (PPI).
Statistical Analysis: The authors employed parametric (Student's t-test, Welch's t-test) and non-parametric (Mann-Whitney U) tests, along with permutation tests and bootstrap confidence intervals, to determine statistical significance and effect sizes.

Key Contributions

Empirical Validation of Controlled Personalization: The paper provides rare, real-world evidence that even a modest level of personalization (20% weight) can yield substantial benefits in a legacy media context without compromising editorial control.
Multi-Dimensional Evaluation: Unlike many prior studies that focus primarily on CTR or revenue, this work evaluates personalization across a broad spectrum of metrics, explicitly linking technical outcomes to journalistic values (diversity, coverage, and bias reduction).
Granular Behavioral Insights: The study analyzes not just if users click, but how they interact (scrolling effort, reading depth) and what they consume (section diversity, popularity bias), offering a nuanced view of user behavior changes.
Industry Implementation: The paper details the practical integration of a recommendation system within a legacy organization, highlighting the collaborative process between data scientists, editors, and product managers to define constraints and weights.

Results
The A/B test yielded statistically significant positive outcomes across both engagement and journalistic value metrics:

User Engagement:
- Increased Efficiency: The treatment group showed a 14.6% increase in CTR compared to the control group.
- Reduced Effort: Users in the personalized group required 11.7% fewer impressions (scrolls) to find content they clicked on, indicating more efficient discovery.
- Exploration: Despite fewer impressions, the number of clicks per user increased by 1.17%, suggesting users found more relevant content in a shorter feed.
- Reading Depth: While reading percentage remained stable, the average activity duration per click increased by 1.67%, indicating sustained interest.
- Subgroup Consistency: These improvements were observed across low, medium, and high-activity user segments, with the highest activity users showing the largest relative CTR gains (21%).
Journalistic Values:
- Diversity: The personalized group exhibited a more even distribution of impressions and clicks across article sections (lower Gini Index), suggesting a more diversified exposure to topics.
- Coverage: Daily click coverage was significantly higher in the treatment group (approx. 4% increase), meaning a broader fraction of the available article pool was engaged with by users.
- Popularity Bias: The personalized recommendations significantly reduced popularity bias. The Average Recommendation Popularity (ARP) decreased by 16.11%, and the Percentage of Popular Items (PPI) clicked by users decreased by 4.11%. This indicates the system successfully surfaced less popular, niche content that users were willing to engage with.

Significance and Claims
The authors claim that their findings demonstrate a viable path for legacy media to adopt personalization technologies while upholding journalistic values. The study suggests that "controlled personalization" successfully aligns user needs (finding relevant content quickly) with editorial goals (promoting content diversity and reducing filter bubbles).

The paper emphasizes that:

Modest interventions work: Even a small weight (20%) for personalization is sufficient to shift user behavior in desired directions without overwhelming editorial curation.
Beyond CTR: Focusing solely on CTR is insufficient; a multi-dimensional analysis is required to capture the full impact of personalization on both business goals and societal values.
Feasibility for Legacy Media: The successful deployment and positive outcomes validate the approach of integrating algorithmic recommendations into traditional newsrooms through cross-functional collaboration and strict editorial constraints.

The authors remain modest regarding long-term effects, noting that the 34-day period is relatively short and that the effect sizes for some diversity metrics were small, though statistically significant. They posit that these initial positive tendencies are encouraging indicators for long-term alignment of content with user preferences and journalistic missions.

Technical Summary: Controlled Personalization in Legacy Media Online Services

More like this