Predicting Custom-Feed Returns for New Bluesky Posts: A Prospective Study
This paper introduces a novel cold-start routing task for predicting whether newly published Bluesky posts will appear in specific custom feeds, presenting a large-scale benchmark dataset and demonstrating that LambdaRank achieves superior performance in ranking feeds likely to return new content.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine the internet as a giant, bustling town square where everyone shouts out their thoughts, jokes, and news all at once. In this chaotic square, there are thousands of different "clubs" or "newsstands" (called custom feeds) that people run. Each club has its own rulebook for deciding which shouts from the square are interesting enough to put on their bulletin board. Usually, when a new person joins a club, the club needs to know who they are to recommend friends. But here's the twist: in this specific town, the new shouts (posts) are the ones that don't have a history yet. They are the "cold starts." The big question for the town is: How do we guess which clubs will want to put a brand-new shout on their board before anyone has even seen it? This is the puzzle of "cold-start routing." It's like trying to predict which specific magazine editors will love a story you just wrote, the moment you finish typing the last period, without having any past sales data to help you.
This paper tackles that exact puzzle for Bluesky, a social media platform where these custom feeds are a major feature. The researchers, Yipeng Wang and Mohit Singhal, set out to build a system that acts like a super-smart matchmaker. Their goal was to take a brand-new post and instantly rank thousands of potential feeds to guess which ones would "return" (show) that post in their top results within the next 24 hours.
To do this, they didn't just guess; they built a massive, time-traveling experiment. They set up a "monitoring panel" of 5,000 different feeds and watched them for a week. They collected 17.804 million public posts. For each post, they waited to see if it actually appeared in the top 50 results of any feed within 24 hours. If it did, they marked it as a "hit." This created a dataset of 1.865 million observable connections between posts and feeds. They call this a "collect-first, label-later" approach because they gathered the raw data first and only labeled it as a success or failure after the future had actually happened, ensuring their predictions were tested against real outcomes, not just guesses.
The team then tried several different computer brains to solve the matching game. They started with simple rules, like "does the author usually post in this feed?" or "does the text look similar to what this feed likes?" These simple methods were okay, but not great. Then, they tried more complex models that could understand the meaning of the text, not just the words. The winner of the competition was a model called LambdaRank.
Here is what the numbers tell us about how well it worked. The researchers tested their system on two separate days of data. Out of all the posts they tested, they focused on the ones that actually got picked up by at least one feed (about 9.04% of the total). For these posts, the LambdaRank model managed to put the correct feeds in the Top 10 list 73.61% of the time (a metric called capped Recall@10). It also scored 0.6127 on a scale of how well it ordered the best matches (NDCG@10) and got a "hit" (finding at least one correct feed in the top 10) 77.49% of the time.
The paper suggests that while the model is very good, it's not perfect. The biggest bottleneck isn't the ranking brain itself, but the first step: finding the right candidates. Even if the model is perfect, it can't pick a feed if it never looked at it in the first place. The researchers found that their system only looked at a small slice of the possible feeds (a "candidate set" of up to 370 feeds out of the 4,233 ranked ones). If they had been able to look at every single possible feed, the system could have performed even better, reaching a theoretical "ceiling" of 0.8448. This means the next big step isn't necessarily making the brain smarter, but making the search wider.
In short, the paper proves that we can successfully predict which custom feeds will pick up a new post, even before the post has any history. It shows that by combining simple text matching with deep understanding of meaning, we can route new content to the right places. However, it also warns that we are currently limited by how many feeds we can check at once. The system is a strong start, but there is still room to grow as the "town square" gets even busier.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.