YC Bench: a Live Benchmark for Forecasting Startup Outperformance in Y Combinator Batches

This paper introduces "YC Bench," a live benchmark using the Y Combinator W26 batch to enable rapid, short-term forecasting of startup outperformance by measuring pre-Demo Day traction signals, thereby reducing the traditional multi-year evaluation cycle to just a few months.

The Gist

Imagine you are a talent scout trying to find the next big superstar in a room full of 200 aspiring actors. The problem? You won't know who actually becomes a movie star for another 7 to 10 years. By then, you've missed the boat, and you can't learn from your mistakes quickly.

This is exactly the problem investors face with startups. They want to predict which companies will become the next Google or Uber, but the "proof" (like a massive sale or huge profits) takes a decade to show up.

Enter "YC Bench."

Think of YC Bench as a "speed-run" version of the startup world. It's a new tool created by Mostapha Benhenda to test how well we can predict startup success, but instead of waiting 10 years, we wait just three months.

Here is the simple breakdown of how it works, using some everyday analogies:

1. The Setting: The "Talent Show" (Y Combinator)

Y Combinator (YC) is like a massive, high-stakes talent show. Every few months, they invite about 200 startups to join a "batch."

• The Catch: They all start at the same time, get the same coaching, and have to perform on the same day (called "Demo Day").
• The Opportunity: Because they are all in the same room, starting at the same time, we can compare them fairly. It's like a 100-meter dash where everyone starts on the same line.

2. The Problem: The "Crystal Ball" is Broken

Usually, investors try to guess who will win the race by looking at the finish line (exits, IPOs). But the finish line is 7 years away.

• The Solution: YC Bench says, "Let's not wait for the finish line. Let's see who is running the fastest right now, three months in."
• The Metric: They created a "Pre-Demo Day Score." This is a single number that tries to guess who is the "fastest runner" before the race is even over.

3. How They Score the Runners

To calculate this score, they look at two things, like a coach judging a runner:

• The "Traction" Score (The Hard Data):
  If a startup has already made money or has thousands of users, they get points. It's like a runner who has already won local races.

  • Analogy: If a startup says, "We have $25,000 in sales," that's a huge point. If they just have a list of people who might buy, that's fewer points.
  • The Twist: They also look at speed. If a company's sales are doubling every month, they get a massive "momentum bonus," like a runner who is sprinting faster than everyone else.

• The "Attention" Score (The Hype):
  Most startups don't have sales data yet. So, the system looks at how much people are talking about them on Google.

  • Analogy: If a startup has no money yet, but 5,000 people are searching for their name on Google, that's like a runner who has a huge fan club cheering them on. The system assumes: "If people are searching for them, they might be onto something."

The Final Score: The system takes the better of the two numbers. If you have money, you win. If you don't have money but you are famous, you still win.

4. The Experiment: The "Google Guess"

The author tested a simple idea: "Can we guess the winners just by looking at who was famous before the race even started?"

• The Setup: They looked at how many times people Googled the startups' names before they even applied to Y Combinator (months before the race started).
• The Result: This simple "Google Guess" was surprisingly good!
  • It correctly identified 55% of the top performers (the ones who would have the most traction by Demo Day).
  • It was 2.75 times better than just guessing randomly.

What does this mean?
It means that even before a startup gets into a famous accelerator, the "hype" and visibility they already have on the internet are strong clues about their future success.

5. Why This Matters

Think of YC Bench as a video game training level for investors and AI researchers.

• Before: Investors had to play the "real game" and wait 10 years to see if their strategy worked. If they lost, they couldn't learn because the game was over.
• Now: With YC Bench, they can play a "training level" that lasts only 3 months. They can test their prediction models, see if they work, fix their mistakes, and try again immediately.

The Bottom Line

This paper introduces a new way to test our ability to predict startup success. Instead of waiting a decade for the results, we can use a clever mix of "sales data" and "Google search volume" to get a reliable prediction in just a few months.

It turns the slow, painful process of investing into a fast, iterative learning loop, helping us find the "superstars" much earlier in the game.

Technical Summary

1. Problem Statement

Forecasting startup success is a critical but notoriously difficult challenge for investors and researchers. The field faces three fundamental constraints:

• Sparse Labels: Meaningful success indicators (exits, IPOs, massive funding rounds) are rare events.
• Slow Evaluation Cycles: These outcomes typically take 7–10 years to materialize, preventing rapid iteration of predictive models.
• Data Bias: Retrospective analyses suffer from survivorship bias and market condition confounding.

Unlike domains like computer vision or NLP, startup prediction lacks standardized, rapidly evaluable benchmarks. The paper argues that the current reliance on long-term outcomes hinders the development of robust machine learning models for early-stage venture capital.

2. Methodology

A. The Benchmark Environment: YC Batches

The authors leverage Y Combinator (YC) batches as a controlled experimental environment.

• Structure: Each batch contains ~200 startups funded simultaneously.
• Constraints: Startups operate under shared constraints and are evaluated at a common endpoint (Demo Day) just three months after the batch begins.
• Advantage: This structure allows for short-term, comparable outcome measurement, reducing cross-domain heterogeneity and enabling evaluation cycles measured in months rather than years.

B. The Prediction Task

• Goal: Identify the top 10% of startups in a batch (the top 20 out of ~200) as they appear the day before Demo Day.
• Rationale: According to YC-specific investor data (Rebel Fund), the top 10% of startups drive 96% of all investor returns.
• Timing: Predictions are ideally made immediately after the batch composition is public, aiming to identify winners before the general crowd at Demo Day.

C. The Metric: Pre-Demo Day Score

To enable rapid evaluation, the authors introduce a composite metric called the Pre-Demo Day Score, which serves as a short-term proxy for long-term outperformance. It combines two signals:

1. Traction Score ($S_{traction}$): For startups with disclosed performance data.
   • Aggregation: Uses a max-of-weighted-metrics approach to handle sparse reporting.
   • Signals: Includes ARR/Revenue (weight 1.0), Pilot Revenue (0.5), LOIs (0.2), Active Users (0.4), etc.
   • Momentum: Includes a velocity multiplier based on month-over-month (MoM) growth: $S_{traction} \times (1 + 10 \cdot g_{MoM})$.
2. Attention Score ($S_{attention}$): For startups without disclosed traction (the majority).
   • Signal: Google search mention counts for the startup's domain during the batch period.
   • Formula: $S_{attention} = w_m \cdot x_m$ (where $w_m = 0.05$).
3. Final Score: The Pre-Demo Day Score is the maximum of the two:
   $$\text{Score} = \max(S_{traction}, S_{attention})$$

D. Evaluation Metrics

Models are evaluated against the observed Pre-Demo Day Scores using:

• Precision@20: The fraction of the top-20 predicted startups that actually appear in the top-20 highest scores.
• Recall@11: The fraction of the 11 known high-traction startups (with disclosed data) recovered within the top-20 predictions.

3. Key Contributions

1. YC Bench: The first live benchmark explicitly leveraging the batch structure of a startup accelerator to forecast early outperformance.
2. Pre-Demo Day Score: A novel KPI combining public traction signals and web visibility to create a rapidly evaluable proxy for long-term success.
3. Live Forecasting Framework: A shift from retrospective analysis to forward-looking prediction, allowing for iteration cycles of ~3–5 months.
4. Open Data: Release of code and data (specifically for the W26 batch) to facilitate reproducibility and further research.

4. Results (Case Study: YC W26 Batch)

The authors tested a simple baseline on the YC W26 batch (196 startups):

• Baseline Model: Ranking startups based on Google search mentions of their domain during the 67 days prior to the YC application deadline (pre-application visibility).
• Performance:
  • Recall@11: 55% (Recovered 6 out of 11 known high-traction startups).
  • Precision@20: 30% (6 out of 20 predicted startups were in the top 20).
  • Lift: 2.75x over a random predictor.
• Observations:
  • The baseline successfully identified high-performers months in advance.
  • Failures occurred primarily in narrow B2B verticals with limited public web presence.
  • Google mentions were highly concentrated (Gini coefficient 0.85), with the top 10% of startups capturing ~82% of total mentions.

5. Significance and Future Work

• Significance: The paper demonstrates that publicly observable signals (like pre-application Google mentions) contain substantial information about startup outperformance, even months before Demo Day. This validates the feasibility of using short-term proxies to train and evaluate forecasting models.
• Implications:
  • Capital Allocation: Investors can identify high-potential deals earlier than the "Demo Day crowd."
  • Research: Provides a standardized environment to apply ML techniques to venture capital, moving beyond sparse, long-term labels.
• Future Directions:
  • Weight Optimization: Moving from heuristic weights to empirically learned weights by correlating early signals with long-term outcomes across historical batches.
  • Data Expansion: Increasing the volume of disclosed traction data across future batches to improve the robustness of the Traction Score.

In conclusion, YC Bench transforms startup forecasting from a slow, retrospective exercise into a rapid, iterative scientific discipline, proving that early digital footprints are strong predictors of future venture success.