💰 quantitative finance

PHBench: A Benchmark for Predicting Startup Series A Funding from Product Hunt Launch Signals

This paper introduces PHBench, a reproducible benchmark linking 67,292 Product Hunt launch signals to Crunchbase funding records to demonstrate that structured launch data statistically predicts Series A outcomes, achieving significant performance gains with an ensemble model while revealing unexpected limitations in zero-shot LLMs and temporal market sensitivity.

Original authors: Yagiz Ihlamur, Ben Griffin, Rick Chen

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yagiz Ihlamur, Ben Griffin, Rick Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a talent scout looking for the next big thing in the startup world. You have a massive list of 67,000 new products that just launched on a popular website called Product Hunt. Your job is to guess which few of these will go on to raise a massive amount of money (Series A funding) within the next 18 months.

The problem? Only about 1 in 128 of these products actually succeeds. It's like trying to find a single golden ticket in a factory full of chocolate bars. Most people guess randomly, or rely on "gut feeling," which isn't very reliable.

This paper introduces PHBench, a new "scorecard" and a set of tools designed to solve this guessing game using math and data instead of gut feelings.

Here is how they did it, explained simply:

1. Building the Map (The Dataset)

The authors gathered a massive list of 67,292 products featured on Product Hunt between 2019 and 2025. They then cross-referenced this list with a giant financial database (Crunchbase) to see which of those companies actually raised money later.

The Result: They found 528 "winners" (companies that raised Series A funding).
The Challenge: Because the winners are so rare (less than 1%), it's a very tricky game to play. If a model just guessed "No" for everyone, it would be right 99% of the time, but it would be useless.

2. The Game Rules (The Benchmark)

To make sure the models are playing fair, the authors created a strict set of rules called PHBench:

The Training Ground: They split the data so models can practice on old data but are tested on new, unseen data (like a final exam).
The Scorecard: They don't just measure "how often you were right." They measure how well you rank the winners. If you put the winning companies at the very top of your list, you get a high score. If you bury them at the bottom, you get a low score, even if you were technically "right" about the losers.
The "F0.5" Metric: This is their special scoring rule. It cares more about not wasting time on fake leads (false positives) than missing a few winners. Think of it like a security guard: it's better to let a few suspicious people pass than to stop 1,000 innocent people.

3. The Contenders: Math vs. AI

The paper tested two types of "guessers":

A. The "Old School" Math Models (Machine Learning)
These are like experienced detectives who look at specific clues:

How many people voted?
How many comments were there?
What day of the week did they launch?
Is the team big?
Is the product about "B2B" (business-to-business) or "AI"?

B. The "Super Smart" AI (Large Language Models / LLMs)
These are the latest, most powerful AI chatbots (like Gemini). The researchers asked them to look at the same clues but without letting them read the product names or descriptions. They only gave the AI numbers (e.g., "500 votes," "Rank #1"). This was to see if the AI could "know" the answer just by understanding the numbers, without reading the story.

4. The Results: Who Won?

The Winner: The Math Detective (Ensemble Model)
The best performer was a "team" of three math models working together.

The Strategy: They combined different models to smooth out the errors.
The Score: They found the winners about 4.7 times better than random guessing.
Key Insight: The most important clues weren't just "how many votes," but the combination of a big team plus high engagement. It's like a sports team: a big team with a star player is more likely to win than just a big team or just a star player alone.

The Loser: The Super Smart AI (LLMs)
Surprisingly, the most advanced AI models performed worse than the simple math models and even worse than a basic statistical baseline.

The Problem: When you strip away the text (names, descriptions) and only give the AI numbers, it gets confused. It acts like a "picker" rather than a "ranker." It might spot the one obvious winner at the very top of the list, but it fails to sort the rest of the list correctly.
The Irony: The "smartest" AI (Gemini 3.1 Pro) actually performed the worst. The authors suggest this might be because the AI was over-corrected to be too cautious when given only numbers.

5. What This Tells Us About the Market

The data didn't just predict winners; it showed us how the market moves:

The Boom and Bust: The model could "see" the 2020–2021 funding boom and the 2022–2023 crash just by looking at the launch signals. This proves the data is real and not just random noise.
The "Team" Signal: The biggest predictor of success wasn't the product idea itself, but the team's ability to rally a crowd. If a large team launches a product and gets a lot of votes, they are much more likely to get funded.
The Niche Matters: Products in specific categories like "APIs," "Payments," and "Fintech" were much more likely to get funded than others, regardless of how many votes they got.

Summary

PHBench is a new, open playground where anyone can test their ideas for predicting startup success. The paper proves that:

Launch signals matter: What happens in the first 24 hours on Product Hunt does predict future funding.
Math beats "Black Box" AI (for now): When you only have numbers and no text, traditional math models are much better at finding patterns than the newest AI chatbots.
Team + Hype = Success: The best signal is a big team getting a lot of attention.

The authors have made all their code, data, and rules public so other researchers can try to beat their score.

Technical Summary: PHBench

Problem Statement
Early-stage venture capital (VC) investors face the challenge of screening thousands of potential startups annually to identify those likely to secure Series A funding. While Product Hunt (PH) provides a structured, time-stamped record of early market reception (upvotes, comments, rankings, maker profiles), the predictive power of these launch signals for subsequent institutional funding remains largely unformalized and unquantified. The core problem is to determine whether observable pre-funding signals from a Product Hunt launch can statistically predict whether a company will raise a Series A round within 18 months, and to establish a reproducible benchmark for evaluating predictive models against this task.

Methodology
The authors construct PHBench, a benchmark dataset and evaluation framework designed to address this prediction task.

Dataset Construction:
- Source: 67,292 featured Product Hunt posts from Q1 2019 to Q4 2025.
- Labeling: Posts are linked to Crunchbase funding records via deterministic domain matching (after normalizing URLs and applying a 50-domain blocklist to exclude platform/hosting domains like GitHub or Webflow).
- Positive Definition: A post is labeled positive if the matched company raises a Series A round between 0 and 548 days (18 months) post-launch. This yields 528 positive labels (0.78% positive rate), creating a highly imbalanced dataset (1:126 ratio).
- Splitting: The data is split into training (70%), validation (10%), and test (20%) sets. Crucially, splits are stratified by year, topic, and label, and enforced at the company level (all posts from the same Crunchbase permalink are in the same split) to prevent label leakage. The test set labels are held out by the maintainers.
Feature Engineering:
- 61 engineered features are derived from launch-day metadata, grouped into: Engagement (votes, comments), Rank (daily/weekly/monthly), Maker (team size, followers), Temporal (launch day, prime window), Topics (24 binary flags), Text (length/word counts), and Interactions (e.g., team size × votes).
- Log transformations are applied to skewed count data.
- A "maker follower" feature is noted to have a temporal leakage limitation (scraped in 2026 rather than at launch), though it is retained with documentation.
Evaluation Protocol:
- Primary Metric: F0.5 score (harmonic mean of precision and recall with $\beta=0.5$ ), prioritizing precision to reflect the high cost of false positives in analyst screening.
- Secondary Metrics: Average Precision (AP), Precision@k (P@50, P@100, P@200), and AUC-ROC.
- LLM Baseline: Three frontier LLMs (Gemini 2.5 Flash, Gemini 3 Flash, Gemini 3.1 Pro) are evaluated in a zero-shot, anonymized setting. Prompts contain only 11 numerical/categorical signals (e.g., rounded vote counts, topic buckets) without semantic content (names, descriptions) to prevent data retrieval.

Key Contributions

Dataset Release: A curated, reproducible dataset of 67,292 PH posts linked to 528 verified Series A outcomes, including a deterministic linkage methodology and a platform blocklist.
Benchmark Framework (PHBench): A standardized evaluation environment with public training/validation splits, private held-out test labels, a public leaderboard (phbench.com), and a five-metric evaluation harness.
Empirical Baselines:
- ML: A systematic comparison of 144 experiments across Logistic Regression, XGBoost, LightGBM, and Random Forest, establishing a champion ensemble.
- LLM: The first quantitative baseline for LLMs on this specific task, evaluating their ability to reason over anonymized numerical signals without semantic context.

Results

ML Performance: The best-performing model is a three-component ensemble (ENS_avg, ENS_ISO, XGB) selected via validation F0.5.
- Test Set Results: Achieves F0.5 = 0.097 and AP = 0.037 (95% CI: 0.024–0.072).
- Statistical Significance: A paired bootstrap test confirms a statistically credible advantage over the Logistic Regression baseline (AP delta: +0.013, $p < 0.001$ ; F0.5 delta: +0.056, $p = 0.016$ ).
- Feature Importance: Interaction terms involving maker count and vote engagement (e.g., makers_x_votes) are the most predictive features, followed by topic categories (API, Payments, Fintech).
LLM Performance:
- All three Gemini models performed below the Logistic Regression baseline on the validation set (Best LLM: Gemini 3 Flash, AP = 0.034 vs. LR AP = 0.044).
- On the test set, the best LLM (Gemini 3 Flash) achieved AP = 0.034, while the champion ML model achieved AP = 0.037.
- Pattern: LLMs exhibited a "picker, not ranker" behavior, concentrating probability mass on high-signal cases but failing to provide a calibrated gradient across the full distribution. Notably, the most capable model (Gemini 3.1 Pro) performed the worst (AP = 0.023).
- Temporal Decay: Both ML and LLM models showed performance decay in 2023–2024 cohorts, tracking the real-world contraction in VC funding, suggesting the models capture genuine market structure rather than noise.

Significance and Claims
The paper claims that PHBench provides the first reproducible framework demonstrating that structured Product Hunt launch signals contain statistically significant predictive information for Series A funding outcomes.

Market Structure: The dataset captures genuine market dynamics, as evidenced by the correlation between model performance decay and the 2020–2021 funding boom versus the 2022–2023 contraction.
ML vs. LLM: In a purely numerical, anonymized regime, structured machine learning models trained on engineered tabular features substantially outperform zero-shot LLMs. The authors suggest that LLMs require rich semantic content (names, descriptions, narratives) to perform effectively in VC prediction tasks, whereas they underperform when restricted to numerical signals.
Practical Utility: The champion model can screen a quarterly batch of ~2,400 featured posts to surface approximately 5 true Series A raises at 10% precision, offering a systematic method to reduce analyst workload.

The authors remain modest regarding the limitations, specifically noting the small number of validation positives (53) which introduces variance in metrics, the conservative nature of domain matching (leading to potential false negatives), and the temporal leakage in follower count features. They position PHBench as a foundation for future work, including text-feature integration and temporal holdout evaluations.