Benchmark Datasets for Lead-Lag Forecasting on Social Platforms

Technical Summary: Benchmark Datasets for Lead-Lag Forecasting on Social Platforms

Problem Definition: Lead-Lag Forecasting (LLF)

The paper formalizes Lead-Lag Forecasting (LLF) as a distinct time-series prediction paradigm. Unlike traditional rolling forecasting (predicting the immediate future of a single channel) or temporal point processes (modeling discrete event arrivals), LLF addresses the prediction of a temporally shifted, correlated outcome channel based on early interactions.

Formally, given an entity $i$ (e.g., a paper or repository), the task is to predict a downstream outcome $z^{(i)}$ derived from a "lag" channel $y$ at a future horizon $H$, using only observations from a "lead" channel $x$ (and potentially early lag data) within a lookback window $\tau$.

• Lead Channel ($x$): Early-phase interactions (e.g., views, downloads, pushes, stars).
• Lag Channel ($y$): Delayed impact signals (e.g., citations, sales, forks).
• Key Constraints: The prediction horizon $H$ is significantly larger than the lookback window $\tau$ (e.g., predicting 5-year impact from 30 days of data), and the model must generalize to unseen entities (cross-series generalization) while transferring information across channels (cross-channel prediction).

The authors argue that while such dual-process dynamics are ubiquitous in science, economics, and social media, the field lacks standardized datasets to systematically study them, as existing benchmarks focus on same-channel, short-horizon forecasting.

Methodology and Datasets

To anchor research in LLF, the authors curated and released two high-volume benchmark datasets, documenting all technical details of curation, cleaning, and anonymization.

1. arXiv Dataset:

   • Scope: 2.3 million academic papers.
   • Lead: Weekly/daily access logs (downloads/views) from 1.07B anonymized users.
   • Lag: Citation counts derived from the Semantic Scholar citation graph (approximating citation time via the citing paper's publication date).
   • Task: Predict 5-year citation counts ($H=1825$ days) based on access data at $\tau \in \{30, 100, 365\}$ days.
   • Split: Random splits for training, validation, and testing, with a specific "unlabeled test" set for papers submitted after the training cutoff to ensure no data leakage.

2. GitHub Dataset:

   • Scope: 3 million repositories (specifically those linked to package managers like PyPI/Conda via Ecosyste.ms).
   • Lead: Push events and Star events.
   • Lag: Fork counts.
   • Task: Predict 5-year fork counts based on push/star data at $\tau \in \{30, 100, 365\}$ days.
   • Split: Similar random splits, with repositories having histories shorter than 5 years placed in an unlabeled test set.

Analysis of Dynamics:
The authors verified the existence of lead-lag relationships through statistical analysis:

• Temporal Shift: In arXiv, accesses consistently precede citations; in GitHub, stars typically appear before forks.
• Predictive Power: Pearson correlation and log-log hexbin plots show that early lead signals (accesses, stars) correlate strongly with long-term lag outcomes (citations, forks), often outperforming early lag signals themselves in the short term.
• Distribution: Both datasets exhibit heavy-tailed distributions (power laws), necessitating specific evaluation metrics.

Evaluation Metrics:

• Classification: Identifying high-impact entities (top 8.1% for arXiv, top 11.3% for GitHub). Metrics: AUROC, F1-score, AUC-PR.
• Regression: Predicting numerical counts. Metrics: Mean Absolute Error (MAE), Log-Mean Absolute Error (MAE-log), and Mean Absolute Percentage Error (MAPE) restricted to high-impact entities to avoid instability from low-count noise.

Key Contributions

1. Dataset Release: The provision of two novel, large-scale datasets (arXiv and GitHub) that are the first to jointly test cross-channel and cross-series generalization in a long-horizon setting.
2. Formalization: Defining LLF as a unified forecasting problem distinct from existing time-series tasks.
3. Empirical Validation: Demonstrating through statistical tests that early usage patterns contain meaningful predictive signals for long-term impact.
4. Baseline Benchmarks: Establishing performance baselines using:
   • Parametric/Non-parametric: Logistic Regression (Vanilla, One-feature, Time-MoE embeddings), KNN, MLP.
   • Deep Learning: InceptionTime, Transformer, and Time-MoE (large) embeddings.
   • Ablations: Comparing single-channel vs. multi-channel inputs and same-channel vs. cross-channel prediction.

Results

The baseline experiments yielded several key findings:

• Predictive Signal Exists: Even simple models using a single cumulative feature (e.g., total accesses at day 30) significantly outperform random baselines (e.g., arXiv F1 rises from 0.08 to 0.31; GitHub F1 from 0.11 to 0.29).
• Cross-Channel Superiority: For early horizons (30–100 days), predicting the lag outcome using the lead channel (e.g., accesses $\to$ citations) consistently outperforms using the lag channel itself (e.g., early citations $\to$ future citations).
• Feature Richness: Using multiple channels (e.g., pushes + stars) generally improves performance over single-channel inputs.
• Model Performance:
  • InceptionTime: Demonstrated superior performance in ranking-based metrics (AUC) for long horizons but showed lower F1 scores for short horizons, potentially due to miscalibration.
  • Deep Learning vs. Simple Models: Deep features (Time-MoE embeddings) generally outperformed non-transformed features, though the gains were not always substantial.
  • Regression: KNN performed strongly on MAE metrics, particularly on GitHub, but underperformed deep learning methods on MAPE for high-impact items, likely due to the skewed data distribution where many entities have zero long-term impact.
• Horizon Dependency: Predictive performance improved as the lookback window $\tau$ increased from 30 to 365 days, validating that more early data facilitates better long-term forecasting.

Significance and Claims

The paper positions LLF as a novel forecasting paradigm essential for understanding social and usage data. Its significance lies in:

• Filling a Gap: Addressing the absence of standardized datasets for cross-channel, long-horizon forecasting, which limits the systematic exploration of lead-lag dynamics.
• Real-World Relevance: Moving beyond artificial test cases to real-world settings (scholarly communication and open-source software) where impact unfolds over years and involves complex, non-deterministic mechanisms.
• Foundation for Future Work: Providing an empirical foundation and rigorous baselines for developing scalable, flexible forecasting architectures capable of generalizing to unseen series.

The authors modestly note limitations: the datasets contain aggregate signals rather than individual-level traces or content metadata; they reflect specific platform dynamics at specific times; and the models capture predictive correlations rather than causal relationships. They emphasize that the datasets should not be used to infer direct causal interventions but serve as a tool to understand the promises and pitfalls of predicting long-term outcomes.