💻 computer science

Multi-Ecosystem Modeling of OSS Project Sustainability

This paper presents an empirical study and quantitative analysis that develops foundation-specific sustainability models and a project triage system based on sociotechnical profiles, demonstrating their effectiveness in forecasting outcomes and guiding recovery strategies for open-source projects across diverse foundations and GitHub.

Original authors: Arjun Ashok, Nafiz Imtiaz Khan, Swati Singhvi, Stefan Stanciulescu, Zhouhao Wang, Vladimir Filkov

Published 2026-02-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Arjun Ashok, Nafiz Imtiaz Khan, Swati Singhvi, Stefan Stanciulescu, Zhouhao Wang, Vladimir Filkov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a talented chef who just invented a delicious new recipe. You want to share it with the world, but you know that cooking alone is hard. You need a kitchen, ingredients, and a team to keep the recipe alive for years.

In the world of software, this "recipe" is an Open Source Project, and the "kitchens" are Foundations like Apache, Eclipse, and OSGeo. These foundations offer mentorship, rules, and a community to help projects survive. But here's the problem: Every kitchen has a different style. Some are strict and formal (like a Michelin-star restaurant), while others are more casual (like a food truck).

The researchers in this paper asked a big question: "Can we build a single 'crystal ball' that predicts if any software project will survive, no matter which kitchen it's in?"

Here is the simple breakdown of their journey and what they found.

1. The Problem: One Size Does Not Fit All

The team looked at thousands of software projects. They noticed that what makes a project successful in the Apache kitchen (which loves email discussions and strict voting) is different from what makes a project succeed in the Eclipse kitchen (which is very corporate and structured) or OSGeo (which focuses on maps and geography).

The Analogy: Imagine trying to predict if a soccer player will become a star. If you train your prediction model only on players from the English Premier League (fast, physical), it might fail miserably when trying to predict the success of a player in Brazilian Jiu-Jitsu (technical, ground-based). The skills are different, even though both are "sports."

2. The Experiment: Testing the Crystal Ball

The researchers built three different types of "crystal balls" (AI models) to predict if a project would:

Graduate: Become a stable, self-sustaining project (The "Success" outcome).
Retire: Die out or be abandoned (The "Failure" outcome).

They tested these models in two ways:

The "Home Court" Test: Training the model on Apache data and testing it on Apache data.
The "Road Trip" Test: Training the model on Apache data and testing it on Eclipse data.

The Result:

Home Court: The models were amazing! They predicted success with over 95% accuracy.
Road Trip: The models stumbled. When they tried to apply Apache's rules to Eclipse projects, accuracy dropped significantly. The "crystal ball" got foggy because the rules of the game had changed.

3. The Solution: The "Project Matchmaker" (OSS-ProF)

Since one single model couldn't handle every kitchen, the researchers invented a smart Router (which they named OSS-ProF).

Think of this like a Travel Agent or a Matchmaker:

Step 1: You bring your new software project to the Travel Agent.
Step 2: The Agent looks at your project's "personality" (how many people talk, how often they code, how they collaborate).
Step 3: The Agent says, "Hey, this project acts exactly like an Apache project! Let's send it to the Apache prediction model."
Step 4: The Apache model then predicts the future with high accuracy.

The Magic: By routing the project to the right expert first, the system became incredibly accurate again, even for projects it had never seen before. It realized that to predict the future, you first need to know where the project belongs.

4. The GitHub Twist

They also looked at projects on GitHub that aren't part of any foundation. These are like "street food" projects—free, chaotic, and unregulated.

Finding: If you train a model on "street food" (GitHub) success, it fails to predict "restaurant" (Foundation) success. Why? Because GitHub success is often just about being popular for a week. Foundation success is about building a long-term, organized team.
However: If you train a model on "restaurant" rules, it can actually predict which "street food" projects will become popular. The strict rules of the foundations seem to cover the basics of success, but the reverse isn't true.

5. What Makes a Project Survive? (The Secret Ingredients)

The researchers looked at what specific "ingredients" (features) mattered most.

The Universal Ingredients: No matter the kitchen, having a steady stream of active cooks (contributors) and people who keep showing up month after month (overlap) is crucial.
The Specific Ingredients:
- In Apache, it matters if people talk on email lists.
- In Eclipse, it matters if the code is organized in specific ways.
- In OSGeo, it matters if a few key experts are leading the charge.

The Big Takeaway

You cannot use a single rulebook to manage all software projects.

Don't guess: If you have a new project, don't just throw it into a generic "success" model.
Do match: First, figure out what kind of project it is (which "foundation" it resembles).
Then predict: Use the specific expert model for that type of project.

This paper gives us a GPS for software sustainability. Instead of driving blind, we can now route our projects to the right path, increasing their chances of surviving and thriving for years to come.

1. Problem Statement

Open Source Software (OSS) projects face high failure rates due to competition for resources and the complexity of distributed communities. To mitigate this, projects often join software foundations (e.g., Apache, Eclipse, OSGeo) which provide governance, incubation, and community support. However, these foundations differ significantly in their policies, funding models, and evaluation criteria.

The core challenges addressed in this paper are:

Lack of Generalizability: Existing sustainability models are often trained on a single foundation's data. It is unclear if a model trained on Apache projects can predict the sustainability of Eclipse or OSGeo projects.
Conceptual Misalignment: The definition of "success" (often used for GitHub projects, based on stars/forks) differs from "sustainability" (used in foundations, based on governance maturity and graduation).
Decision Support Gap: There is no systematic framework to help a new project decide which foundation is the best fit, nor a unified way to predict sustainability across heterogeneous ecosystems.

2. Methodology

2.1 Dataset Construction

The authors curated a multi-ecosystem dataset comprising 464 projects across four environments:

Apache Software Foundation (ASF): 262 projects (205 Graduated, 57 Retired).
Eclipse Foundation (EF): 161 projects (142 Graduated, 19 Retired).
OSGeo Foundation (OF): 20 projects (13 Graduated, 7 Retired).
GitHub (Non-Foundation): 21 projects (5 Successful, 16 Unsuccessful), labeled based on Joblin et al.'s criteria.

Data Pre-processing:

Cleaning: Excluded non-source code files and removed bot accounts (using substring matching and activity thresholds).
De-aliasing: Merged multiple user accounts belonging to the same individual using name normalization and Jaro-Winkler similarity.
Reply Inference: Reconstructed social interaction graphs by inferring reply relationships in unstructured email/issue threads based on temporal and sequential cues.
Scope: Analysis was restricted to the incubation phase (or early development for GitHub) to prevent label leakage (e.g., avoiding using post-retirement inactivity as a predictor).

2.2 Feature Engineering (Sociotechnical Networks)

The study utilizes 13 sociotechnical features derived from monthly network traces:

Social Features: Number of active developers, clustering coefficient, graph density, number of disconnected components, weighted mean degree, and social network overlap (developers active in consecutive months).
Technical Features: Technical network density, developers per file, unique files, files per developer, and technical network overlap.
Hybrid Feature: Developers contributing to both social and technical aspects.
Normalization: A project-scale-aware normalization was used (dividing by the number of active developers) to handle varying project sizes without distorting signals.

2.3 Modeling Architecture

Three Deep Neural Network architectures were proposed to handle the longitudinal time-series nature of the data:

Bi-LSTM: Bidirectional Long Short-Term Memory with attention mechanisms, layer normalization, and residual connections.
Transformer: Replaces recurrent units with self-attention mechanisms for parallel processing of sequences.
Bi-DLSTM: Bidirectional Dilated LSTM to capture multi-scale temporal dependencies with a larger receptive field.

Training Strategy:

Loss Function: Focal Loss was employed to address significant class imbalance (more graduated/successful projects than retired/failed ones).
Validation: 5-fold stratified cross-validation.
Interpretability: SHAP (SHapley Additive exPlanations) values were used to determine feature importance.

2.4 The OSS-ProF Framework

To address cross-ecosystem prediction, the authors introduced OSS-ProF (OSS Project-Foundation Router). This is a two-stage pipeline:

Router (Classifier): A model that takes a project's sociotechnical features and classifies it into the most analogous ecosystem (ASF, EF, OF, or GitHub).
Predictor: The specific foundation-trained sustainability model (e.g., the ASF model) is applied to the project based on the router's classification.

3. Key Contributions

First Multi-Foundation Sustainability Study: The first empirical study to systematically analyze sustainability prediction across Apache, Eclipse, and OSGeo, alongside GitHub.
OSS-ProF Router: A novel framework that routes projects to the most appropriate ecosystem-specific model, significantly improving prediction accuracy compared to a "one-size-fits-all" approach.
Construct Validation: Empirical evidence distinguishing "Success" (GitHub) from "Sustainability" (Foundations), showing they are related but not interchangeable constructs.
Feature Importance Analysis: Identification of universal vs. ecosystem-specific drivers of sustainability.

4. Key Results

4.1 Intra-Foundation Performance

Models trained and tested within the same foundation achieved high performance:

ASF: Bi-LSTM F1-score = 95.60%.
EF: Bi-LSTM F1-score = 90.21%.
OF: Bi-DLSTM F1-score = 89.29%.
GitHub: Bi-DLSTM F1-score = 97.62%.

4.2 Cross-Foundation Transferability (RQ1)

Limited Transferability: Models trained on one foundation generally failed when applied to another (F1-scores dropped by 15–40%).
Asymmetry: Models trained on structured foundations (ASF/EF) could predict GitHub success reasonably well (F1 ~85–93%), but models trained on GitHub failed to predict foundation sustainability (F1 ~30–40%). This suggests foundation sustainability requires governance signals absent in raw GitHub success metrics.

4.3 Effectiveness of OSS-ProF (RQ2)

The two-step routing pipeline significantly outperformed direct cross-foundation application:

ASF-like projects: F1 improved from ~70% (direct transfer) to 96.20%.
EF-like projects: F1 improved to 96.15%.
OSGeo-like projects: F1 improved to 92.83%.
GitHub-like projects: F1 improved to 95.76%.

4.4 Feature Importance (RQ4)

Universal Signals: Social network overlap ( $s\_net\_overlap$ ) and number of active contributors ( $s\_num\_nodes$ ) were consistently predictive across all ecosystems.
Ecosystem-Specific Signals:
- Eclipse: Clustering coefficient was highly predictive.
- OSGeo: Weighted mean degree (influence of specific contributors) was critical.
- GitHub: Temporal continuity of file-level collaboration was more important.

4.5 Case Studies

Analysis of four failed projects (Droids, SmartHome, Geos, Forem) confirmed that the models correctly tracked temporal degradation. The features that declined most sharply during the "downturn" period aligned with the global feature importance rankings (e.g., loss of developer overlap and contributor engagement).

5. Significance and Implications

Practical Utility: The OSS-ProF framework provides a data-driven tool for project maintainers and foundation mentors. It can triage incoming projects, suggesting the most suitable foundation or predicting the likelihood of sustainability based on early sociotechnical traces.
Theoretical Insight: The study validates Contingency Theory in OSS: there is no universal "best" governance model. Sustainability is context-dependent, and effective prediction requires matching the project's sociotechnical profile to the specific ecosystem's norms.
Actionable Recommendations:
- Projects should not rely on a single global model for sustainability prediction.
- Early intervention strategies should be tailored to the specific ecosystem (e.g., focusing on clustering in Eclipse-like projects vs. contributor retention in Apache-like projects).
- "Success" on GitHub is a necessary but insufficient condition for "Sustainability" in a foundation; governance maturity and community diversity are distinct, critical factors.

6. Limitations & Future Work

Sample Size: OSGeo and GitHub datasets were small, leading to higher variance in results.
Survivorship Bias: The study relies on digital traces; projects that fail silently without generating data are excluded.
Coverage: The study covers three major foundations but misses single-vendor (e.g., Android) or cloud-native (e.g., CNCF) foundations.
Future Work: Expanding to more foundations, exploring hybrid architectures, and developing unified sustainability measurement frameworks.