💻 computer science

Leveraging Language Models to Discover Evidence-Based Actions for OSS Sustainability

This paper presents a reproducible, two-layer RAG-based pipeline that leverages large language models to systematically mine software engineering literature and transform scattered empirical findings into 1,312 structured, evidence-based actionable recommendations (ReACTs) to guide Open Source Software projects toward sustainability.

Original authors: Nafiz Imtiaz Khan, Vladimir Filkov

Published 2026-02-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nafiz Imtiaz Khan, Vladimir Filkov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a ship (an Open Source Software project). You have a high-tech radar (an AI model) that can tell you, "Warning! Storm coming in 3 months! You might sink!"

But here's the problem: The radar is great at predicting the storm, but it's terrible at telling you what to do about it. It doesn't say, "Batten down the hatches," or "Call the crew for extra shifts." It just screams "DANGER!" and leaves you panicking.

This paper is about building a Cheat Sheet that tells captains exactly how to fix their ships, based on decades of research from the world's best navigators.

Here is the breakdown of how they did it, using simple analogies:

1. The Problem: The "Prediction vs. Action" Gap

For years, researchers have built models that can predict if a software project will fail (like a weather forecast). But these models are like a doctor saying, "You have a fever," without prescribing medicine. They use complex, abstract signals (like "code complexity" or "developer network centrality") that are hard for a project manager to act on.

2. The Solution: Mining the Library with a Robot Librarian

The authors realized that the answers were already written down in thousands of academic papers. But reading 829 papers (from top conferences like ICSE and FSE) to find specific advice is impossible for a human to do quickly.

So, they built a Robot Librarian (a Large Language Model, or LLM) to do the heavy lifting.

The Library: 829 research papers about Open Source Software.
The Robot: An AI called Mixtral-8x7B (think of it as a super-smart, fast reader).
The Task: The robot's job is to read every paper and extract "ReACTs" (Researched Actionables).

3. What is a "ReACT"?

Think of a ReACT as a recipe card. It's not just a vague suggestion; it's a specific instruction backed by proof.

The Action: "Add a 'Welcome' bot to your chat."
The Impact: "New contributors will stay 2x longer."
The Evidence: "A study of 12 projects proved this works."

4. The Two-Layer Filter (The "Quality Control" Station)

AI can sometimes "hallucinate" (make things up). To prevent this, the authors built a two-step quality control process, like a factory assembly line:

Layer 1: The Extraction (The Miner): The robot reads the papers and pulls out all the potential recipe cards.
Layer 2: The Refinement (The Inspector): The robot goes back and checks its own work. It asks itself three tough questions:
1. Did the paper actually say this? (If no, throw it away—it's a hallucination).
2. Did the paper say what the result would be? (If no, mark it as "Impact Unknown").
3. Did the paper provide proof? (If no, mark it as "Evidence Unknown").

They also checked if the advice made logical sense (Soundness) and if it was clear enough to follow (Preciseness).

5. The Result: A Massive Cookbook

After processing 829 papers, the robot found 1,922 potential recipe cards.

1,312 of them were "Gold Standard": They were logical, clear, and had both a predicted impact and scientific proof.
These were sorted into 8 categories, like "Onboarding New Contributors," "Testing Code," "Security," and "Community Management."

6. Putting It to the Test: The "Ship Simulator"

To prove this works, the authors tested it on real-world projects (Apache Software Foundation projects) using a tool called APEX.

The Scenario: They looked at a project called CommonsRDF that was struggling. The APEX tool showed the ship was taking on water (low contributors, low code quality).
The Fix: Instead of guessing, the maintainers looked at the "Cheat Sheet."
- Problem: Not enough new people joining.
- Solution from Cheat Sheet: "Have mentors acknowledge new pull requests." (Proven to increase retention).
- Problem: Code is messy.
- Solution from Cheat Sheet: "Sync branches before big changes." (Proven to reduce conflicts).

They showed that by following these evidence-based steps, a project can steer itself away from disaster.

The Big Takeaway

This paper bridges the gap between Academic Theory and Real-World Practice.

Before this, if a project was failing, a maintainer had to guess what to do or read hundreds of papers to find a clue. Now, they have a structured, evidence-based menu of actions. If your project has "low contributor retention," you don't guess; you look up the "Onboarding" section of the menu, pick the proven action, and do it.

It turns the "black box" of software research into a practical toolkit for keeping the digital world running.

1. Problem Statement: The Prediction-Action Gap

Open Source Software (OSS) underpins the global digital economy, yet over 90% of OSS projects eventually fail. While recent Machine Learning (ML) models have achieved high accuracy in predicting OSS sustainability risks using socio-technical features (e.g., developer activity, code churn), they suffer from a critical limitation: they are not actionable.

The Gap: Predictive models tell maintainers that a project is at risk but not what to do about it. The features driving these predictions are often abstract signals rather than concrete practices.
The Opportunity: Decades of empirical Software Engineering (SE) research contain a vast, underutilized body of evidence regarding concrete practices that improve project health. However, manually extracting these insights from thousands of papers is infeasible.
The Goal: To bridge the gap between prediction and intervention by using Large Language Models (LLMs) to mine SE literature and extract ReACTs (Researched Actionables)—concise, evidence-linked recommendations for maintainers.

2. Methodology

The authors propose a scalable, reproducible pipeline combining Retrieval-Augmented Generation (RAG) with a two-layer prompting strategy to extract and validate ReACTs from 829 top-tier SE papers (ICSE and FSE, 2000–2025).

A. Data Collection & Preprocessing

Corpus: 829 articles from ICSE and FSE containing "Open Source" or "OSS" in titles/abstracts.
RAG Pipeline: To ensure context accuracy and prevent hallucinations, the authors built a local RAG system.
- PDFs are parsed into single text chunks (preserving full article context).
- Text is embedded using instructor-large and stored in ChromaDB.
- The LLM retrieves the full article context before generating recommendations.

B. Two-Layer Extraction & Refinement Pipeline

The process involves three distinct tasks executed by the LLM:

Layer 1: ReACT Derivation (Extraction)
- Goal: Extract candidate ReACTs from the text.
- Experiment: Evaluated 4 open-source LLMs (Llama3-8B, Mixtral-8x7B, MistralLite-7B, Mistral-Nemo-12B) against 3 prompting techniques (Zero-Shot, Chain-of-Thought, Reason+Action).
- Selection: Mixtral-8x7B with Chain-of-Thought (CoT) prompting was selected as the best performer based on BLEU, ROUGE, BERTScore, and METEOR metrics against a manually annotated gold set (10 papers).
Layer 2: Refinement & Reliability (Filtering)
- Goal: Filter hallucinations and extract supporting data.
- Follow-up Prompting: A sequential binary/descriptive questioning process is applied to each candidate ReACT:
  - Hallucination Check: Does the article explicitly mention this action? (If No $\rightarrow$ Discard).
  - Impact Extraction: Does the article state the impact? (If Yes $\rightarrow$ Extract; If No $\rightarrow$ Classify as "Actionable without Impact").
  - Evidence Extraction: Does the article provide empirical evidence? (If Yes $\rightarrow$ Extract; If No $\rightarrow$ Classify as "Actionable without Evidence").
Layer 3: Reliability Assessment
- Goal: Evaluate logical consistency and clarity.
- Soundness: Is the recommendation logically consistent and free of contradictions?
- Preciseness: Is the recommendation clear, specific, and actionable?

C. Validation

Human Sanity Check: Two independent annotators evaluated a stratified sample of 100 ReACTs.
Inter-Annotator Agreement (IAA): High agreement (94–98%) was observed between humans and the LLM on soundness, preciseness, and impact/evidence presence.

3. Key Contributions

Scalable Extraction Pipeline: Successfully scaled ReACT extraction from a manual study of 186 papers to 829 papers using an automated LLM-RAG pipeline.
Optimized Model Configuration: Identified Mixtral-8x7B with Chain-of-Thought prompting as the optimal configuration for extracting actionable insights from technical literature.
Curated ReACT Catalog: Produced a validated dataset of 1,922 ReACTs, of which 1,312 (68%) met strict quality criteria (Sound, Precise, with explicit Impact and Evidence).
Practice-Oriented Categorization: Organized ReACTs into 8 categories (e.g., New Contributor Onboarding, Code Standards, CI/CD Automation) to facilitate mapping to specific project needs.
Operational Integration (Case Studies): Demonstrated practical application by mapping ReACTs to APEX (Apache Project Explorer) sustainability signals. The authors showed how maintainers can use APEX to detect "downturns" and select specific ReACTs to intervene.

4. Key Results

Extraction Volume: From 829 papers, 2,023 candidate ReACTs were initially extracted.
Quality Filtering:
- 101 were hallucinations (not in source text) and removed.
- 1,673 (87%) had explicit impact statements.
- 1,458 (76%) had explicit empirical evidence.
- 1,901 (99%) were Sound (logically consistent).
- 1,697 (88%) were Precise (clear and specific).
Final Catalog: 1,312 ReACTs met all criteria (Sound + Precise + Impact + Evidence).
Category Insights:
- Automated Testing/QA had the highest volume (31.6% of ReACTs).
- New Contributor Onboarding had the lowest (3.1%), reflecting a historical research bias toward technical over social practices.
- DevOps/CI/CD categories showed the highest completeness (85.4%), while Community Collaboration showed the lowest (54.1%), suggesting technical practices are easier to operationalize into evidence-based actions than social ones.

5. Significance and Future Work

Bridging Research and Practice: This work transforms scattered academic findings into a structured, searchable catalog of "what works," directly addressing the "prediction-action gap" in OSS sustainability.
Actionable Guidance: It provides maintainers with a concrete workflow: Detect risk via tools (like APEX) $\rightarrow$ Map to ReACT categories $\rightarrow$ Implement evidence-based interventions.
Limitations:
- Venue Bias: The corpus is limited to ICSE/FSE, potentially underrepresenting practitioner-focused venues (e.g., MSR, industry reports).
- Model Constraints: Hardware limitations prevented the use of larger models (e.g., 300B+ parameters) or advanced prompting (e.g., Tree-of-Thought), which might yield higher nuance.
- Verification: While the ReACTs are evidence-based (linked to studies), the paper does not empirically verify that implementing these specific ReACTs caused project recovery in the case studies (which were retrospective).

Conclusion: The study establishes a reproducible framework for using LLMs to convert decades of SE research into a practical, evidence-based toolkit for OSS maintainers, moving the field from merely predicting failure to actively preventing it.