Agile Story-Point Estimation: Is RAG a Better Way to Go?

1. Problem Statement

In Agile software development, Story Point (SP) estimation is a critical activity during sprint planning, typically performed via Planning Poker. While effective, this manual process is time-consuming, prone to human biases (e.g., anchoring, dominance by senior members), and inconsistent.

• Goal: To automate SP estimation to reduce time and bias while maintaining accuracy.
• Gap: Previous automated approaches (Machine Learning, Deep Learning) often lack interpretability or rely on static feature extraction without leveraging the "context" of similar past tasks in a way that mimics human reasoning.
• Research Question: Can Retrieval-Augmented Generation (RAG), which combines retrieving similar historical tasks with a Large Language Model (LLM) generator, provide a more effective and interpretable solution for SP estimation compared to existing baselines?

2. Methodology

The study proposes a two-stage RAG pipeline applied to 23 open-source software projects (31,960 tasks) from the Tawosi et al. dataset.

A. Data Preprocessing

• Source: Tasks from 23 open-source projects (filtered to remove incomplete tasks, non-Fibonacci SPs, and tasks with post-assignment changes).
• Split: Chronological split (80% training, 20% testing) to prevent data leakage from future tasks.
• Filtering: Removed URLs, logs, and code blocks to clean text.

B. The RAG Pipeline

1. Retriever Module:

   • Input: A new task (Title + Description).
   • Embedding Models: Two models were tested to generate vector embeddings:
     • BAAI: bge-large-en-v1.5
     • SBERT: all-mpnet-base-v2
   • Mechanism: Calculates cosine similarity between the new task and the training set to retrieve the top-$k$ most similar historical tasks.
   • Hyperparameters: Tested $top\_k$ values (2, 3, 4) and Temperature settings (0, 0.1, 0.2, 0.3).

2. Generator Module:

   • Model: Llama-3.2-3B-Instruct (chosen for balance between performance and computational cost).
   • Prompt Engineering: The generator receives the new task and the retrieved similar tasks (with their known SPs).
   • Instruction: The model is explicitly instructed to analyze complexity relative to the retrieved examples and output a single Fibonacci number (0, 1, 2, 3, 5, 8, etc.) as the estimate.

C. Evaluation Metrics

• Primary Metric: Mean Absolute Error (MAE).
• Secondary Metric: Median Absolute Error (MdAE).
• Statistical Tests:
  • Kruskal-Wallis Test: To check for significant differences across project sizes (Small, Mid, Large).
  • Wilcoxon Signed-Rank Test: To compare RAG performance against baselines and between embedding models.

D. Baselines

The RAG approach was compared against four established methods:

1. TF-IDF: Classical machine learning (SVM).
2. Deep-SE: Deep learning (LSTM + RHWN).
3. LHC-SE: Clustering-based (LDA + Hierarchical Clustering).
4. LHCtc-SE: Clustering-based with task type/component features.

3. Key Contributions

• First RAG Application in SP Estimation: This is the first study to explore RAG for Agile effort estimation, explicitly retrieving both titles and descriptions to mimic the human "Planning Poker" review process.
• Interpretability: Unlike "black box" deep learning models, the RAG approach provides transparent reasoning by showing developers the specific historical tasks the model used to derive its estimate.
• Empirical Analysis of Parameters: Systematic evaluation of how retrieval parameters ($top\_k$, temperature) and project size affect RAG performance.
• Comprehensive Benchmarking: Rigorous statistical comparison against state-of-the-art baselines across 23 diverse projects.

4. Results

The study yielded several nuanced findings:

• RQ1 (Parameters): Optimal parameters ($top\_k$ and temperature) vary by project size and embedding model.

  • Small/Mid projects: Often performed best with $top\_k=2$ and low temperature.
  • Large projects: Benefited from higher $top\_k$ (4) to handle noise, sometimes requiring higher temperature for flexibility.
  • Conclusion: No single "universal" configuration exists; settings must be tuned per context.

• RQ2 (Project Size):

  • Result: No statistically significant difference in performance (MAE) across Small, Mid, and Large projects ($p > 0.05$).
  • Observation: While mid-sized projects showed slightly lower mean MAE, the variance was too high to claim size is a determining factor for RAG effectiveness.

• RQ3 (Embedding Models):

  • Result: No statistically significant difference between BAAI and SBERT ($p = 0.16$).
  • Conclusion: Both models provide comparable retrieval quality for this specific task.

• RQ4 (Comparison to Baselines):

  • Result: While RAG models outperformed specific baselines on individual projects (e.g., RAG-SBERT beat LHC-SE on 7 small projects), the Wilcoxon signed-rank tests showed no statistically significant difference between RAG and the baselines overall ($p > 0.05$).
  • Conclusion: RAG is competitive but does not consistently surpass existing state-of-the-art methods in terms of raw numeric accuracy.

5. Significance and Conclusion

• Primary Finding: RAG is a viable and competitive alternative for automated story point estimation, offering interpretability that traditional ML/DL models lack. However, it does not currently offer a statistically significant accuracy advantage over established baselines like Deep-SE or TF-IDF.
• Practical Value: The system serves best as a Decision Support Tool. By retrieving similar historical tasks, it aids developers in reasoning during planning meetings, potentially reducing bias and discussion time, even if the final numeric prediction isn't always more accurate than a human consensus.
• Limitations & Future Work:
  • The dataset relies on human-labeled SPs, which may contain inherent biases.
  • Future work should focus on cleaning task descriptions more rigorously, fine-tuning generators on project-specific data, and testing in industrial (commercial) environments to validate robustness.

Final Verdict: RAG is a "better way" in terms of transparency and workflow integration, but not yet a "better way" in terms of statistically superior prediction accuracy compared to existing specialized models.