Applying a Requirements-Focused Agile Management Approach for Machine Learning-Enabled Systems

Technical Summary: Applying a Requirements-Focused Agile Management Approach for Machine Learning-Enabled Systems

Problem Statement
Machine Learning (ML)-enabled systems present unique challenges to traditional Requirements Engineering (RE) and Agile management. Unlike deterministic software, ML systems rely on data dependence, experimentation, and uncertain model behavior, making results dependent on data quality and modeling choices. While Agile management is conceptually aligned with the iterative nature of ML, its direct application often fails due to:

• Unrealistic customer expectations and misalignment between business stakeholders and ML practitioners.
• Difficulties in sprint planning and effort estimation due to the experimental nature of ML tasks (e.g., data preprocessing, feature engineering).
• A lack of well-defined guidelines for tailoring agile practices to ML projects, leading to communication gaps and ambiguity in requirements.
  Existing approaches often fail to fully address how ML-specific concerns should be identified and operationalized into actionable requirements from the project's inception.

Methodology: The RefineML Approach
The authors propose and evaluate RefineML, a requirements-focused approach for the continuous and agile refinement of ML-enabled systems. RefineML integrates three core components derived from state-of-the-art literature and a systematic mapping study:

1. PerSpecML: A requirements specification approach that structures ML concerns into five perspectives: System Objectives, User Experience, Infrastructure, Model, and Data.
2. Agile4MLS: An agile management framework that adapts Scrum for ML, introducing dual backlogs (Software and ML), "Data Stories" and "Model Stories," and a "two-sprints-ahead" principle to decouple ML experimentation from software development.
3. Lean R&D Concepts: Specifically, the Minimal Valuable Model (MVM) (a viable, non-optimal model for early value delivery) and Layers of Done (LoD) (milestones defining model robustness and validation).

RefineML Process Phases
The approach structures the project lifecycle into four phases:

• Initial Specification: Using PerSpecML, the team defines requirements across the five perspectives to establish a common understanding of goals and evaluation criteria.
• Conception: The Product Owner structures two complementary backlogs: a Software Backlog (conventional User Stories) and an ML Backlog (Data and Model Stories). "Enablers" are defined to manage infrastructure and architectural dependencies.
• Technical Feasibility: The team conducts Data Feasibility (assessing data availability/quality), Model Feasibility (evaluating algorithm suitability), and implements a Demo API. The Demo API emulates the expected ML output, allowing the software team to proceed with integration while the ML team continues experimentation.
• Agile Development: Following Agile4MLS, the team executes synchronized iterations. The ML team works "two sprints ahead," evolving the Demo API into an MVM and subsequent LoDs, while the software team implements features based on the emulated interface.

Evaluation Context and Data Collection
RefineML was applied in a six-month Industry–Academia Collaboration (IAC) between PUC-Rio and EXA, a Brazilian cybersecurity company. The project aimed to develop ML features for online fraud protection, including scam message detection, URL analysis, and screenshot analysis.

• Participants: A team of ten practitioners (developers and data scientists), a Project Lead, and Business Owners.
• Data Collection: A questionnaire based on the Technology Acceptance Model (TAM) was administered to the development team to assess Perceived Usefulness, Ease of Use, and Intention to Use. Semi-structured interviews were conducted with the Project Lead and Business Owner.
• Analysis: Quantitative data was analyzed using descriptive statistics, while qualitative data underwent thematic analysis.

Key Results and Findings

• Problem-Solving Capability: RefineML successfully delivered a production-ready suite of ML components within six sprints. The team delivered a working MVM early in the cycle that outperformed the existing fraud prevention solution. The approach facilitated the orchestration of dual-track work (ML experimentation vs. product delivery), reducing blocking dependencies.
• Suitability of Phases:
  • Initial Specification: Perceived as effective for aligning expectations, though some found it "long" without strong facilitation.
  • Conception & Feasibility: The dual-backlog structure and the "Enablers" concept were viewed as crucial for separating experimentation from integration. Data and Model Feasibility were deemed "must-have" steps to avoid analysis paralysis.
  • Development: The "two-sprints-ahead" principle and MVM approach were highly valued for providing "peace of mind" to data scientists and preventing the "search for 99% accuracy" trap.
• Overall Acceptance: Stakeholders reported high perceived usefulness and an intention to continue using RefineML. The Business Owner noted the approach "makes sense" and expressed intent to adopt it for future initiatives.
• Limitations:
  • Operationalization: Translating PerSpecML concerns into actionable ML backlog items (Data/Model Stories) proved difficult for less experienced practitioners and relied heavily on experienced facilitation.
  • Estimation: The stochastic nature of ML made effort estimation a persistent, unsolved challenge. Practitioners noted that predicting outcomes (e.g., accuracy improvements) remains unreliable.
  • Complexity: Junior developers found the approach complex without experienced guidance, whereas senior stakeholders found it "genuinely simple."

Significance and Claims
The paper claims that RefineML offers a practical, requirements-focused framework that bridges the gap between agile management and the experimental nature of ML. Its primary significance lies in:

1. Improved Communication: By making ML requirements, constraints, and progress explicit through artifacts like LoDs and dual backlogs, the approach reduces the perception of ML as a "black box" and improves alignment between business and technical teams.
2. Feasibility Scaffolding: The early feasibility phases (Data/Model) provide a structural framework to ground projects in technical reality, surfacing risks early and preventing late-stage failures.
3. Co-evolution Governance: The dual-track structure allows for the continuous refinement of ML models without delaying software delivery, enabling incremental value delivery through the MVM concept.

The authors modestly conclude that while RefineML effectively manages communication and governance, it does not resolve the intrinsic uncertainty of ML effort estimation. They suggest that future improvements should focus on lowering entry barriers for less experienced practitioners and developing better mechanisms for ML-aware effort estimation.