Data Requirement Goal Modeling for Machine Learning Systems

Technical Summary: DRGM: Data Requirement Goal Modeling for ML-Based Systems

Problem Statement
The integration of Machine Learning (ML) into critical applications has shifted the software development paradigm from logic-driven to data-driven. Unlike conventional systems, ML-based system behavior is primarily determined by data properties and the models constructed from them rather than predefined functionalities. Consequently, data requirements—encompassing properties, constraints, and quality attributes of training data—exert a profound influence on model performance, interpretability, and fairness.

Despite the existence of Requirements Engineering (RE) methods for functional and non-functional requirements, traditional approaches are insufficient for the inductive nature of ML. Current literature highlights a gap in systematically capturing, evaluating, and analyzing alternative datasets specifically for data requirements. Issues such as data bias, lack of representativeness, and data quality flaws have led to significant failures in real-world deployments, including clinical diagnosis models and hiring algorithms. There is a need for a structured approach to guide non-experts in identifying data requirements and evaluating datasets before utilization to ensure robustness, reliability, and fairness.

Methodology
The authors propose the Data Requirement Goal Model (DRGM), a framework utilizing Goal-oriented Requirement Language (GRL) to represent objectives, constraints, and trade-offs. The methodology consists of three primary phases:

1. Construction of the Initial DRGM:
   The authors conducted a literature review to identify challenges faced by data scientists and requirement engineers. These challenges were categorized into four primary themes: Data Quantity, Data Quality, Data Management, and Data Ethics. These categories were mapped to GRL elements, specifically using "soft goals" to capture essential data properties (e.g., Data Availability, Data Accuracy, Data Privacy). The initial model includes decomposition links and contribution links to illustrate relationships between goals and tasks.

2. Development of a Customization Mechanism:
   Recognizing that data requirements are highly context-dependent and vary by ML problem type, the authors designed a customization mechanism. This mechanism categorizes goals into three sets:

   • Set #1 (Fixed Importance): Goals critical across all ML problems, such as Data Quantity (Availability, Accessibility) and Data Quality (Completeness, Accuracy, Representativeness).
   • Set #2 (ML Problem Type Dependent): Goals whose importance and Key Performance Indicators (KPIs) vary based on the problem type (e.g., Classification, Regression, Time Series). For instance, Data Balancedness is critical for classification but less so for regression. The mechanism provides specific KPI calculation rules (e.g., the "rule of 10" for data size, Shapiro-Wilks test for skewness).
   • Set #3 (Context Dependent): Goals varying based on the specific data context, such as Data Ethics (Legality, Privacy, Safety) and Data Management (Security, Logging, Freshness). Importance levels are adjusted based on factors like whether the data involves human subjects, sensitive information, or requires frequent updates.

   The customization process is guided by two UML activity diagrams: one for ML Problem Type and one for Context.

3. Dataset Evaluation:
   Once the DRGM is customized, users evaluate candidate datasets using GRL evaluation strategies. The process involves assigning qualitative values (0–100) to leaf elements. Unrelated elements are set to "satisfied" to avoid negative impact. A dataset is considered viable if it satisfies the "Data Actor" with a score above a proposed threshold of 70. If no dataset meets the threshold, additional preprocessing or alternative data sources must be sought.

Key Contributions

1. Goal-Oriented Modeling for Data: The paper proposes the use of goal modeling to capture key data requirements for ML system development, linking data properties to higher-level system goals.
2. Customization Mechanism: The authors developed a mechanism to adapt the DRGM to various ML problem types and contexts, supported by UML activity diagrams to assist requirements engineers, particularly those with limited ML expertise.
3. Empirical Validation: The approach was evaluated through two illustrative examples based on real-world projects: a medical classification system for anemia diagnosis and a time-series regression system for Global Horizontal Irradiance (GHI) forecasting.

Results
The validation through the two case studies demonstrated that the data requirements identified by the DRGM align with the actual outcomes and challenges reported in the source studies:

• Anemia Diagnosis Case: The customized DRGM correctly identified that the initial dataset (80 participants) suffered from significant imbalance (class ratios of 3:5:12:40) and lack of representativeness regarding skin tone and gender. The evaluation model highlighted these deficiencies, explaining why the project faced issues with recall and accuracy for specific demographics and was ultimately halted.
• GHI Forecasting Case: The model correctly identified that while the data source was authoritative and sustainable, the dataset lacked temporal representativeness for a specific city (Najran) due to missing months and atypical precipitation values in the training period. The evaluation showed that the dataset failed to satisfy the Data Actor when applied to this new context, mirroring the real-world experimental failure.

Significance and Claims
The paper claims that the proposed DRGM framework is practical and effective for guiding non-experts in data planning. By systematically identifying data requirements and providing a mechanism to evaluate alternative datasets, the approach helps ensure that datasets meet critical requirements before deployment. The authors emphasize that the framework aids in the elicitation, specification, and verification of data needed for ML systems.

However, the authors maintain a modest stance regarding the scope and limitations of their work:

• The customization mechanism is currently designed for common supervised ML problems (regression and classification) and does not fully address pre-trained deep learning models or Large Language Models (LLMs).
• Some KPI values are derived from gray literature and practitioner heuristics (e.g., the "rule of 10"), which may introduce subjectivity compared to peer-reviewed sources.
• The process still requires manual modification of the goal model, which could lead to errors for users unfamiliar with GRL.
• The model does not eliminate the need for consultation with legal and domain experts, particularly regarding data legality and specific representativeness issues, though it aims to streamline the selection process and filter out inapplicable datasets.

The authors conclude that while the DRGM provides a structured approach to data requirements, further evaluation across a wider range of ML problem types and the development of automated tool support (e.g., via a chatbot interface) are necessary future directions.