Machine Learning in the Wild: Early Evidence of Non-Compliant ML-Automation in Open-Source Software

1. Problem Statement

The increasing availability of Machine Learning (ML) models, particularly foundation models, has led to their integration into diverse software applications. While beneficial in low-risk domains, their deployment in mission-critical and safety-critical systems (e.g., healthcare, autonomous driving, finance) raises significant concerns regarding:

• Regulatory Compliance: Many applications fall under strict regulations like the EU AI Act, which classifies certain use cases (biometrics, critical infrastructure, law enforcement, etc.) as "High-Risk" and mandates transparency and human oversight.
• Terms of Use (ToU) Violations: Proprietary foundation models often have strict ToUs prohibiting their use for high-stakes automated decisions or specific professional domains (e.g., medical advice) without human review.
• Lack of Oversight: There is a lack of empirical evidence regarding whether open-source software (OSS) developers are actually implementing necessary safeguards (like human-in-the-loop mechanisms) or adhering to these regulations and ToUs when building automated decision-making systems.

2. Methodology

The authors conducted a preliminary empirical study involving 173 open-source Python projects hosted on GitHub.

• Data Collection & Selection:
  • Domains: The study focused on 16 application domains identified as "High-Risk" based on the EU AI Act (e.g., Healthcare, Autonomous Driving, Finance, Cybersecurity).
  • Filtering: Researchers searched GitHub topics related to these domains, filtered for Python projects, and intersected the list with projects depending on the huggingface/transformers library to ensure ML usage.
  • Final Sample: 173 projects were selected for manual analysis.
• Analysis Framework: The study addressed three Research Questions (RQs) through manual code and documentation review:
  • RQ1 (Decision Mechanism): Determining if projects use ML for fully automated decisions or require human-in-the-loop (HITL) validation. This involved analyzing READMEs, source code imports, and control flow to trace how model outputs influence system behavior.
  • RQ2 (Post-Processing): Identifying safeguards applied to raw model outputs. The team looked for patterns like confidence thresholds, schema enforcement, rule-based filtering, and multi-agent coordination.
  • RQ3 (Compliance): Evaluating adherence to model ToUs and regulations. Researchers retrieved ToUs from model owners (e.g., Hugging Face, GitHub) and compared them against the project's application domain and decision-making logic.

3. Key Contributions

• Empirical Evidence of Non-Compliance: The paper provides the first systematic evidence of how open-source projects in high-risk domains utilize ML models, specifically highlighting the gap between current practices and regulatory/ToU requirements.
• Taxonomy of Automated Decisions: The authors classified ML-driven decisions into nine distinct categories (e.g., AI Recommender, Automated Control, Threat Detection), moving beyond simple classification to understand the impact of the decision.
• Identification of Post-Processing Patterns: The study cataloged common strategies developers use to mitigate ML risks, such as output structuring and confidence thresholds, revealing that while safeguards exist, they are often insufficient for full regulatory compliance.
• ToU Violation Analysis: The study quantifies the extent of ToU violations, specifically noting that proprietary Large Language Models (LLMs) are frequently used in prohibited high-risk contexts without the required human oversight.

4. Key Results

RQ1: Usage in Decision Making

• Automation Level: Out of 173 projects, 45 (26%) employ ML for automated decision-making.
  • Fully Automated: 36 projects (e.g., DriveVLM directly controlling vehicle throttle/brake).
  • Partially Automated (HITL): 9 projects (e.g., Medical-Assistant-Multi-Agents where a human validates diagnoses).
• Decision Categories: The most common category was AI Recommender (20 projects), followed by Automated Control (7) and Threat Detection (4). Notably, "AI Recommenders" in high-risk domains (like finance) can lead to severe consequences despite not being strictly "automated" in the code.

RQ2: Post-Processing Safeguards

• Prevalence: 40 out of 45 projects apply at least one post-processing step before acting on model output.
• Common Techniques:
  • Output Structuring (13): Using regex to fix format inconsistencies.
  • Output Consistency (11): Applying domain-specific rules (e.g., ensuring acceleration limits in autonomous driving or preventing incompatible drug prescriptions).
  • Auto-Refiner (9): Feedback loops that re-evaluate and re-run models.
  • Confidence Thresholds (7): Blocking actions if the model's confidence score is below a specific level (e.g., >80% for phishing detection).
• Gap: Despite these measures, many projects still rely on raw outputs for critical actions without explicit human intervention, failing to meet "High-Risk" regulatory standards.

RQ3: Compliance with ToU and Regulations

• High Violation Rate: 25 out of 45 projects (approx. 56%) violate the Terms of Use of the ML models they employ.
• Nature of Violations:
  • Violations primarily involve proprietary LLMs (GPT-4o, Gemini, LLaMA, etc.).
  • Common violations include using models for automated high-stakes decisions in sensitive areas (healthcare, finance) without human review, which is explicitly prohibited by the models' ToUs.
  • GPT-4o and Gemini were the most frequently cited models in violation cases.
• Regulatory Context: Most analyzed projects fall under the EU AI Act's "High-Risk" definition, which legally requires human oversight. The study found that while some projects implement HITL, many do not, creating a legal compliance gap.

5. Significance and Future Work

• Significance: This study highlights a critical "wild west" scenario where open-source developers are deploying ML in high-risk environments without adhering to legal frameworks or model licenses. It underscores the urgent need for better tooling and guidelines to ensure software safety and regulatory compliance.
• Future Directions: The authors propose developing automated tools to detect regulatory violations and ToU breaches in software systems. They aim to create a framework to assist developers in evolving their systems to be legally compliant and safer.
• Threats to Validity: The authors acknowledge that their sample is limited to Python projects using transformers and that manual analysis introduces potential human error, though they mitigated this with dual-author validation.

Conclusion: The paper serves as a wake-up call for the software engineering community, demonstrating that while technical safeguards (post-processing) are being implemented, legal and ethical compliance (ToU and Human-in-the-loop requirements) is frequently neglected in open-source ML automation.