Socio-Technical Anti-Patterns in Building ML-Enabled Software: Insights from Leaders on the Forefront

Technical Summary: Socio-Technical Anti-Patterns in Building ML-Enabled Software

Problem Statement
Despite the economic success and ubiquity of machine learning (ML)-enabled software, a significant gap remains between model development and successful productionization. While existing literature extensively addresses technical hurdles such as testing, pipelines, and automation (MLOps), there is a scarcity of research focusing on the socio-technical challenges inherent in building these systems. Building ML software is inherently a multi-person, multi-team endeavor, yet studies often overlook the organizational, management, and communication barriers that prevent models from reaching production. This paper posits that many failures are not technological but stem from organizational silos, leadership vacuums, and miscommunication, echoing observations that "MLOps is an organizational problem."

Methodology
To investigate these socio-technical challenges, the authors conducted the largest qualitative empirical study in this domain to date. The research employed Reflexive Thematic Analysis (RTA), a method allowing researchers to engage analytically with data rather than merely summarizing it.

• Data Source: The study analyzed 73 videos (totaling 66 hours of content) from the MLOps.community, a global interest group with over 11,000 members. This corpus includes talks, meetups, and coffee sessions featuring practitioners from diverse roles, company sizes, and geographies.
• Sampling Strategy: From an initial pool of 210 videos, the authors applied a two-phase selection process:
  1. Induction Phase: Initial filtering based on the keyword "team" to identify socio-technical discussions, followed by manual familiarization and coding.
  2. Deduction and Refinement Phase: Using extracted codes to identify relevant passages in the remaining videos, refining themes, and extracting recommendations.
• Demographics: The speakers included management (31.3%), leaders (27.6%), and data scientists/engineers (22.7%), representing companies ranging from startups to large enterprises across North America, Europe, and other continents.
• Analysis: The authors iteratively developed themes, eventually structuring findings into anti-patterns (symptoms), causes, and recommendations.

Key Contributions
The paper's primary contribution is the identification and analysis of 17 socio-technical anti-patterns rooted in organizational and management issues rather than purely technical deficiencies. The study contextualizes these findings with prior research (e.g., Nahar et al., Kim et al.) to validate existing results and highlight new insights, particularly from a management perspective.

Key Results: The 17 Anti-Patterns
The findings are categorized into three main organizational areas:

1. Organizational Silos

• Model to Product Integration:
  • AP1 (Long release cycles): Tight coupling between model and product teams leads to bottlenecks.
  • AP2 (Tension between teams): Mutual complaints arise from a lack of understanding between data scientists and software engineers.
  • AP3 (Blocked data scientists): Data scientists are diverted from research to production tasks due to handover friction.
  • Causes: Clash of cultures/tools (C1) and missing engineering competence in data science teams (C2).
• Data Producer to Consumer Integration:
  • AP4 (Requesting data is hard): Data producers resist exposing data due to lack of perceived value or conflicting goals.
  • AP5 (Tight technical coupling): Unclear handover processes lead to code/data dependencies across teams.
  • AP6 (Partial data availability): Training data differs from production data, causing model degradation.
  • Causes: Lack of awareness/common goals (C3) and missing documentation/responsibilities (C4).

2. Communication Deficits

• Redundant Development:
  • AP7 (Features not discoverable): Teams build semantically similar features independently.
  • AP8 (Redundant ML infrastructure): Duplicate pipelines and tools are built by different teams.
  • AP9 (Shadow IT): Teams bypass central infrastructure providers, creating security risks.
  • Cause: Missing intersections, documentation, and trust between teams (C5).
• Management vs. Data Science:
  • AP10 (Burnout): Data scientists feel they produce no tangible value due to endless iterations.
  • AP11 (Tension with management): Conflict arises from misaligned expectations regarding business value and research uncertainty.
  • Causes: Missing data science processes (C6) and communication mismatches between technical and non-technical stakeholders (C7).

3. Leadership Vacuum

• Headless-Chicken-Hiring:
  • AP12 (Staff with insufficient skills): Hiring data scientists for engineering-heavy tasks (e.g., pipelines).
  • AP13 (No product for data scientists): Hiring for a specific use case that disappears, leaving the employee without a role.
  • Causes: Unclear roles/titles (C8), uneducated hiring (C9), and market skill shortages (C10).
• Résumé-Driven Development:
  • AP14 (Tools mismatch goals): Technology choices are driven by individual resume building rather than business needs.
  • Causes: Data scientists disconnected from business value (C11) and missing decision-makers (C12).
• Hype-Driven Product Creation:
  • AP15 (Stuck with PoCs): Organizations remain in "proof-of-concept hell" without moving to production.
  • AP16 (Everything needs ML): The belief that ML is the solution for every problem.
  • AP17 (Infeasible products): Projects launched without necessary data or talent.
  • Causes: Missing organizational strategy (C13), management lack of ML knowledge (C14), missing translation between stakeholders (C15), and missing production metrics (C16).

Recommendations
The paper suggests that while tools (e.g., Model Registries, Feature Stores) can mitigate symptoms, the root causes require organizational change. Recommendations include:

• Restructuring teams to be cross-functional.
• Pairing data scientists with software engineers for code reviews.
• Establishing central platforms as contracts for data quality and documentation.
• Implementing strong development processes and documentation tailored to ML.
• Hiring for specific skills rather than ambiguous titles.
• Creating rigorous processes to validate use cases and data availability before development begins.

Significance and Claims
The authors claim that this study provides a holistic view of ML development challenges by leveraging a unique dataset of practitioner discussions, including insights from leaders and managers often absent in developer-centric studies. By framing these issues as anti-patterns, the paper aims to raise awareness that socio-technical problems are often the root cause of production failures, frequently misdiagnosed as technical debt.

The study validates and extends previous findings (e.g., from Nahar et al., Kim et al., Granlund et al.) by confirming the existence of organizational silos and cultural clashes while adding new dimensions regarding leadership vacuums and hiring practices. The authors modestly note that while their findings are generalizable due to the diverse sample, further research is needed to determine if these anti-patterns are specific to ML or are artifacts of broader socio-technical challenges in software engineering. Ultimately, the paper serves as a guide for practitioners and managers to identify and address non-technical barriers to successful ML adoption.