Technical Summary: Exploring Collaboration and Communication Challenges in ML Engineering Teams: Insights From the Semiconductor Industry

Problem Statement

The integration of Machine Learning (ML) into complex software systems introduces significant challenges in collaboration and communication (CoCo) among engineering teams. While prior research has extensively documented these challenges in software-centric environments (e.g., Nahar et al., 2022; Busquim et al., 2024b), there is a critical empirical gap regarding hardware-centric contexts, specifically the semiconductor industry.

In semiconductor manufacturing, ML-enabled systems are not standalone digital products but are tightly coupled with physical manufacturing processes, sensor systems, and strict quality constraints. This environment imposes unique CoCo challenges:

• Strict Data Governance: Proprietary data, privacy regulations, and sensor limitations restrict data accessibility and cloud usage.
• Long Development Cycles: Production timelines are rigid, and errors can lead to costly, irreversible manufacturing failures.
• Interdisciplinary Complexity: Teams must coordinate diverse roles including data scientists, software engineers, optical physicists, process engineers, and hardware specialists, each with distinct vocabularies and priorities.
• Matrix Organization: The studied company operates without a centralized ML Engineering (MLE) team; instead, roles are dynamically assigned across departments, leading to blurred boundaries and fragmented accountability.

The core problem is that existing CoCo taxonomies, derived from software-only contexts, fail to fully capture how these interdisciplinary dynamics unfold under the constraints of high-stakes, precision-driven hardware production.

Methodology

The authors conducted a qualitative study involving semi-structured interviews with practitioners at a leading global semiconductor company.

• Participants: 12 professionals were interviewed across 10 sessions (some participants were interviewed in pairs). The cohort included data scientists, software/system developers, ML engineers, process engineers, and physicists. Their experience ranged from 2 to 25 years in software engineering and 1 to 13 years in ML.
• Procedure:
  1. Preparation: A semi-structured interview protocol was developed and pilot-tested. Ethical approval was obtained from the company's workers' council.
  2. Execution: Interviews (30–45 minutes) were conducted remotely, audio-recorded, and transcribed using automated tools (aTrain with Whisper), followed by manual review and participant validation.
  3. Analysis: The authors employed an iterative open-coding process. Two authors independently coded transcripts, followed by joint calibration sessions to merge codes, resolve disagreements, and refine definitions. This resulted in a stable codebook used to identify recurring themes.
• Research Questions:
  • RQ1: What are the usual CoCo practices and tools used?
  • RQ2: What are the specific CoCo challenges faced?
  • RQ3: How do practitioners mitigate these challenges?

Key Contributions

1. Contextual Analysis: One of the first in-depth qualitative analyses of CoCo in ML engineering within a global semiconductor company, highlighting how hardware constraints and manufacturing processes shape system development.
2. Identification of 16 Challenges: A taxonomy of CoCo challenges, distinguishing between widely known issues in ML systems and those specifically shaped or amplified by the semiconductor domain and matrix organizational structure.
3. Documentation of Mitigation Strategies: Reporting 19 concrete practices and coordination mechanisms used by practitioners to address these challenges, revealing both effective bottom-up strategies and gaps requiring organizational support.

Results

1. Roles and Responsibilities

The study found significant role ambiguity. In the absence of specialized roles (e.g., dedicated data engineers), data scientists often function as generalists, managing end-to-end pipelines from data extraction to operations. Roles such as "Algorithm Developer" and "Physicist" overlap in model training, while "Process Engineers" and "Development Engineers" jointly shape requirements. This blurring of boundaries is exacerbated by the matrix structure, where individuals are dynamically assigned to projects.

2. CoCo Practices and Tools

• Tools: Collaboration relies heavily on Microsoft Teams, Outlook, Azure DevOps (for version control and CI/CD), and SharePoint/Samba for data sharing.
• Documentation: Practices are heterogeneous, ranging from PowerPoint presentations and OpenAPI specs to README files and Architecture Decision Records (ADRs). However, documentation is often insufficient, lacking rationale for design choices or data semantics.
• Structure: Communication flows through product owners and project managers in a matrix structure, with "Communities of Practice" providing technical guidance.

3. Identified Challenges (16 Total)

The study identified 16 recurring challenges, with Unclear roles and duties (C1) being the most cited (9 mentions).

• Amplified Known Issues:
  • Unclear roles/duties (C1): Amplified by the matrix structure, making it difficult to locate responsible parties across departments.
  • Early-stage fragmented communication (C2): Misunderstandings about LLM capabilities or integration requirements are costlier due to long production cycles.
  • ML knowledge deficiency (C3): Stakeholders often lack awareness of ML complexities, leading to unrealistic expectations about time and resources.
  • Rework (C5): Significant time is spent re-explaining basic ML concepts to different stakeholders repeatedly.
• Context-Specific Challenges:
  • Data governance complexity (C10): Strict regulations on production/metrology data limit cloud usage and standard ML practices.
  • Unclear ongoing tasks (C11): Lack of visibility into what other distributed teams are doing due to the matrix structure.
  • Employee tenure concerns (C12): Uncertainty regarding external partners or temporary roles hinders long-term coordination.
  • Missing technical leader (C16): The absence of a central technical authority to align decisions across disciplines and organizational boundaries.

4. Mitigation Approaches (19 Total)

Practitioners employ various strategies to cope with these challenges:

• Meetings (Scrum/Sprint): Used to align on processes, data, and model performance (11 mentions).
• Early clarification through direct communication: Resolving expectation gaps before integration begins.
• Software and ML literacy: Cross-training to help non-ML staff understand limitations and ML staff understand software constraints.
• Clear roles and duties: Defining responsibilities to reduce overlap.
• Technical manager/mentor: Individuals with hands-on ML experience to bridge management and technical gaps.
• Thorough and verified documentation: Iterative documentation processes involving multiple departments.
• Mediator/Middle communication layer: Centralizing communication to prevent fragmentation.
• Well-defined blueprints: Reusable templates for use cases to improve reproducibility.

Significance and Claims

The paper claims that CoCo should be treated as a first-class engineering concern rather than a "soft" issue. In the semiconductor industry, the consequences of CoCo breakdowns are not merely delays in delivery but can directly impact production reliability, introduce operational risks, and cause irreversible manufacturing failures.

The study argues that while many CoCo challenges (e.g., role ambiguity, communication gaps) are fundamental to the socio-technical nature of ML systems and appear in software-centric contexts, they are recontextualized and amplified by hardware-driven constraints. Specifically:

• Data Governance: Strict policies limit the applicability of standard ML tools.
• Physical Coupling: Misalignment between software and physical processes makes errors more costly to correct.
• Organizational Structure: The matrix organization and lack of centralized MLE teams intensify visibility and coordination issues.

The authors conclude that while bottom-up strategies (e.g., communities of practice, direct communication) are currently effective, there is a need for systemic organizational interventions (e.g., defined responsibility matrices, technical leadership roles) to sustainably address these challenges. The findings offer a grounded understanding for practitioners in similar high-tech manufacturing sectors and provide a basis for future research into socio-technical dynamics in hardware-constrained ML environments.