Trust in LLM-Supported Engineering Work: The Role of Downstream Dependency

Technical Summary: Trust in LLM-Supported Engineering Work: The Role of Downstream Dependency

Problem Statement
While Large Language Models (LLMs) are increasingly integrated into professional engineering workflows for tasks such as drafting, coding, and configuration, existing research largely focuses on isolated interactions (e.g., code completion or discrete decision support). There is a significant gap in understanding how engineers judge LLM-generated outputs when those outputs become embedded in connected, multi-stage workflows. In such settings, the immediate plausibility of an output is insufficient; engineers must determine if an output can be interpreted, verified, controlled, and safely integrated into subsequent stages (e.g., dataset preparation, model training, or deployment). The core problem is that trust in these contexts is not merely a judgment of the LLM's quality, but a situated judgment regarding the relationship between the output, the engineer's expertise, and the downstream consequences of carrying that output forward.

Methodology
This study employs a qualitative single-case design within a Small and Medium-sized Enterprise (SME) specializing in robotics synthetic data pipelines. The research focuses on four engineers with direct experience in the pipeline and its LLM-supported features. Data collection involved three complementary sources:

1. Semi-structured interviews: Conducted to explore engineers' reflective accounts of trust, verification practices, and risk.
2. Task-based think-aloud sessions: Participants worked through a fifteen-task representation of the synthetic data pipeline, verbalizing their reasoning while interacting with the system.
3. Screen-recorded behavioral coding: Interactions were recorded and coded to identify observable verification, correction, and decision-making activities.

The analysis utilized reflexive thematic analysis (Braun & Clarke) on verbal data and behavioral coding on screen recordings. The study was interpreted through a Joint Cognitive Systems perspective, treating the engineer, interface, and workflow as an interdependent unit responsible for maintaining control over the process.

Key Contributions
The paper makes three primary contributions:

1. Empirical Description: It documents how engineers in a multi-stage synthetic data pipeline treat LLM outputs not as final results, but as "useful starting points" that require inspection and modification before integration.
2. Conceptual Development: It introduces Downstream Dependency as a central concept. This describes the reliance-relevant relationship between a present LLM-generated output and the later work that consumes it. It is defined by the output's propagation reach and the engineer's capacity to detect, contain, and reverse resulting problems.
3. Identification of Practices: It identifies two associated practices engineers use to manage downstream dependency:
   • Stage-dependent verification: The form of checking required varies by workflow stage (e.g., itemized value checks for parameters vs. structural explanations for modules).
   • Scoped control: The need for engineers to understand and limit the scope of system changes, ensuring interventions are visible, bounded, and reversible.

Results
The findings indicate that trust is conditional and shifts based on task risk, verifiability, control, and anticipated downstream consequences.

• Downstream Dependency: Engineers judged outputs based on whether errors could be caught locally or if they would propagate to critical downstream stages (e.g., training datasets or annotations). Reliance was often withheld or modified (e.g., reprompting or manual correction) when the output risked affecting future stages, even if the immediate output appeared plausible.
• Stage-dependent Verification: Verification needs were not uniform. For parameter tuning, engineers required itemized checklists and value comparisons. For module generation, they required high-level explanations of structural intent and pipeline fit.
• Scoped Control: Engineers expressed a strong preference for systems that allowed them to see intermediate steps and intervene before large system actions were finalized. Unexpected changes to parts of the configuration outside the immediate prompt (e.g., scene-level settings changing due to an asset-level request) significantly reduced trust and prompted additional inspection.
• Behavioral Evidence: The study observed that verification is a continuous activity running through the process, not just a final check. Engineers frequently returned to earlier configuration steps to trace problems, demonstrating that trust is maintained through the ability to observe and correct the state of the joint system.

Significance and Claims
The paper argues that trust in LLM-supported engineering work is not located solely in the LLM or the individual engineer but is produced through the interaction of system behavior, professional judgment, verification practices, and workflow structure.

• Theoretical Shift: The study shifts the focus from isolated output quality to workflow-level judgment. It posits that "appropriate reliance" in engineering is determined by the engineer's ability to anticipate downstream consequences and maintain control over the propagation of errors.
• Design Implications: The authors propose that LLM support for engineering should be designed to be inspectable, staged, bounded, and reversible. Specifically, interfaces should:
  • Make changes visible, bounded, and reversible (supporting scoped control).
  • Provide verification support that corresponds to the function of the workflow stage (e.g., checklists for parameters, structural explanations for modules).
  • Make relevant context and assumptions visible before generation.
  • Support prospective assessment of downstream consequences (e.g., propagation indicators).

Limitations and Scope
The authors explicitly state that these findings are exploratory, derived from a single case study with four participants in one organization. The study does not benchmark the technical accuracy of the LLMs, nor does it claim that the proposed framework is universally necessary or sufficient for all engineering contexts. The findings are most transferable to professional settings where LLM support is embedded in multi-stage workflows and users remain accountable for the outcomes. The study distinguishes between reported trust and behavioral reliance, noting that engineers may rely on systems they do not fully trust due to workflow constraints, or withhold reliance from useful systems when verification is too difficult or downstream risks are too high.