Technical Summary: "So There's a Catch-22 Here": How Early Adopters Who Build Multi-Agent LLM Systems Conceptualize Transparency

Problem Statement

Multi-agent Large Language Model (LLM) systems are rapidly emerging as complex architectures where multiple agents collaborate to orchestrate tasks. While transparency is a cornerstone of Responsible AI (RAI), existing frameworks—such as Explainable AI (XAI), model cards, and interpretability metrics—were largely developed for non-agentic, single-model systems. These traditional approaches fail to address the unique opacity introduced by distributed architectures, including inter-agent communication, task delegation, and emergent behaviors. Consequently, there is a critical gap in understanding how transparency should be defined, achieved, and sustained in multi-agent contexts. This complexity creates uncertainty for developers debugging these systems, users relying on their outputs, and institutions seeking to regulate them. The central research question addresses how early adopters of these systems perceive transparency and what implications these perceptions hold for designing future transparent systems.

Methodology

The study employs an interpretivist approach to investigate the transparency practices of early adopters.

• Participants: The researchers conducted semi-structured interviews with 13 early adopters (8 technical roles, e.g., developers and data scientists; 5 non-technical roles, e.g., product managers and designers) from a single large technology organization. All participants had first-hand experience developing or experimenting with multi-agent AI frameworks (e.g., AutoGen, TaskWeaver, custom systems) in professional contexts.
• Data Collection: Interviews combined retrospective tasks and reflective activities. Participants were asked to describe their experiences developing multi-agent systems, reflect on everyday usage, and articulate their interpretations of transparency. While transparency was explicitly queried, it also emerged unprompted as a foundational concern throughout discussions on motivations, limitations, and workflows.
• Analysis: Researchers recorded and transcribed nine hours of interviews. Using iterative thematic analysis, they generated 89 codes to capture interpretations of transparency, capabilities, limitations, and design implications. Three researchers engaged in a seven-week iterative coding process to refine themes, ensuring analytical rigor and identifying patterns across participant accounts.

Key Contributions

The paper makes four primary contributions:

1. Empirical Insight: It provides one of the first empirical studies on how early adopters understand and practice transparency in multi-agent LLM-based systems.
2. Framing Identification: It identifies divergent yet complementary framings of transparency, including reproducibility, boundary-setting, debugging, visualization, trust-building, compliance, and auditing.
3. Theoretical Synthesis: It synthesizes these perspectives into a multidimensional framework that positions transparency as a "situated socio-technical practice," highlighting tensions and synergies across developer, user, and governance needs.
4. Design Implications: It discusses implications for Human-Computer Interaction (HCI) and AI design, emphasizing that transparency must be tailored to specific stakeholder groups and contexts rather than treated as a singular technical feature.

Key Findings

The analysis reveals that transparency is not a singular concept but a layered construct shaped by role, context, and system maturity. The findings are categorized by what transparency entails, how it is achieved, and why it matters.

1. The "What": Divergent Definitions

• Capability Boundaries: For non-technical users, transparency involves clarifying what a system can and cannot do to prevent over-reliance and form realistic mental models.
• Internal Visibility: For developers, transparency is a diagnostic tool requiring deep visibility into agent reasoning, context, and communication to isolate bugs and refine orchestration.
• Observability: Transparency functions as a mechanism for both high-level summaries (for trust) and granular traces (for debugging).

2. The "How": Mechanisms and Mediation

• Visualization: Participants advocated for UI metaphors (e.g., group chats, directed graphs, process flows) to make agent-to-agent interactions comprehensible.
• Evaluation: Transparency is achieved through "lenses" such as bias detection tools, fact-checkers, and telemetry metrics that allow users to critically evaluate outputs.
• Traceability: A distinction was made between proactive transparency (built-in logs) and reactive transparency (audit logs triggered by unexpected behaviors or errors).

3. The "Why": Purposes and Tensions

• Reproducibility: Open-source models, code, and reproducibility packages are viewed as essential for community accountability and independent verification.
• Auditability: In compliance-heavy domains, transparency is required to trace data usage, verify licensing, and explain decision chains to satisfy regulatory standards.
• Trust-Building: Transparency serves as a pragmatic necessity to prove system functionality and reliability, often requiring "proof" of performance rather than just technical exposure.
• The "Catch-22": A significant tension emerged where transparency is crucial for long-term trust and governance, yet often deprioritized during early adoption because users first need to be convinced the system "actually works." This creates a cycle where transparency is often reactive (damage control) rather than proactive.

Proposed Framework: Multi-Dimensional Transparency

The authors propose a framework situating transparency across three interrelated dimensions:

1. Developer-Focused Transparency: Prioritizes depth, observability, debugging, and technical reproducibility. It is designed for skilled actors who need full system exposure to ensure reliability.
2. User-Focused Transparency: Prioritizes usability, trust, and boundary communication. It abstracts complexity into interpretable forms (e.g., high-level summaries, visualizations) to foster understanding without overwhelming non-technical users.
3. Governance-Focused Transparency: Prioritizes accountability, ethics, and compliance. It involves documentation (model cards, data sheets) and traceability to satisfy institutional and regulatory requirements.

The framework also distinguishes between Proactive Transparency (built-in mechanisms like open-source practices) and Reactive Transparency (triggered by failures). The authors argue that effective design requires integrating both, balancing the resource intensity of proactive measures with the pragmatic needs of reactive investigation.

Significance and Limitations

Significance:
The paper argues that transparency in multi-agent systems must be understood as a situated socio-technical practice rather than a static technical feature. It extends XAI research by shifting the focus from individual model interpretability to the comprehensibility of collective, orchestration-level systems. By centering on early adopters, the study reveals practical challenges and emergent expectations that will shape the future trajectory of multi-agent AI design. It highlights the need for HCI and AI research to move beyond single definitions of transparency toward a pluralistic approach that negotiates the needs of builders, users, and regulators.

Limitations:
The authors acknowledge several limitations:

• Sample Scope: The study involved a small sample (n=13) from a single large technology organization, which may not capture the diversity of perceptions found in smaller companies, academia, or independent communities.
• Perceptual Nature: As an interpretivist study, the findings reflect participant perceptions rather than objective measures of system transparency.
• Temporal Context: Given the nascent stage of multi-agent LLM systems, participant accounts may evolve as the technology matures, scales, and faces increased regulatory oversight.

The paper concludes that future work should explore longitudinal shifts in transparency needs across the system lifecycle and investigate high-stakes sectors (healthcare, finance, law) where opacity carries heightened risks.