Ties of Trust: a bowtie model to uncover trustor-trustee relationships in LLMs

1. Problem Statement

The rapid dominance of Large Language Models (LLMs) in high-stakes domains, particularly politics, has exposed critical gaps in existing trust frameworks. Current models suffer from three primary limitations:

• Oversimplification (G1): Existing trust models (e.g., Mayer et al., Ghosh) are often reductionist and one-directional, focusing exclusively on either the trustee (system attributes like transparency) or the trustor (user intentions), failing to capture the dynamic, bidirectional nature of trust in adaptive AI environments.
• Lack of Inter-relationship Analysis (G2): Previous studies rarely explore the complex intra- and inter-relationships between the trustor's contextual factors (e.g., ideology, demographics, past experiences) and the trustee's systemic elements (e.g., scientific discipline, human-in-the-loop features, organizational stakeholders).
• Under-explored Human-Centric Elements (G3): There is insufficient evidence regarding how specific human-centric features within the trustee (such as Human-in-the-Loop (HITL) validation or the involvement of specific experts like data journalists vs. political scientists) influence user trust.

The authors argue that to redefine trust in LLMs, a holistic model is required that treats trust as a sociotechnical ecosystem rather than a linear transaction.

2. Methodology

The authors propose and validate a novel Bowtie Model through a mixed-methods explanatory study.

A. The Bowtie Model Framework

The model conceptualizes trust as a bowtie structure with three core components:

1. Trustor Side (Left): Unfolds from broad categories (Demographics, Ideologies, Background, Perceptions) to specific contextual factors (e.g., education level, familiarity, past experiences).
2. Trustee Side (Right): Unfolds from broad systemic elements (Scientific Discipline, Scientists, Products, Stakeholders) to specific elements (e.g., AI/LLMs, data scientists, political discourse tools, hosting organizations).
3. The Core: The connecting mechanism that reveals:
   • Intra-relationships within the trustor side.
   • Bidirectional inter-relationships between trustor factors and trustee elements.
   • Intra-relationships within the trustee side.

B. Study Design

• Participants: 29 participants recruited via purposive sampling (ensuring demographic and occupational diversity) for an on-site interactive workshop. A subset of 10 participants was selected for follow-up semi-structured interviews using critical sampling.
• Experimental Tool: A custom political discourse analysis tool built using ChatGPT (API) to analyze Greek political speeches, identifying topics, sentiment, and polarization.
• Procedure:
  • Baseline Collection: Questionnaires on demographics, political orientation, and AI attitudes.
  • Interactive Tasks (A2):
    • I1 (Sufficiency): Users decided whether to trust the tool's summary or read the raw speech.
    • I2 (HITL Comparison): Control group saw AI-only results; Experimental group saw AI results validated by humans.
    • I3 (Expert Validation): All users saw results validated by data journalists.
    • I4 (Transparency): Users assessed results without access to the underlying speech text (lack of transparency).
  • Data Collection: Quantitative data via Likert scales and logs; Qualitative data via semi-structured interviews focusing on experiences, perceptions of stakeholders, and HITL mechanisms.
• Analysis:
  • Quantitative: Spearman's correlation (for small sample size/non-normal distribution) and point-biserial correlation.
  • Qualitative: Deductive thematic analysis based on the Bowtie model's themes.

3. Key Contributions

1. The Bowtie Model of Trust: A novel, extensible framework that moves beyond one-directional trust models to systematically map the bidirectional relationships between user contextual factors and LLM systemic elements.
2. Empirical Evidence of Trust Dynamics: The study provides the first evidence-based mapping of how specific trustor factors (e.g., political engagement, technical background) interact with specific trustee elements (e.g., HITL, transparency, stakeholder type).
3. Human-Centric Insights: Detailed analysis of how Human-in-the-Loop (HITL) and transparency mechanisms specifically alter trust attitudes in political contexts.

4. Key Results & Findings

The study yielded eight major takeaways:

• Trustor Actions & Competence: Users with negative predispositions toward LLM competence (fear of bias/hallucination) actively sought verification (reading raw text) rather than trusting the visualization. Trust is heavily influenced by the user's belief in the system's alignment with their own judgment.
• Expertise Shapes Trust: Users with a background in science/technology had more realistic (and sometimes lower) trust in LLMs as a discipline, viewing them as "faulty" tools rather than "god-like" solutions. Conversely, high interest in AI correlated with higher trust and willingness to use it.
• Domain Knowledge vs. Tech Competence: Trust in the specific political tool was driven more by the user's domain knowledge (political awareness) and their perception of the scientists involved than by the tool's raw technological performance.
• Differentiation of Scientists: Trust varied significantly by the type of expert involved:
  • Data Scientists: Neutral/Positive (focused on design/utility).
  • Data Journalists: Mixed/Skeptical (concerns about political bias and lack of objectivity).
  • Political Scientists: Highly Positive (viewed as most relevant and capable of unbiased analysis).
• Stakeholder Skepticism: Trust in private tech companies was low due to profit motives, while public universities were highly trusted. EU funding agencies elicited controversial views (seen as both supportive and overly restrictive).
• Impact of HITL: Trust significantly increased when human involvement was disclosed. However, the type of human mattered; data journalists did not boost trust as much as general human validation due to skepticism about their objectivity in political matters.
• Impact of Transparency: Lack of transparency (hiding the source text) caused a consistent decline in trust. Users relied on their own political knowledge to fill the gap, often leading to lower accuracy assessments if the AI's output conflicted with their expectations.
• Inherent Overlaps: Trust attitudes toward the "tool" and the "scientific discipline" are deeply intertwined. Users unconsciously transfer their general views on AI competence to specific tool evaluations.

5. Significance and Implications

• For Policymakers: Highlights the need to improve public AI literacy, particularly among non-experts, to mitigate misconceptions. Policies should encourage transparent methodologies and clear disclosure of human involvement (HITL) to build robust trust.
• For Practitioners/Developers: LLM-based tools in high-stakes domains should integrate HITL approaches but must carefully select the type of human validator (e.g., political scientists over journalists for political discourse) to maximize trust. Transparency (providing source data) is non-negotiable for maintaining high trust levels.
• For Researchers: The Bowtie model offers a reproducible, extensible baseline for studying trust in other sociotechnical domains (e.g., biomedical LLMs), moving the field away from isolated, one-directional trust metrics toward a holistic ecosystem view.
• Theoretical Impact: The paper challenges the possibility of "universal trust" in LLMs, suggesting that trust is inherently contextual and that the opacity and lack of intentionality in LLMs make complete, uninhibited trust a utopic goal.

In conclusion, the paper successfully demonstrates that trust in LLMs is not a static attribute of the technology but a dynamic, multi-faceted relationship shaped by the interplay of user psychology, domain expertise, and specific system design choices.