Technical Summary: Belief Coevolution in a Social Network of Generalist and Specialist Large Language Models

Problem Statement

As Large Language Models (LLMs) are increasingly deployed in multi-agent environments for tasks ranging from collaborative reasoning to debate, the mechanisms governing how beliefs form, propagate, and reach consensus within these populations remain poorly understood. While existing research focuses on behavioral outcomes (e.g., task completion, voting), there is a lack of controlled studies isolating the specific drivers of belief diffusion. It is unclear whether collective outcomes in LLM populations are driven primarily by domain specialization (the underlying model's training data), social-role assignment (persona-style prompting), or network structure. Furthermore, it is unknown if classical opinion-dynamics models can explain these emergent behaviors or if the complexity of heterogeneous LLM populations requires new theoretical frameworks.

Methodology: CoevolveSim

The authors introduce CoevolveSim, a network-mediated belief diffusion framework designed to isolate and study three key factors:

1. Domain Specialization: The specific LLM powering an agent (e.g., a generalist vs. a specialist finetuned on medical, chemical, or cybersecurity data).
2. Social Role: A persona-style label assigned to the agent (e.g., "Physician," "Engineer") communicated to the agent and its neighbors.
3. Network Structure: The topology of connections between agents (Erdős-Rényi vs. Watts-Strogatz).

Experimental Design

• Agents: The study utilizes 15 LLMs from the Llama-3 family. One serves as a generalist (Llama-3.1-8B-Instruct), while 14 are specialists finetuned on domain-specific corpora.
• Task: Agents discuss 20 medical-indication statements (some true, some false).
• Process:
  • Round 0: Agents provide initial beliefs without neighbor information.
  • Rounds 1–10: In each synchronous round, an agent receives a summary of its neighbors' beliefs from the previous round and updates its own belief.
  • Belief Extraction: Beliefs are quantified using next-token probabilities for "correct," "incorrect," or "neither," discretized into a single state.
• Scenarios: The authors run 1,280 controlled simulations across four scenarios to isolate variables:
  • I. Baseline: All generalists, uniform role ("LLM").
  • II. Random Roles: All generalists, random social roles assigned.
  • III. Specialists (Random): Mixed generalists and specialists with random roles.
  • IV. Specialists (Matched): Mixed agents with roles aligned to their domain (e.g., a medical specialist assigned the "Physician" role).
• Analysis: The study employs linear mixed-effects models to analyze agent-level metrics (plasticity, directedness, outgoing influence) and population-level metrics (consensus change). Additionally, a hierarchy of surrogate models (M1–M4) is fitted to determine which classical opinion-dynamics mechanisms (persistence, local/global belief composition, agent identity) are necessary to reproduce the observed dynamics.

Key Results

1. Drivers of Belief Diffusion

• LLM Heterogeneity is the Primary Driver: Introducing specialist LLMs (Scenario III vs. II) more than doubles the shift in population-level consensus compared to merely assigning random social roles to generalists. Heterogeneity creates significant asymmetries in influence, leading to the emergence of "opinion-leader-like" agents (agents whose updates consistently drive neighbors' updates).
• Social Roles Reshape Individual Trajectories, Not Consensus: Assigning random social roles to generalist agents (Scenario II) increases individual belief "plasticity" (frequency of change) but causes beliefs to oscillate rather than converge monotonically. Consequently, the impact on population-level consensus is minimal compared to the introduction of specialists.
• Role-Specialization Alignment: Aligning social roles with underlying domain expertise (Scenario IV) has negligible effects on individual belief trajectories and slightly dampens the consensus change observed in Scenario III.
• Network Structure: Network topology (ER vs. WS) modulates how individual agents revise beliefs (affecting plasticity and directedness) but has a negligible effect on the final population-level consensus.

2. Explanatory Power of Opinion-Dynamics Models

The study reveals a dissociation between the mechanisms required to predict individual behavior versus collective outcomes:

• Generalist Populations (Scenarios I & II): Simple models based on individual persistence (M1) are sufficient to reproduce population-level consensus. Adding social features (neighborhood composition, identity) provides little additional explanatory power for the collective outcome.
• Heterogeneous Populations (Scenarios III & IV): Simple persistence models fail to reproduce consensus. Accurately reproducing collective dynamics requires models that incorporate population-level or local belief composition (M2/M3). Furthermore, predicting individual belief transitions in heterogeneous populations requires incorporating agent identity (domain specialization and social role).

Significance and Claims

The paper claims that realistic simulation of belief diffusion in multi-agent LLM systems requires a diverse set of underlying LLMs, not persona prompting alone.

• Limitations of Persona Prompting: While social roles alter individual update behaviors, they do not generate the substantive population-level diversity or consensus shifts seen when introducing domain-specialized models.
• Dissociation of Levels: The factors that best predict individual belief transitions (agent identity, local neighborhood) are not the same as those required to reproduce collective consensus (global belief composition). This suggests that agent-level and population-level explanations of belief diffusion are only partially aligned.
• Emergence of Influence: Unlike homogeneous populations where influence is uniform, heterogeneous populations naturally give rise to opinion leaders and followers, driven by the underlying knowledge priors of the specialist models.

The authors conclude that understanding LLM belief diffusion requires moving beyond behavioral outcomes to analyze the interplay of model heterogeneity, social structure, and the specific mechanisms of belief revision, noting that classical opinion-dynamics models only partially capture these complex dynamics.