Agent Role Structure and Operating Characteristics in Large Language Model Clinical Classification: A Comparative Study of Specialist and Deliberative Multi-Agent Protocols

This study demonstrates that, even with fixed model parameters, altering the internal role decomposition of multi-agent LLM systems from a generic deliberative to a feature-specialist structure acts as a structured inductive bias that systematically reshapes sensitivity-specificity trade-offs and error distributions across different clinical classification tasks.

The Gist

Imagine you are trying to diagnose a patient's illness, but instead of asking one doctor, you ask a team of AI "doctors." This paper asks a very specific question: Does it matter how we organize that team?

The author, Callum Anderson, wanted to know if simply changing the job descriptions of the AI agents (while keeping the AI's "brain" exactly the same) would change the final diagnosis.

Here is the breakdown using simple analogies:

The Setup: The Same Brain, Different Jobs

Usually, when people build AI teams, they change the software, the training, or the rules. This study was different. The author used the exact same AI model (a specific version of Llama 3.1) for every single test. The only thing that changed was how the work was divided up.

He tested two different "team structures" on two common medical datasets (Heart Disease and Diabetes):

1. The "Generalist" Team (Generic Deliberative)

• The Analogy: Imagine two senior doctors sitting in a room. Both of them read the entire patient file from start to finish. They both look at the heart rate, the blood sugar, the age, and the family history. Then, they write down their opinion, and a third doctor (the "Judge") listens to both and makes the final call.
• The Vibe: "Let's look at the whole picture together."

2. The "Specialist" Team (Feature-Specialist)

• The Analogy: Imagine a medical team where the work is split up.
  • Doctor A is only allowed to look at one specific thing (e.g., just the cholesterol level). They ignore everything else.
  • Doctor B is only allowed to look at one different thing (e.g., just the blood pressure).
  • They write their notes based only on that one thing, and then the "Judge" combines their notes with the full file to make a decision.
• The Vibe: "You look at the engine; I'll look at the tires. Don't talk to each other."

The Results: It's Not Just About Being "Right"

The study found that changing the team structure didn't just make the AI slightly better or worse. It completely shifted the type of mistakes the AI made.

Think of medical diagnosis as a balance scale between two fears:

1. Missing a disease (False Negative): Telling a sick person they are healthy.
2. Crying Wolf (False Positive): Telling a healthy person they are sick, causing unnecessary panic and tests.

Case Study A: Heart Disease (The Cleveland Dataset)

• The Generalist Team: Was okay, but made a moderate number of mistakes on both sides.
• The Specialist Team: Became a super-skeptic.
  • Result: It got better at spotting healthy people (fewer false alarms).
  • Trade-off: It became slightly worse at spotting sick people (it missed a few more heart cases).
  • Analogy: The Specialist team was like a security guard who is very strict about letting people in (diagnosing disease), but very quick to let people out (saying they are healthy).

Case Study B: Diabetes (The Pima Dataset)

• The Generalist Team: Again, did a solid, balanced job.
• The Specialist Team: Became a super-worrier.
  • Result: It caught almost every single diabetic patient (very high sensitivity).
  • Trade-off: It screamed "DANGER!" for many healthy people who weren't actually sick (lots of false alarms).
  • Analogy: On this dataset, the Specialist team was like a smoke detector that is so sensitive it goes off when you just toast a piece of bread.

The Big Takeaway

The most important lesson from this paper is that how you organize the AI is just as important as the AI itself.

• The "Inductive Bias": The author calls this a "structured inductive bias." In plain English, this means the way you tell the AI to think (by giving it a specific job) acts like a hidden rule that shapes its personality.
• Why it matters: In real medicine, you might want different "personalities" for different tasks.
  • If you are screening for a deadly disease (like cancer), you want the "Super-Worrier" (Specialist on Pima) so you don't miss anyone, even if it means more false alarms.
  • If you are confirming a diagnosis before a risky surgery, you want the "Super-Skeptic" (Specialist on Heart) so you don't operate on a healthy person.

Summary

This paper proves that you don't need to retrain a complex AI to change how it behaves. You can simply rewrite the job descriptions for the AI agents. By swapping between "Generalists" who see the whole picture and "Specialists" who focus on tiny details, you can dial the AI's behavior up or down to fit the specific safety needs of a medical situation.

It's like having a Swiss Army knife: the tool is the same, but if you use the screwdriver blade instead of the knife blade, you get a completely different result.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) are increasingly used for clinical decision support, the specific impact of internal role decomposition within multi-agent systems remains poorly understood. Previous studies comparing single-agent and multi-agent prompting often confound architectural changes with variations in model configuration, training data, or decoding parameters. Consequently, it is unclear whether the observed performance differences in clinical prediction tasks are due to the structure of the agent roles (e.g., specialization vs. generalization) or other experimental variables. The paper seeks to isolate role decomposition as the sole manipulated variable to determine how it systematically alters classification behavior and operating characteristics (sensitivity vs. specificity) without modifying model weights.

2. Methodology

The study employs a controlled experimental design comparing two deterministic multi-agent protocols under identical conditions.

• Experimental Setup:

  • Model: All agents use the same base model (llama3.1:8b) with fixed weights.
  • Decoding: Deterministic settings (temperature = 0) to ensure reproducibility.
  • Workflow: Both protocols utilize a Directed Acyclic Graph (DAG) structure where upstream agents generate structured intermediate assessments, which are then processed by a downstream adjudicator.
  • Datasets: Two tabular clinical benchmarks from the UCI repository:
    1. Cleveland Heart Disease: 303 records, 13 features (binary classification: disease present/absent).
    2. Pima Indians Diabetes: 768 records, 8 features (binary classification).
  • Control: Model weights, decoding temperature, computational budget, aggregation logic, and implementation code are held constant across all experiments.

• Protocols Compared:

  1. Generic Deliberative (GD): Two parallel "generalist" clinician agents. Each agent evaluates the entire patient record independently and produces a structured opinion (signal, strength, justification).
  2. Feature-Specialist (FS): Two parallel "specialist" agents. Each agent is restricted to evaluating only one specific clinical feature (e.g., one agent looks at blood pressure, another at glucose). They produce structured opinions based solely on their assigned feature.
  • Adjudication: In both cases, a third agent (the adjudicator) receives the full patient record and the structured outputs from the two upstream agents to make the final binary prediction.

• Evaluation Metrics:

  • Standard classification metrics: Accuracy, Macro-Precision, Macro-Recall, Macro-F1.
  • Operating characteristics: Sensitivity (Recall for positive class) vs. Specificity (Recall for negative class).
  • Confusion matrices to analyze error distribution (False Positives vs. False Negatives).

3. Key Contributions

• Isolation of Role Decomposition: The study is the first to rigorously isolate internal role structure as the independent variable in LLM clinical classification, proving that prompt-level architecture alone can shift operating characteristics.
• Demonstration of Inductive Bias: It establishes that role organization functions as a "structured inductive bias," altering error distributions (sensitivity-specificity trade-offs) without any changes to model parameters or training.
• Dataset-Dependent Effects: The paper reveals that the effect of role specialization is not uniform; it depends heavily on the nature of the dataset (e.g., categorical vs. continuous features, class imbalance).

4. Results

The results show that altering role structure systematically reshapes performance, but the direction of the shift varies by dataset:

• Cleveland Heart Disease Dataset:

  • Feature-Specialist (FS) outperformed Generic Deliberative (GD) in overall accuracy (0.72 vs. 0.65) and Macro-F1 (0.71 vs. 0.65).
  • Operating Shift: FS shifted the operating point toward higher specificity (+0.22) and lower sensitivity (-0.13).
  • Error Analysis: FS significantly reduced False Positives (29 vs. 66) but increased False Negatives (57 vs. 39). This suggests that restricting agents to single features prevents weak signals from accumulating into a false positive diagnosis, making the system more conservative.

• Pima Indians Diabetes Dataset:

  • Generic Deliberative (GD) outperformed Feature-Specialist (FS) in overall accuracy (0.68 vs. 0.51) and Macro-F1 (0.64 vs. 0.49).
  • Operating Shift: FS induced a pronounced class asymmetry, achieving very high recall for the positive class (0.95) but extremely low recall for the negative class (0.27).
  • Error Analysis: FS resulted in a massive increase in False Positives (365 vs. 104) compared to GD. In this context, splitting the data by features led to over-sensitivity, where individual features triggered positive classifications even when the overall record did not support it.

5. Significance and Implications

• Architectural Design as a Control Mechanism: The findings demonstrate that multi-agent prompt architecture is not merely a superficial implementation detail but a core modeling decision. It can be used as an explicit mechanism to tune the sensitivity-specificity trade-off in safety-critical systems.
• Contextual Deployment:
  • In scenarios where False Positives are costly (e.g., unnecessary patient anxiety or invasive follow-up tests), a Feature-Specialist architecture (as seen in the Cleveland results) may be preferable to reduce noise.
  • In scenarios where False Negatives are critical (e.g., missing a diagnosis), a Generic Deliberative approach might be safer, as it allows for holistic reasoning that can integrate weak signals (as seen in the Pima results).
• Future Directions: The authors argue for formalizing frameworks to tune LLM behavior through systematic manipulation of role decomposition. Future work should explore these effects in multiclass tasks, higher-dimensional EHRs, and with probabilistic decoding to better understand calibration and generalizability.

In conclusion, the paper proves that how an LLM is prompted to organize its internal reasoning (specialist vs. generalist) fundamentally changes its decision-making profile, offering a new lever for optimizing clinical AI systems without retraining models.