Technical Summary: Social Chain of Thought (SCoT)

Problem Statement

Medical diagnostic reasoning is a high-stakes domain for Large Language Models (LLMs), where transparency and accuracy are critical. While monolithic LLMs dominate current medical AI applications, they face inherent limitations in complex differential diagnosis, particularly regarding the "endogeneity problem" of self-evaluation. As noted by Huang et al. (2024), single-model self-evaluation propagates correlation biases inherent to the generation source, which cannot be solved merely by scaling parameters or increasing inference time. Furthermore, existing multi-agent approaches have not clearly established when such systems are necessary, why they outperform monolithic inference, or how they handle the trade-off between compute scaling and architectural heterogeneity. The core challenge is to determine if structured, multi-agent deliberation can recover ground-truth diagnoses that single models miss, particularly in difficult cases, without simply relying on brute-force compute scaling.

Methodology

The paper introduces Social Chain of Thought (SCoT), a multi-round pipeline designed to structure multi-agent interaction as a deliberative framework for collaborative reasoning. The methodology is grounded in medical differential diagnosis and clinical Delphi methods.

Architecture and Pipeline

SCoT operates on a backend LLM (tested across various open-weight and closed-weight models) to generate a panel of five medical specialists. These agents are "persona-conditioned" to simulate distinct medical specialties relevant to the input case. The pipeline consists of seven distinct rounds:

1. Specialized Ranking: Agents assess the case from their specific specialty's perspective and self-assess relevance.
2. Symptom Management: Agents perform a triage process, simulating the treatment of obvious or life-threatening conditions before specialist consultation.
3. Team Independent Differentials: Each agent provides an independent list of likely diagnoses based on their specialty.
4. Master List: Independent differentials are compiled into a deduplicated, unranked list.
5. Refinement: A three-sub-round debate phase where agents support, challenge, or remain neutral regarding diagnoses on the Master List. Agents must provide evidence, reason through challenges, and update positions based on counter-evidence.
6. Voting: Agents rank diagnoses by plausibility. Scores are aggregated using Borda counting, weighted by heuristic components (diagnostic insight, breadth, and actionability).
7. Can't Miss: A final safety check where agents identify critical or life-threatening diagnoses that may have been missed.

Experimental Design

The system was evaluated using the Open-XDDx benchmark, a physician-curated dataset of 570 clinical vignettes.

• Baselines: Performance was compared against single-agent monolithic inference and a "Best-of-n" scaling approach (repeated sampling with majority voting).
• Ablation Studies:
  • Model Size: Tested across Qwen-2.5 (1.5B to 32B), Gemma-4, and GPT-5 variants to determine viability thresholds.
  • Agent Count: Compared 5-agent vs. 3-agent configurations.
  • Single-Agent Pipeline: Ran a single agent through the full 7-round SCoT pipeline to isolate the effect of the structure from agent heterogeneity.
  • Temperature: Tested "innovative" (0.7) vs. "conservative" (0.3) agent personas.
• Metrics: Standard information retrieval metrics (Recall, Precision, F1) adapted for differential diagnosis, measuring True Positives (ground-truth recovery), False Negatives (missed ground truth), and False Positives.

Key Contributions

1. SCoT Pipeline: A reproducible, open-weight architecture for multi-agent systems in applied diagnostic reasoning.
2. Open Source Framework: A framework for studying SCoT as a support tool in clinical settings.
3. Evidence of Social Scaling: Demonstrated that multi-agent performance gains are distinct from resource scaling (compute), showing that heterogeneity drives recall improvements that monolithic inference cannot replicate.
4. Ablation Insights: Isolated performance metrics by removing multi-round scaling effects, proving that the "social" aspect (interaction) is the primary driver of improvement, not just the number of inference passes.
5. Viability Thresholds: Identified a minimum model capability floor (between 1.5B and 3B parameters) below which multi-agent heterogeneity becomes destabilizing rather than productive.

Results

• Recall Improvement: SCoT improved recall by 4 to 12 percentage points across most backend models compared to single-agent baselines.
• Viability Threshold: The 1.5B parameter model (Qwen-2.5-1.5B) showed a degradation in performance (-3.7 percentage points) under SCoT, indicating that models below a certain capability threshold lack the knowledge to effectively critique weak proposals from other agents. The 3B model, however, gained +11.8 points.
• Structure vs. Compute:
  • Single-Agent Pipeline: When a single agent ran through the SCoT pipeline, recall degraded (-5.44 points) while precision increased. This suggests the pipeline acts as a filter that narrows the candidate space, which is beneficial for precision but harmful for recall without heterogeneity.
  • Best-of-n: Repeated sampling (Best-of-35) failed to match the recall of the 5-agent SCoT configuration, confirming that the gains are not merely due to increased compute but to the diversity of perspectives.
• Difficulty-Conditioned Consensus: SCoT provided the most significant gains in the hardest cases (where monolithic baselines performed poorly).
  • In the hardest quartile (Q1), SCoT boosted the F1 score by 15.1 percentage points (a 71.4% relative lift).
  • In the easiest cases (Q4), SCoT actually reduced precision by 8.3 points, suggesting "overthinking" in simple cases.
  • For cases where the baseline completely failed (0 ground-truth recovery), SCoT recovered at least one ground-truth diagnosis in 53.9% of cases.
• Convergence Dynamics: In difficult cases, a significant portion of true-positive diagnoses (up to 36% in baseline-failure cases) were only recovered during the late-stage refinement rounds, demonstrating that social scaling allows the system to revise initial positions and recover missed diagnoses.

Significance and Claims

The paper claims that SCoT offers a complementary approach to iterative scaling for medical diagnostic systems. Its significance lies in demonstrating that social scaling—structuring interaction to encourage deliberation and diverse exploration—is essential for complex reasoning tasks where monolithic inference struggles.

The authors argue that for high-stakes domains like healthcare, relying solely on optimization and narrow reasoning paths puts users at risk. SCoT leverages the inherent capacity of LLMs for deliberative reasoning to broaden the space of active inference. The system is most effective not as a replacement for human care, but as a mechanism to recover ground-truth diagnoses in uncertain or complex scenarios where single models fail. The paper modestly concludes that while SCoT shows promise in pushing models to consider alternatives they might miss in solitude, its viability is contingent on the underlying model's capability to support a balance of heterogeneity and consensus. Future work is suggested to explore these dynamics in domains beyond diagnostic reasoning.