The Collective Turing Test: Large Language Models Can Generate Realistic Multi-User Discussions

1. Problem Statement

While Agent-Based Modeling (ABM) is widely used to simulate collective human behavior (e.g., polarization, segregation), traditional models often oversimplify human interactions, limiting their utility in policy design. Recent advances in Generative AI and Large Language Models (LLMs) offer a potential solution by enabling agents to communicate via natural language. However, a critical gap remains: the validity of using LLMs to simulate multi-user social media conversations has not been rigorously tested.

Existing validation methods often rely on:

• Comparing AI outputs against crowd-worker generated text (which lacks the authenticity of real user data).
• Automated ML classifiers (e.g., BERT) rather than human perception.
• Single-turn interactions rather than complex, multi-turn threads.

The authors pose the question: Can LLMs generate multi-user social media discussions that are indistinguishable from authentic human conversations?

2. Methodology

The study employs a "Collective Turing Test" design, preregistered to ensure scientific rigor.

Data Collection & Stimuli Generation

• Source Data: Authentic Reddit conversations were scraped from the Pushshift dataset (December 2023) to ensure they post-date the training cutoffs of the models used (GPT-4o: Oct 2023; Llama 3: Dec 2023).
• Topics: 16 posts across 8 categories (food, leisure, opinions, etc.).
• Structure Preservation: The conversational tree structure (who replied to whom) and word counts of original comments were preserved.
• AI Generation: Two LLMs (GPT-4o and Llama 3 70B) were prompted to generate replies matching the original word counts and thread structure.
  • Prompting Strategy: Minimalist. Models were instructed to act as Reddit users without specific personas or few-shot examples.
  • Parameters: Three temperature settings (0.2, 0.7, 1.2) were used to vary creativity.
• Stimuli: 574 unique stimuli were created, varying by model, temperature, and conversation length (1, 2, 4, 6, 8, or 16 comments).

Experimental Design

• Participants: 251 English-fluent participants recruited via Prolific (203 retained after attention checks).
• Task: Participants viewed side-by-side pairs of conversations on the same topic. One was authentic (human), the other AI-generated. Both had identical structure and length.
• Objective: Identify which conversation was human-generated.
• Additional Tasks: Participants reported confidence levels and provided qualitative feedback on cues used to distinguish the texts.
• Analysis: A multilevel logistic regression was used to analyze success rates, accounting for participant nesting.

3. Key Contributions

1. First Real-World Validation: This is the first study to test LLM mimicry of social media behavior using authentic human conversations as the control, rather than synthetic or crowd-worker data.
2. The "Collective Turing Test" Framework: Extends the classic Turing Test from a single dyadic interaction to a multi-user, multi-turn group discussion context.
3. Model Comparison: Provides empirical evidence comparing the "human-likeness" of a leading commercial model (GPT-4o) versus a high-performance open-weight model (Llama 3 70B) in informal social settings.
4. Qualitative Cues: Identifies specific linguistic and stylistic markers humans use to detect AI, offering a roadmap for improving future simulation realism.

4. Key Results

Overall Deception Rate

• Participants correctly identified human conversations 61% of the time.
• Conversely, 39% of the time, participants mistook AI-generated conversations for human ones.
• This exceeds the traditional Turing Test threshold (30% misclassification), indicating that LLMs can convincingly simulate social media discourse.

Model Performance

• Llama 3 70B was significantly harder to distinguish from humans than GPT-4o.
  • Success rate for identifying humans in Llama 3 threads: 56% (barely better than random chance).
  • Success rate for identifying humans in GPT-4o threads: 66%.
• Reasoning: Llama 3's training on internet-scale corpora (including forums) and its less "polished" output style (due to lack of heavy commercial safety tuning compared to GPT-4o) made it appear more organic and informal.

Conversation Length

• Non-Linear Relationship: The hypothesis that longer conversations are easier to detect was only partially supported.
  • Detection accuracy improved as length increased up to 8 comments.
  • Beyond 8 comments, accuracy declined, approaching random chance at 16 comments.
• Interpretation: Very short threads lack context for detection; moderately long threads allow pattern recognition; very long threads may induce participant fatigue or mask AI artifacts through variance.

Qualitative Findings (Human Detection Cues)

Participants relied on three main categories to identify AI:

1. Style (82.5% of cues):
   • Tone: AI lacked sarcasm, irony, and humor.
   • Formality: AI text was perceived as "too polished," "robotic," or overly polite.
   • Lack of Slang/Errors: AI rarely used typos, domain-specific slang, or profanity.
2. Content (49.1% of cues):
   • Conformity: AI tended to agree excessively with others, lacking the conflict or polarization typical of Reddit.
   • Authenticity: AI failed to provide personal anecdotes or genuine emotional depth.
   • Verbosity: AI provided excessive explanations even when word counts were matched.
3. Format (17.5% of cues):
   • Visual cues like emoji usage and response length (perceived as too long).

5. Significance and Implications

For Social Simulation (ABM)

• Validation: The study confirms that LLMs, particularly Llama 3 70B, are viable agents for simulating social media environments.
• Recommendation: Researchers aiming to simulate naturalistic, informal interactions should prefer models like Llama 3 over highly aligned commercial models like GPT-4o, as the latter may be too safe and polite to capture the nuance of online conflict.

For Safety and Misinformation

• Risk of "AI Swarms": The high deception rate (39%) highlights a significant risk. Bad actors could use LLMs to generate coordinated, realistic-looking discussions to manufacture artificial consensus, engineer polarization, or spread misinformation without detection.
• Limitations in Conflict Simulation: Current LLMs struggle to simulate toxic, polarized, or emotionally charged discourse effectively. They tend to be overly agreeable and neutral, which may limit their utility in studying extreme online behaviors unless specific "jailbreaking" or persona engineering is applied.

Future Directions

The authors suggest extending this framework to other models (e.g., Claude, Mistral), testing in multilingual contexts, and exploring persona-driven prompting to see if assigning specific, flawed human personas improves the simulation of emotional and controversial dynamics.

Conclusion

The paper concludes that while LLMs have reached a level of sophistication where they can deceive humans in group social media settings, they still exhibit distinct "tells" regarding politeness, lack of personal narrative, and excessive conformity. This duality presents both a powerful tool for computational social science and a warning regarding the potential for inauthentic content generation.