Point of Order: Action-Aware LLM Persona Modeling for Realistic Civic Simulation

1. Problem Statement

Large Language Models (LLMs) hold significant potential for simulating complex, multi-party deliberations in civic settings (e.g., courtrooms, school boards, city councils). However, realistic simulation is currently hindered by a critical data limitation: speaker-attributed data.

• The Gap: Public meeting recordings (often on Zoom) are transcribed via Automatic Speech Recognition (ASR), which typically assigns generic, anonymous labels (e.g., Speaker_1, Speaker_2).
• Consequence: Without consistent identity linking across multiple recordings, models cannot learn the specific personas, behavioral patterns, or interaction styles of individual participants. This prevents the simulation of "what-if" scenarios involving specific real-world actors.
• Additional Complexity: Real-world deliberations involve noise, overlapping speech, and rapid transitions between formal procedure and spontaneous debate, requiring models to be robust to these conditions while maintaining coherence.

2. Methodology

The authors propose a reproducible pipeline that transforms raw public Zoom recordings into high-fidelity, speaker-attributed datasets with rich metadata, followed by a specialized fine-tuning framework.

A. Data Collection & Speaker Linking Pipeline

The authors developed a lightweight multimodal framework to link speakers across different video recordings without requiring specialized metadata beyond the video itself.

1. Visual Cue Extraction: Using the Zoom "gallery view," the system identifies active speakers via colored borders. It crops the participant name tile and uses Optical Character Recognition (OCR) (with super-resolution for low-quality videos) to extract names.
2. Audio-Visual Alignment: ASR transcripts (generated via Whisper) are aligned with the visual diarization timeline to assign canonical speaker labels to text segments.
3. Identity Propagation: Names are normalized and clustered using fuzzy matching to ensure the same individual is identified consistently across different meetings.

B. Action-Aware Persona Modeling

To capture the behavior of speakers, not just their identity, the authors introduce Action-Aware Persona Modeling.

• Metadata Extraction:
  • Persona Profiles: Extracted from long monologues using LLMs (GPT-5) to summarize goals, tone, boundaries, and policy positions.
  • Topic Extraction: Meeting transcripts are chunked and summarized to identify global meeting topics.
  • Action Tags: A taxonomy of pragmatic speech acts (e.g., [propose_motion], [call_vote], [ask_clarification]) is automatically assigned to every utterance. These tags describe the function of the speech rather than just sentiment.
• Dataset Construction: Transcripts are serialized into a ChatML-style format. Each training example includes the conversation history, the target speaker's persona profile, meeting topics, and action tags prepended to utterances.
• Fine-Tuning: The authors use Parameter-Efficient Fine-Tuning (PEFT) (specifically QLoRA) to adapt large models (LLaMA-3.1-70B, Qwen-2.5-72B, GPT-OSS-120B) to specific speakers. This allows for efficient adaptation to dozens of individuals.

C. Simulation Setup

Simulations are run in a round-robin fashion where agents act as specific participants. The system supports:

• Time-Aware vs. Time-Unaware: Agents can be prompted with explicit timestamps and agenda items to maintain temporal coherence.
• System Prompts: Structured prompts include persona descriptions, in-context examples, micro-profiles (quantitative behavioral stats), and conversation context.

3. Key Contributions

1. Multimodal Speaker-Linking Pipeline: A scalable method to assign stable identities to speakers in public Zoom recordings using visual cues and OCR, bypassing the need for manual annotation.
2. Action-Aware Framework: A novel approach demonstrating that augmenting training data with structured pragmatic action tags significantly enhances simulation fidelity, even for non-fine-tuned models.
3. Three New Public Datasets: The release of the first speaker-labeled datasets for government deliberations:
   • DC Court of Appeals: 10 transcripts, ~193k words.
   • Albemarle County School Board: 32 transcripts, ~594k words.
   • Waipā District Council: 80 transcripts, ~933k words.
   • Total: ~1.72 million words across judicial, educational, and municipal contexts.
4. New Evaluation Metrics:
   • Classifier Fool Rate (CFR): Measures how often a classifier mistakes generated text for real text from a specific speaker (fidelity).
   • Speaker Attribution Accuracy (SAA): Measures how well a classifier can distinguish between different speakers in the same meeting (distinctiveness).

4. Results

The authors evaluated their models across three dimensions: linguistic quality, persona fidelity, and identity distinctiveness.

• Perplexity (PPL): Fine-tuning with action tags reduced perplexity by 67% (e.g., from 20.37 to 6.64 for GPT on the Albemarle dataset), indicating a much better fit to the target persona's linguistic patterns.
• Classifier Fool Rate (CFR): Fine-tuning increased CFR by 121% (from 0.29 to 0.64), showing that generated outputs are stylistically convincing and indistinguishable from real speakers to a classifier.
• Speaker Attribution Accuracy (SAA): Improved by 96% (from 0.23 to 0.45), proving the models capture unique rhetorical cues rather than just generic domain language.
• Human Evaluation (Turing Test): In a blind test, human annotators correctly identified simulated transcripts only 45.5% of the time (below the 50% chance level), indicating the simulations are often indistinguishable from real deliberations.
• Temporal Grounding: Adding time-aware cues significantly improved topic coverage for LLaMA (71.4% → 94.4%) and Qwen (81.4% → 99.2%), helping models navigate long meetings without losing the agenda.
• Emergent Behavior: Simulations successfully reproduced institutional rituals (e.g., roll calls, moments of silence, formal openings) and procedural logic (e.g., question-and-answer sequences in court) without explicit programming of these rules.

5. Significance and Implications

• Policy Analysis & "What-If" Scenarios: This framework enables the creation of a "digital twin" for civic institutions. Policymakers can simulate how outcomes change with different agendas, participant strategies, or procedural rules in a risk-free environment.
• Scalability: The pipeline is reproducible and relies on publicly available data, making it applicable to a wide range of deliberative bodies beyond the specific datasets released.
• Ethical Considerations: The authors acknowledge the dual-use nature of this technology. While it aids in understanding democratic processes, realistic persona models could be misused for disinformation or manipulating public opinion. They call for community guidelines on the responsible deployment of such models.
• Technical Advancement: The work demonstrates that combining structured metadata (action tags) with parameter-efficient fine-tuning is a superior strategy for modeling complex human interaction compared to prompting alone or generic fine-tuning.

In summary, this paper bridges the gap between raw public data and high-fidelity multi-agent simulation, providing a robust toolkit for studying and simulating the dynamics of democratic deliberation.