Calibrating the Instrument: Controllability of an LLM-Driven Synthetic Population

Technical Summary: Calibrating the Instrument – Controllability of an LLM-Driven Synthetic Population

Problem Statement
Generative Synthetic Populations (GSP)—convergences of statistical population synthesis, agent-based modeling (ABM), and Large Language Model (LLM) agents—offer promise for urban simulation and institutional communication research. However, a fundamental validation gap exists: before a GSP instrument can be trusted to model real-world populations (external validity), its internal mechanics must be verified. Specifically, does the instrument respond to stimuli of known valence in an ordered, replicable, and group-structured manner? The paper argues that without establishing this property, termed controllability, any claims regarding the instrument's ability to track human behavior are logically premature. The authors posit that a synthetic population must first be shown to "track itself"—recovering the latent structure imposed upon it—before it can be used to track real humans.

Methodology: The SIVE Experiment
To address this, the authors introduce SIVE (Synthetic Instrument Validation Experiment), a minimal-model calibration bench designed to isolate the instrument's response function from the noise of real-world complexity.

• Experimental System (Montelago): A fictional Italian municipality with no real-world referent, populated by 120 synthetic personas. Each persona possesses a known, encoded latent attitudinal structure (specifically, institutional trust levels: Low, Medium, High) and a demographic profile, but the encoded scores are withheld from the LLM during inference. The LLM receives only the persona's background story and demographic data.
• Stimuli: The population is exposed to seven experimental conditions involving institutional communications about a municipal water network. These span a valence spectrum:
  • POS/POS2: Strongly positive (replicas for noise estimation).
  • NEG: Strongly negative.
  • POSW/POSW2: "Weakly positive" signals (POSW was a miscalibrated stimulus; POSW2 was a recalibrated version).
  • PLA: A thematically orthogonal placebo (Neighborhood Festival).
  • CTRL: No message.
• Procedure: Each of the 120 agents undergoes seven independent runs (one per condition). In each run, agents provide pre-stimulus ratings, react to the stimulus via free-text at three temporal intervals, and provide post-stimulus ratings.
• Parameters: Experiments were conducted across a temperature sweep ($t \in \{0.2, 0.5, 0.7\}$) using the DeepSeek deepseek-chat model.
• Validation Criteria: Seven pre-registered criteria (C1–C7) were defined to assess:
  1. Fidelity (C1): Correlation between encoded latent attitudes and pre-stimulus responses.
  2. Stability (C2): Consistency of pre-stimulus baselines across different conditions.
  3. Noise Floor (C3): Intrinsic variability measured via replica stimuli (POS vs. POS2) and single-profile repetition.
  4. Specificity (C4): Null effect of the placebo (PLA) compared to control (CTRL).
  5. Sensitivity (C5): Ordered response to stimuli of known valence.
  6. Ordering (C6): Rank correlation between observed and expected condition rankings.
  7. Receipt Check (C7): Detection of information adequacy shifts regardless of message valence.

Key Results
All seven pre-registered criteria passed at all three temperature settings ($t=0.2, 0.5, 0.7$).

• Fidelity & Stability: The instrument faithfully recovered the latent trust structure (C1: $r > 0.89$ for the primary trust variable) and maintained stable baselines across conditions (C2: $r_{min} > 0.85$).
• Noise Characterization: The cross-agent noise floor ($\sigma_{cross} \approx 1.4$) was found to be roughly double the intrinsic instrument noise ($\sigma_{instr} \approx 0.77$), the latter isolated via single-profile repetition. Crucially, the intrinsic noise was temperature-invariant.
• Calibration Success (POSW/POSW2): A central finding emerged from a stimulus calibration failure. The "weakly positive" stimulus (POSW), designed to be neutral-to-positive, was identified by the instrument as functionally negative, driving trust scores below the control baseline. Post-hoc analysis revealed the text encoded "problem-plus-uncertainty-plus-institutional-passivity." A recalibrated stimulus (POSW2) restored the expected ordering, demonstrating the instrument's capacity to diagnose design flaws in the stimuli themselves.
• Sensitivity & Ordering: The instrument successfully discriminated between stimuli, with the observed ranking of conditions matching the expected ranking (C6: $\tau \approx 1.0$) once the recalibrated POSW2 was included.
• Micro-Dynamics: While aggregate metrics showed consistent trends, individual agent trajectories revealed qualitatively distinct psychological mechanisms. For instance, under a negative stimulus, "Low Trust" agents exhibited either passive resignation or active civic mobilization, while "High Trust" agents experienced a collapse in perceived source credibility rather than just a drop in trust.

Significance and Claims
The paper claims that SIVE establishes a necessary logical prior for GSP research: controllability. The authors argue that characterizing an instrument's response function is distinct from, and prerequisite to, validating its external fidelity against human behavior.

• Internal Validity First: The study demonstrates that a GSP instrument can be rigorously calibrated to respond to known inputs in a structured way, independent of any comparison to real human data.
• Diagnostic Utility: The experiment highlights the instrument's ability to serve as a diagnostic tool for the stimuli themselves, revealing that "weakly positive" messages can be interpreted as negative by skeptical audiences due to institutional passivity.
• Precision: The study quantifies the instrument's noise floor, showing it is more precise than naive cross-agent analyses suggest, and that this precision is stable across temperature variations.
• Scope: The authors explicitly state that SIVE does not validate the instrument's ability to model real populations (external validity) or the specific dynamics of the Caffaro/Brescia case study that motivated the work. Instead, it provides the "characterized instrument" required to credibly pursue those future questions.

The paper concludes that the Montelago instrument is controllable and that the experimental logic of SIVE can be adapted to other instruments, providing a foundational step for the maturation of LLM-driven social simulation.