🤖 AI

Rethinking the Evaluation and Optimization of LLM-Based Social Simulation

This paper addresses the limitations of accuracy-based evaluation and hard-label training for LLM-based social simulations by introducing a subjectivity coefficient to quantify behavioral variability and proposing Subjectivity-Adaptive soft-Label Training (SALT), a method validated on the newly constructed SUBJSIM benchmark that effectively handles the inherent subjectivity of human responses.

Original authors: Pei Wang, Xu Chen, Ji-Rong Wen

Published 2026-08-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Pei Wang, Xu Chen, Ji-Rong Wen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet corners of social science, researchers have long sought to understand how people think, decide, and interact. For decades, the primary tools for this work have been surveys and controlled experiments, where real people are asked questions or placed in scenarios to see how they react. These methods are powerful, but they are also slow, expensive, and difficult to scale up to study complex global phenomena like political polarization or market shifts. Recently, a new tool has emerged: large language models. These are the sophisticated computer programs that can write stories, answer questions, and hold conversations. Scientists have begun using them to simulate human behavior, creating digital populations that can be tested in virtual environments. The hope is that these simulations could offer a low-cost, controllable way to study society. However, a fundamental problem has held this field back. When a computer program tries to mimic a human, how do we know if it is actually doing a good job? The standard way to check has been to ask a simple question: did the computer pick the exact same answer that a real person gave? If the answer matches, the simulation is considered successful. But this approach assumes that human behavior is like a math problem, where there is one single correct answer waiting to be found. In reality, human choices are far more fluid. The same person, faced with the same situation on two different days, might reasonably choose differently. A group of people with similar backgrounds might split their votes among several options. Treating a single human response as the only truth ignores this natural variety and the uncertainty that defines how we actually live.

A team of researchers at Renmin University of China has now challenged this standard approach, arguing that the way we currently test and train these social simulations is fundamentally flawed. They propose that human behavior is not a fixed target but a cloud of possibilities, and that trying to force a computer to hit a single point within that cloud is a mistake. To prove this, the team introduced a new way of thinking about "subjectivity," or how much a decision relies on personal feeling rather than objective fact. They found that for tasks like coding or math, where there is a clear right answer, the old methods work fine. But for social questions, where people's opinions vary widely, the old methods fail. When researchers train a model to simply copy the single answer a person gave, the model learns to be rigid and narrow. It forgets the other reasonable choices a person might have made, effectively losing the very essence of human unpredictability. Furthermore, when they try to judge the model's success by checking if it matches that single answer, they are often misled. A model might be terrible at capturing the full range of human thought but still get a high score just by luck, or a perfect model might get a low score simply because it didn't guess the one specific answer the tester happened to pick.

To fix this, the researchers developed a new method called Subjectivity-Adaptive soft-Label Training, or SALT. Instead of forcing the computer to memorize one specific answer, SALT teaches it to understand the landscape of possible answers. The system looks at many different situations that are similar to the one being studied and gathers the answers people gave in those nearby scenarios. It then blends these answers together to create a "soft" target, which is a map showing how likely different choices are. Crucially, the system is smart enough to know how much blending to do. If a question is very objective, like a math problem, it blends very little and sticks close to the single observed answer. But if a question is highly subjective, like asking about personal values, it blends answers from a much wider circle of similar situations to capture the full spread of human opinion. This allows the model to learn the true shape of human behavior rather than just a single snapshot of it.

To test this idea, the team had to build something that did not exist before: a massive dataset where they could see the full range of human choices, not just one. They recruited 193 people to answer 100 different survey questions covering topics like politics, health, and culture. But instead of just picking one answer, each person was asked to distribute a set of probability points across all the possible options, effectively telling the researchers how likely they thought each choice was. This created a rich map of 19,300 unique scenarios, showing exactly how people's opinions were distributed. Using this new benchmark, they trained their models. The results were striking. The standard method, which tries to copy a single answer, performed poorly, often failing to capture the true diversity of human thought. The new SALT method, however, significantly outperformed every other approach. It reduced the error in predicting human behavior by large margins, particularly for the most subjective questions where human opinion is most varied. In the most difficult cases, where people's answers were widely scattered, the new method improved the prediction of the full response distribution by nearly 97 percent compared to the old way of training, as measured by a reduction in KL divergence.

The study also revealed why the old methods were struggling. When researchers tried to use techniques that simply grouped people by broad categories, like their age or job, the results were still noisy and inaccurate. It turned out that the key was not just grouping people, but finding the right amount of similarity to look for. The adaptive nature of SALT allowed it to find this balance automatically, pulling in just enough similar examples to build a reliable picture without mixing in unrelated opinions. The researchers found that this approach works best when the model is allowed to be uncertain, mirroring the way humans actually are. The work suggests that to truly simulate society, we must stop treating human choices as errors to be corrected and start treating them as data to be understood in their full complexity. By shifting from a rigid search for a single correct answer to a flexible understanding of a range of possibilities, we can build simulations that are not just statistically accurate, but genuinely faithful to the messy, wonderful unpredictability of human life.

Technical Summary: Rethinking the Evaluation and Optimization of LLM-Based Social Simulation

1. Problem Statement

Large Language Model (LLM)-based social simulation offers a scalable alternative to traditional social science methods like surveys and behavioral experiments. However, a fundamental challenge arises in evaluating and optimizing these models: the inherent subjectivity of human behavior.

Current prevailing practices treat social simulation as an objective task (similar to coding or math), relying on accuracy-based evaluation (checking if the model selects the single observed human response) and hard-label training (fine-tuning the model to reproduce that single response). The paper argues this approach is fundamentally flawed for social simulation because:

Stochasticity: Human behavior is stochastic, not deterministic. The same individual in the same situation may act differently on different occasions.
Distributional Nature: An observed response is merely one draw from an underlying response distribution, not the ground truth itself.
Misalignment: Training on a single hard label collapses the model's output distribution to a point mass, while accuracy-based evaluation fails to capture the fidelity of the model's full probability distribution. Consequently, a model with high accuracy may still poorly represent the true diversity of human behavior, and a perfect model might be penalized if its most probable action differs from the single observed draw.

2. Methodology

2.1 The Subjectivity Coefficient

To rigorously analyze the failure of current methods, the authors introduce the subjectivity coefficient ( $s(x)$ ), an entropy-based metric quantifying the inherent randomness of human responses in a specific decision context $x$ .

Definition: $s(x) = \frac{H(\phi^*(\cdot|x))}{\log K}$ , where $H$ is Shannon entropy, $\phi^*$ is the true latent response distribution, and $K$ is the number of actions.
Scale: $s(x) \in [0, 1]$ . Values near 0 indicate objective tasks (deterministic behavior), while values near 1 indicate highly subjective tasks (uniform distribution).
Theoretical Insight: The authors prove that as $s(x)$ increases, the error of single-label training (point-mass solutions) grows, bounded below by the subjectivity level. Similarly, accuracy becomes an unreliable metric because the gap between a model's top action and the single observed draw increases with subjectivity.

2.2 Subjectivity-Adaptive Soft-Label Training (SALT)

To address these issues, the paper proposes SALT, a training framework that replaces hard labels with soft distributional labels aggregated from semantically similar contexts.

Mechanism:
1. Grouping: Contexts are partitioned into groups sharing the same candidate action set.
2. Neighborhood Construction: For a given context $x$ , a neighborhood $N(x)$ is formed by retrieving semantically similar contexts within a radius $r(x)$ .
3. Soft Label Generation: The observed actions of neighbors are pooled to create an empirical distribution $\hat{\phi}(\cdot|x)$ .
4. Adaptive Radius: The core innovation is that the radius $r(x)$ $r (x)$ is adaptive based on the estimated subjectivity of the context.
  - High Subjectivity: The model estimates a high effective number of actions ( $K^*_{eff}$ ), triggering a larger radius to pool more neighbors and reduce statistical error.
  - Low Subjectivity: The radius shrinks, causing SALT to naturally fall back to standard single-label training, avoiding unnecessary smoothing of deterministic targets.
Theoretical Basis: The radius is derived from a bias-variance tradeoff analysis (Theorem 1), balancing the bias introduced by merging different distributions against the statistical error of estimating a distribution from few samples.

2.3 The SubjSim Benchmark

Since existing datasets only record single responses, they cannot support distributional evaluation. The authors constructed SubjSim, a new benchmark:

Scale: 19,300 context pairs involving 193 annotators and 100 subjective survey questions across 8 domains (e.g., economy, politics, health).
Data Collection: Annotators provided response-propensity distributions by allocating probability balls across options, creating ground-truth distributions for evaluation.
Training Constraint: Crucially, during training, models only see the single "hard" action (the option with the most balls), simulating real-world data scarcity. Evaluation is performed against the full distribution.

3. Key Contributions

Problem Identification: The paper identifies and rigorously analyzes why accuracy-based evaluation and hard-label training fail for social simulation as subjectivity increases, introducing the subjectivity coefficient to formalize this gap.
Methodological Innovation: It proposes SALT, a training method that adaptively aggregates observations from semantically similar contexts into soft labels, dynamically adjusting the aggregation radius based on the estimated subjectivity of each input.
Data Contribution: It introduces SubjSim, a benchmark with human-annotated response distributions, enabling the first rigorous evaluation of distributional fidelity in LLM-based social simulation.

4. Experimental Results

Experiments were conducted using Qwen3-8B as the backbone, comparing SALT against zero-shot, SFT, DPO, PPO, and DSA (a distribution-level baseline).

Performance: SALT significantly outperformed all baselines across four divergence metrics (KL, JSD, TVD, MMD).
- Aggregate Improvements: Reduced KL divergence by 77.6%, JSD by 45.9%, TVD by 31.2%, and MMD by 52.7% compared to standard SFT.
Subjectivity Stratification:
- In low-subjectivity regimes, SALT performed comparably to SFT (sometimes slightly lower on specific metrics), confirming that hard-label training remains effective for deterministic tasks.
- In mid- and high-subjectivity regimes, SALT showed massive gains (e.g., 96.9% reduction in KL for high-subjectivity questions), demonstrating that adaptive aggregation is critical when single observations are poor proxies for the true distribution.
Ablation Studies:
- Fixed-radius or fixed-neighbor-count methods performed worse than the adaptive approach, confirming the necessity of calibrating the neighborhood size to the context's subjectivity.
- Global frequency pooling (ignoring semantic similarity) performed poorly, highlighting the need for context-specific aggregation.

5. Significance and Claims

The paper claims that its work fundamentally shifts the paradigm for LLM-based social simulation from reproducing a single observed outcome to capturing the underlying distribution of human behavior.

Theoretical Validity: It provides a formal justification for why standard NLP metrics (accuracy) are insufficient for social science tasks and offers a theoretical framework (subjectivity coefficient) to distinguish between objective and subjective domains.
Practical Feasibility: It demonstrates that high-fidelity distributional modeling is achievable even when training data is limited to single observations per context, provided the model leverages semantic similarity to infer distributional structure.
Future Direction: The authors position this as a foundational step, noting that future work could extend this framework to sequential, multi-agent, and open-ended simulation settings.

The paper maintains a modest tone, acknowledging that while SALT significantly improves distributional alignment, it relies on the assumption of semantic smoothness (Lipschitz continuity) and that the effectiveness of the method is contingent on the quality of the embedding space and the estimation of subjectivity.