Not-quite-human tastes: the stylized omnivorousness of LLM survey surrogates

Technical Summary: Not-quite-human tastes: the stylized omnivorousness of LLM survey surrogates

Problem Statement
Large Language Models (LLMs) are increasingly utilized as "silicon surrogates" for human respondents in survey research, offering a promise of cheap, scalable, and fatigue-free data collection. While prior work has established that LLMs can approximate aggregate public opinion on political and moral issues, their fidelity in the domain of cultural taste remains an open empirical question. Cultural taste is distinct from simple opinion; it is fundamentally relational, structured by complex associations between genres and deeply aligned with social space (class, age, gender, race). The paper investigates whether silicon samples can faithfully reproduce not just the ecological estimates (aggregate liking rates) of cultural consumption, but also the relational structures and multivariate associations that define human taste ecologies. The urgency of this inquiry is driven by the rapid commercial adoption of "synthetic panels" by market research firms and the risk of "synthetic contamination" in standard survey data.

Methodology
The study employs a mixed-methods approach to evaluate the fidelity of silicon samples against a canonical human dataset: the Survey of Public Participation in the Arts (SPPA).

• Data Generation: The authors generated 277,470 silicon surrogates (30 synthetic responses per real respondent) using three distinct LLM providers: OpenAI (GPT-4.0), Anthropic (Claude 3.7 Sonnet), and DeepSeek (R1). These models were prompted to adopt personas matching the demographic characteristics of 9,249 real SPPA respondents and answer binary "like/dislike" questions regarding 17 music genres.
• Ecological Fidelity Analysis: The study compares aggregate "liking" rates between silicon and human samples. It utilizes Jaccard distance to measure the dissimilarity between the set of genres liked by a human respondent and their silicon counterpart. The authors test whether discrepancies are explained by a "WEIRD-bias" (favoring Wealthy, Educated, Industrialized, Rich, Democratic populations) or a "minority penalty."
• Relationality Analysis: To assess the structure of taste, the authors construct 17×17 Cramer's V covariation matrices for both human and silicon samples. This measures the strength of association between pairs of genres (e.g., does liking jazz correlate with liking opera?). The study compares these matrices to determine if silicon samples preserve the complex "relationality" of human taste or if they merely reproduce aggregate statistics without structural coherence.
• Multivariate Association Analysis: Using a mixed-effects meta-regression framework, the authors model the coefficients of the relationship between music tastes and four social dimensions: age, income, gender, and race. They calculate the coefficient bias introduced by silicon samples, comparing the regression slopes derived from synthetic data against those from the original SPPA data.

Key Results
The study finds that while silicon samples can produce surface-level approximations of human taste, they are "highly stylized facsimiles" that fail to capture the underlying logic of cultural consumption.

1. Systematic Positive Bias (Hyper-Omnivorousness): Silicon samples exhibit a pervasive positive bias, "over-liking" genres significantly more than human respondents. This results in a "hyper-omnivorous" profile, particularly for non-popular genres. For instance, OpenAI surrogates inflated the liking of folk music by +63.61 percentage points. This bias is not explained by a minority penalty; in fact, silicon samples were less faithful for wealthier, more educated, and white respondents, contradicting the hypothesis that LLMs are simply biased toward underrepresented groups.
2. Loss of Relational Structure: The complex relationality of human taste is "completely lost" in silicon samples. While the average strength of association (mean Cramer's V) appeared similar between human and silicon data, the specific pairings were uncorrelated (Pearson's r = 0.06). Silicon samples attenuated strong human associations (e.g., the link between blues and classic rock) and manufactured spurious associations between unrelated genres (e.g., hip-hop and bluegrass). The relational fidelity was conditional on popularity, with non-popular genres showing the greatest structural divergence.
3. Distorted Social Alignments: Silicon samples fail to preserve known cultural alignments between taste and social space:
   • Age: Silicon samples over-associate tastes with youth, amplifying negative age coefficients for genres like alternative rock and pop, while attenuating the "elder-coded" nature of classic rock.
   • Class (Income): Silicon samples resurrect "anachronistic" class hierarchies reminiscent of the early 20th century, imposing strong positive income gradients on "highbrow" genres (classical, jazz, opera) where human data shows null or weak associations.
   • Gender: Silicon samples "caricaturize" gender associations, exaggerating them for genres like Broadway and pop while erasing or reversing non-stereotypical associations (e.g., reversing the female preference for country music).
   • Race: Silicon samples "invent" fictive race-taste associations, exaggerating racialized preferences for genres like reggae and hip-hop, and fabricating racial associations where none exist in human data (e.g., assigning a non-white bias to electronic music).

Significance and Claims
The paper claims that silicon sampling, in its current state, is not a faithful substitute for human respondents in the study of cultural taste. The authors argue that the "plausibility" of silicon samples is deceptive; they reproduce summary statistics (aggregate liking) while failing to capture the organization of taste that gives it sociological meaning.

The significance of these findings lies in two areas:

1. Methodological Warning: Researchers utilizing silicon samples for cultural sociology risk drawing fundamentally incorrect inferences about the structure of taste and its relationship to social stratification. The "relational" and "multivariate" failures mean that silicon data cannot be used to study the ecology of taste, the evolution of genre boundaries, or the social location of cultural preferences.
2. Theoretical Insight: The study highlights that LLMs, despite their training on vast human text, do not possess a latent "social space" that mirrors human cultural reality. Instead, they produce "stylized" outputs that reflect algorithmic artifacts (such as a tendency toward positive affirmation and a reconstruction of outdated class hierarchies) rather than the nuanced, contingent, and often contradictory nature of human cultural consumption.

The authors conclude that while silicon sampling may offer utility for specific, coarse-grained tasks, its application to cultural taste is currently limited by systematic biases that render it "not-quite-human."