Co-design of LLM-based preference agents: participation may drive overtrust

Technical Summary: Co-design of LLM-based Preference Agents

Problem Statement
Large Language Models (LLMs) are increasingly utilized to simulate human preferences in research and as autonomous agents, raising critical concerns regarding validation, misrepresentation, stereotyping, and the exclusion of genuine human voices. While co-designing agents with the users they represent is proposed as a solution to ensure alignment and trust, there is a risk that participatory processes may mask systematic misalignment. Specifically, the paper investigates whether the transparency and engagement inherent in co-design create an "overtrust engine," where users perceive high fidelity and alignment that does not exist at an independent, aggregate level. This issue is particularly acute in domains like household energy, where agents may influence complex decisions, and where users often lack settled views, making them susceptible to having their preferences shaped rather than merely represented.

Methodology
The study employed a participatory mixed-methods approach involving 12 UK-based participants in the domain of home energy use. The process followed a three-stage workflow:

1. Background Survey: Participants completed a survey capturing demographics, values (Schwartz scale), personality (TIPI), and energy-related behaviors to generate a provisional agent description.
2. Co-design Interview: Using a secure instance of Microsoft 365 Copilot (GPT-5), the researcher generated provisional agent descriptions based on survey data. Participants reviewed, corrected, and refined these descriptions. They then tested the agent against hypothetical energy scenarios. If significant divergence occurred between the participant and the agent, the agent's description was iteratively updated to improve fidelity.
3. Validation Survey: An independent assessment phase where participants and their finalized agents were presented with a new set of energy scenarios. This allowed for a side-by-side comparison of human and agent responses to measure alignment objectively, separate from the participant's subjective perception during the co-design phase.

Data analysis involved thematic analysis of interview transcripts (using NVivo) and qualitative/quantitative comparison of validation survey responses. The study adhered to GRAMMS and COREQ guidelines for mixed-methods and qualitative reporting.

Key Results

• Subjective vs. Objective Alignment: There was a significant gap between perceived and independently assessed alignment. 10 out of 12 participants strongly agreed that their agent represented their preferences well. However, independent validation revealed mixed alignment, with agents frequently diverging from human responses in direction and reasoning.
• Systematic Agent Characteristics: Agent responses were markedly more homogeneous, decisive, and abstract than the human sample.
  • Agents rarely selected "neither" options, whereas humans did.
  • Agents favored "strong" over "somewhat" responses.
  • Agents relied on high-level principles (e.g., transparency, choice), while humans grounded responses in specific personal circumstances (e.g., family needs, specific activities).
• The "Overtrust Engine": The study identified a mechanism where participation drives overtrust. This is fueled by:
  • The Barnum/Forer Effect: Participants accepted general, plausible agent statements as highly personalized due to the co-design process.
  • The IKEA Effect: Participants valued the agent more highly because they helped build it.
  • Social Desirability Bias: Participants often agreed with agents that articulated "better" or more socially acceptable versions of their views, even when it contradicted their initial stance.
  • Limited Testing: The iterative refinement process ensured participants saw aligned outcomes during the interview, obscuring the agent's inability to generalize to unseen scenarios.
• Preference Plasticity: In several instances, participants updated their own views to match the agent's response, suggesting that the agent was not just representing preferences but actively shaping them.

Key Contributions

1. Identification of the "Overtrust Engine": The paper proposes that in participatory preference agent design, the process of co-design itself can act as a mechanism that promotes trust while concealing systematic misalignment. It argues that alignment should be viewed not as a fixed state achieved through co-design, but as an enacted process that can be manipulated by the design interface.
2. Empirical Evidence of Homogenization: The study provides empirical evidence that even when agents are co-designed with specific individuals, they tend to revert to model-default behaviors (homogeneity, decisiveness) that flatten the diversity of human preferences.
3. Reframing Validation: The paper challenges the notion that user-perceived alignment is sufficient for validation. It demonstrates that when the validator is the user themselves, the "ground truth" is unstable due to the plasticity of human preferences and the influence of the agent on those preferences.
4. Methodological Insight: The study highlights the co-design interview as a rich data collection method in its own right, capable of surfacing user priorities and challenging misrepresentations, while simultaneously revealing the limitations of using agents as research proxies.

Significance and Claims
The paper claims that while co-design is valuable for maintaining human involvement and allowing for the challenge of misrepresentation, it does not guarantee algorithmic fidelity. The primary significance lies in the warning that participation and process transparency can act as a veil for structural misalignment.

If deployed at scale, agents that appear to serve individual users well (due to the overtrust engine) could collectively flatten the diversity of preferences expressed to markets, services, and policy-makers. This creates a structural risk where agents, guided by training data and post-training alignment (RLHF) rather than genuine individual nuance, systematically skew outcomes while remaining invisible to the individual user. The paper concludes that for preference agents to be viable, ongoing independent validation and human oversight are prerequisites, rather than relying solely on the initial co-design phase to ensure alignment.