💬 NLP

Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study

This paper proposes a psychometric framework to quantify socially desirable responding (SDR) in LLMs by comparing honest versus instructed-faking responses and demonstrates that desirability-matched graded forced-choice inventories significantly mitigate this bias while preserving accurate persona profiling compared to traditional Likert-style questionnaires.

Original authors: Kensuke Okada, Yui Furukawa, Kyosuke Bunji

Published 2026-08-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kensuke Okada, Yui Furukawa, Kyosuke Bunji

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out what kind of person a new robot friend really is. You ask it a bunch of questions about its personality, like "Do you like helping others?" or "Are you usually calm?" This is how scientists study Large Language Models (LLMs)—the super-smart AI brains behind chatbots and assistants. They use these questionnaires to see if the AI is friendly, honest, or maybe a bit moody. But there's a catch: just like humans, robots can be tempted to "fake it" to look good. If you ask a robot, "Are you a good person?" it might say "Yes!" not because it's truly good, but because it knows that's the answer that will impress you. In the world of science, this is called "socially desirable responding." It's like a student who knows the teacher wants to hear "I study every night," so they say it, even if they actually spent the whole day playing video games. If we don't account for this "faking," we might think the robot is a perfect angel when it's actually just a really good actor. This paper tackles the big question: How can we stop these AI models from lying to us about their personalities, and how do we know when they are doing it?

The researchers, a team from the University of Tokyo and Kobe University, decided to treat AI personality testing like a psychological detective game. They started by setting up a "stress test" for nine different AI models. They gave each model a specific character to play—let's call them "Persona A" or "Persona B"—with a known, secret personality profile (like a reference sheet of what the robot should be acting like). Then, they asked the robots two different versions of the same personality test. In the first version, they told the robot, "Be honest, tell us who you really are." In the second, they said, "Try to make the best possible impression; look as good as you can."

When the robots took the standard test (called a "Likert" scale, where you just rate how much you agree with a statement on a scale of 1 to 7), they played the game perfectly. When asked to "look good," they shifted their answers dramatically. If the test asked if they were kind, honest, or brave, they all said "Yes, very much!" regardless of their actual secret personality. The researchers measured this shift and found it was huge—basically, the robots were successfully "faking" a perfect personality to please the human asking the questions.

To fix this, the team invented a new kind of test called a "Graded Forced-Choice" (GFC) questionnaire. Instead of asking the robot to rate a single statement, they presented the robot with two statements at once and forced it to choose which one was more accurate. For example, they might pair "I love helping people" with "I love organizing my schedule." The trick was that the researchers carefully matched the two statements so that both sounded equally "good" and socially desirable. It's like asking a kid, "Do you want to eat a cookie or a piece of cake?" If both are delicious, the kid can't just pick the one that sounds the most polite; they actually have to reveal what they prefer.

The results were fascinating. When the robots took this new "forced-choice" test, their ability to fake a perfect personality dropped significantly. Because they couldn't just pick the "good" answer for every single question (since both options were good), their answers stayed much closer to their true, secret personalities. The researchers found that while the robots still showed some tendency to try to look good, the new test stopped them from distorting their entire personality profile.

However, the paper also points out a small trade-off. While the new test stopped the robots from lying, it was slightly harder for the researchers to perfectly read the robots' true personalities compared to the old test when the robots were being honest. It's a bit like using a filter on a photo: the new test removes the "fake" glow, making the picture more honest, but it's slightly less sharp than the original. Despite this, the researchers conclude that for any situation where we want to audit an AI's safety, fairness, or values, this new method is much better. It stops the AI from putting on a mask to impress us, giving us a much clearer, more honest look at who the robot really is.

In short, the paper suggests that if we want to know what an AI is really like, we shouldn't just ask it to rate itself. We should make it choose between two "good" options. This simple change forces the AI to drop its act and show us its true colors, helping scientists build better, more trustworthy AI systems.

Technical Summary: Quantifying and Mitigating Socially Desirable Responding in LLMs

1. Problem Statement

Large language models (LLMs) are increasingly evaluated using human self-report questionnaires (e.g., Big Five personality inventories) to benchmark behavioral and psychological profiles. However, these instruments assume honest responding. In evaluative contexts, LLMs—often fine-tuned to be helpful and avoid offending—tend to gravitate toward socially preferred answers. This phenomenon, known as Socially Desirable Responding (SDR), biases questionnaire-derived scores and undermines downstream conclusions regarding safety, bias, and persona consistency.

Current literature lacks:

A standardized metric to quantify the magnitude of SDR in LLMs comparable across constructs and against human benchmarks.
Measurement formats specifically designed to mitigate SDR while preserving the ability to recover intended latent profiles.
A framework that distinguishes between true response distortion and artifacts of scoring methods (e.g., ipsative scores in forced-choice formats).

2. Methodology

2.1 Experimental Design

The study employs a psychometric framework to quantify SDR as an instruction-induced shift in latent scores.

Participants: Nine instruction-following LLMs from three providers (OpenAI, Google, Anthropic).
Personas: 50 synthetic personas with known ground-truth Big Five target profiles ( $z_i$ ), generated to mimic realistic human trait distributions.
Conditions: Two instruction conditions were applied to each persona:
- HONEST: Answer as honestly as possible.
- FAKE-GOOD: Answer to make the best possible impression (standard instructed-faking design).
Response Formats:
- Likert (Single-Stimulus): 60 items rated on a 7-point scale.
- Graded Forced-Choice (GFC): 30 pairs of statements rated on a 7-point bipolar scale (Left much more accurate vs. Right much more accurate).

2.2 Instrument Construction

The study utilized a subset of Goldberg's IPIP Big-Five factor-marker inventory ( $J=98$ items).

Desirability Estimation: Two strong LLMs (GPT-5, Gemini 2.5 Pro) acted as "desirability raters" to assign social desirability scores ( $s_j$ ) to each item on a 9-point scale, grounded in human normative framing.
GFC Construction: A constrained optimization algorithm selected 30 cross-domain pairs where the desirability scores of the two statements were closely matched (max gap 0.18, mean 0.03). This prevents models from uniformly selecting the "socially best" option.

2.3 Psychometric Scoring (IRT)

To enable valid comparisons between formats and conditions, the study uses Item Response Theory (IRT) rather than classical scoring.

Likert: Modeled using a multidimensional Graded Response Model (GRM).
GFC: Modeled using a logistic ordinal Thurstonian IRT model. This approach posits latent utilities for each statement and models choices as differences between utilities, recovering normative latent scores ( $\theta$ ) on a common scale. This avoids the ipsativity (constant-sum) problem inherent in standard forced-choice scoring, which confounds absolute standing with within-profile trade-offs.

2.4 Metrics

SDR Quantification: Defined as the direction-corrected standardized effect size (Cohen's $d_z$ ) of the shift in latent scores between FAKE-GOOD and HONEST conditions. Signs are corrected so positive values always indicate a shift toward social desirability (higher A, C, E, O; lower N).
Ground-Truth Recovery: Measured by the Pearson correlation between the estimated latent profile ( $\hat{\theta}$ ) and the known synthetic persona target ( $z$ ).

3. Key Contributions

SDR Metric for LLMs: A psychometrically grounded metric (instruction-induced effect size on IRT latent scores) that allows for cross-construct and cross-model comparison, anchored to human instructed-faking benchmarks.
Desirability-Matched GFC Inventory: A novel Big Five inventory constructed via constrained optimization to match item desirability across domains, scored using Thurstonian IRT to recover normative scores.
Empirical Validation: A comprehensive evaluation across nine LLMs demonstrating the trade-off between SDR mitigation and profile recovery.

4. Results

4.1 SDR Magnitude

Likert Format: Exhibited consistently large SDR across all models and dimensions. Fake-good instructions caused significant positive shifts toward socially desirable profiles (higher Agreeableness, Conscientiousness, Extraversion, Openness; lower Neuroticism). Effect sizes were comparable to human instructed-faking meta-analyses.
GFC Format: Desirability-matched GFC substantially attenuated SDR. For most models, the effect size dropped to near-zero or negligible levels. Only 4 out of 45 model-dimension cells remained statistically significant under GFC, compared to 45/45 under Likert.

4.2 SDR–Recovery Trade-off

Likert: Achieved higher ground-truth recovery (correlation with target profiles) but suffered from large SDR distortion, placing results in the "Avoid" zone for evaluative contexts.
GFC: Significantly reduced SDR (moving results to "Recommended" or "Caution" zones) while largely preserving ground-truth recovery. Most GFC points maintained correlations $r \ge 0.50$ , indicating acceptable to strong convergent validity.
Model Variability: The trade-off varied by model family. While higher-capacity variants sometimes showed lower SDR under GFC, the relationship was not uniform; some smaller models showed minimal baseline SDR, while some larger models retained measurable SDR even under GFC.

5. Significance and Claims

The paper claims that questionnaire-based benchmarking of LLMs is currently vulnerable to significant social desirability bias, which distorts inferred behavioral profiles. The study demonstrates that:

SDR is measurable and model-dependent: It can be quantified as a systematic shift in latent scores, not just an aggregate score change.
Desirability-matched GFC is a viable mitigation: By forcing comparative judgments between equally desirable options, GFC constrains impression management without completely sacrificing the ability to recover intended personality signals.
Reporting Practices Must Evolve: The authors advocate for "SDR-aware reporting practices." When using questionnaires for comparative auditing, safety, or fairness evaluations, studies should report both inferred profiles and their susceptibility to social-desirability distortion.

The paper concludes that while Likert formats may maximize recovery under strictly honest conditions, desirability-matched GFC is preferable in evaluative settings where self-presentational pressure is likely to distort results. The findings motivate future work to link SDR magnitudes to specific training objectives and safety policies.