QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Models

1. Problem Statement

The increasing use of Large Language Models (LLMs) for generating synthetic survey responses and data annotation has raised significant concerns regarding robustness and reliability. Previous research indicates that LLM responses to questionnaire-style prompts are highly sensitive to:

• Prompt Perturbations: Small changes in formatting, question order, or answer option ordering can drastically alter outputs (e.g., recency bias, token bias, or "A-bias").
• Presentation Modes: The way questions are presented (sequentially vs. all at once) affects alignment with human data.
• Generation Methods: Different techniques for extracting closed-ended answers (e.g., token probabilities vs. open-ended generation) yield varying levels of accuracy and computational efficiency.

Currently, there is a lack of standardized, modular tools that allow researchers to systematically control these variables, run large-scale perturbation experiments, and parse results automatically without extensive custom coding.

2. Methodology: The QSTN Framework

The authors introduce QSTN (pronounced "Question"), an open-source Python framework designed to facilitate robust, reproducible, and scalable questionnaire inference with LLMs.

Core Architecture

QSTN provides a fully modular pipeline where every component can be swapped independently. It supports both local inference (via vllm) and remote inference (via OpenAI API).

Key Modules:

1. Questionnaire Presentation:
   • Sequential: Questions asked one by one in a continuous conversation context.
   • Battery: All questions presented in a single prompt; the model answers all in one response.
   • Single-item: Each question is asked in a new, isolated context (no memory of previous questions).
2. Prompt Perturbation:
   • Structural: Randomization of question order and answer option order.
   • Semantic: Synonym replacement and full paraphrasing.
   • Typographical: Introduction of key typos, letter swaps, or keyboard-adjacent errors.
   • Scale/Option Manipulation: Reversing scales (e.g., 1–5 to 5–1), removing "Don't know" options, or transforming even/odd Likert scales.
3. Response Generation Methods:
   • Token Probability-Based: Extracting probabilities directly from the model's output logits.
   • Restricted Generation: Forcing the model to output specific tokens or JSON structures (e.g., {"answer": "A"}).
   • Open Generation: Generating free text followed by a classification step (LLM-as-a-judge).
   • Verbalized Distribution: Asking the model to output a probability distribution over options (e.g., {"A": 0.4, "B": 0.1}).
4. Parsing & Evaluation:
   • Automatic parsing of JSON, raw text, and structured outputs.
   • Integration of "LLM-as-a-judge" for parsing unstructured responses.

User Accessibility

• Code-based: A lightweight Python package requiring only three function calls for basic usage.
• No-Code Interface: A live GUI (Gradio) allowing researchers to upload questionnaires, configure perturbations, and run inference without writing Python code.

3. Key Contributions

• Systematic Framework: The first open-source tool to unify questionnaire presentation, prompt perturbation, and response generation into a single, modular pipeline.
• Scalability: Capable of handling massive-scale experiments (the authors evaluated >40 million survey responses).
• Accessibility: Democratizes robust LLM experimentation via a no-code GUI and simplified API.
• Empirical Validation: Provides extensive empirical evidence on how specific design choices impact model alignment with human data.

4. Key Results

The authors conducted extensive experiments using 10 different instruction-tuned models (including Llama 3, Qwen, Gemma, Phi, and Gemini) across various datasets (ANES, GLES, ATP).

A. Questionnaire Presentation

• Alignment: The Battery presentation (all questions in one prompt) significantly improved subpopulation distributional alignment (measured by Wasserstein distance) compared to Single-item presentation, with an approximate 8% improvement for large models.
• Efficiency: The Battery method drastically reduced computational costs. For a 70B model, it required 1 API call vs. 16 for Sequential, reducing inference time from ~9 hours to ~1.5 hours and significantly lowering token usage.
• Individual Accuracy: While presentation mode had a marginal effect on individual-level accuracy (Mean Absolute Error), it had a profound effect on the distribution of answers across demographic groups.

B. Prompt Perturbation

• Sensitivity: Models exhibited significant non-robustness to perturbations.
  • Recency Bias: Models favored the last option in a list. Moving an option to the end increased selection frequency by >20x for some models (e.g., Llama-3.1-8B).
  • Scale Effects: Models showed a strong bias toward middle categories in Likert scales (67% more frequent for 5-point, 89% for 11-point).
  • Robustness Hierarchy: Larger models (e.g., Llama-3.3-70B, Gemini-1.5-Pro) were more robust to perturbations than smaller models.
• Impact: Reversing answer options and introducing typos had the most detrimental impact on robustness.

C. Response Generation Methods

• Best Performer: The Verbalized Distribution Method (asking the model to output a probability distribution) yielded significant improvements in subpopulation alignment across most datasets compared to standard restricted choices.
• Worst Performer: Token Probability-Based Methods (using raw logits) generated misaligned survey responses and are not recommended.
• Efficiency: Restricted Generation Methods (forcing specific JSON outputs) offered the best balance of accuracy and computational efficiency, outperforming Open Generation methods.

5. Significance and Conclusion

• Reproducibility: QSTN addresses the "replication crisis" in LLM-based survey research by standardizing how experiments are constructed and evaluated.
• Cost Reduction: By optimizing presentation modes (Battery) and generation methods (Restricted/Verbalized), researchers can achieve better alignment with human data at a fraction of the compute cost.
• Reliability: The framework emphasizes that "one prompt does not fit all." It encourages researchers to run robustness checks (perturbations) to ensure findings are not artifacts of specific prompt formatting.
• Future Impact: The authors hope QSTN will become a standard tool for synthetic data generation, persona studies, and the analysis of LLM behavior, moving the field toward more rigorous and reliable in-silico surveys.

Availability:

• Code: MIT License on GitHub (dess-mannheim/QSTN).
• GUI: Available on Hugging Face Spaces.
• Documentation: Includes video tutorials and extensive API documentation.