Technical Summary: AIn't Nothing But a Survey? Using Large Language Models for Coding German Open-Ended Survey Responses on Survey Motivation

Problem Statement

The classification of open-ended survey responses is a resource-intensive task traditionally requiring manual human coding or supervised machine learning (ML) models that demand extensive training data, expertise, and preprocessing. While Large Language Models (LLMs) offer promising capabilities for text classification due to their linguistic adaptability and ability to handle short, context-poor inputs, existing research on their application to survey data is limited. Most studies focus on English-language responses, non-complex topics, or single model families. Furthermore, concerns regarding reproducibility, reliability, and the generalizability of findings to non-English languages (specifically German) and complex, multi-category coding schemes remain unresolved. This study addresses the gap in understanding the conditions under which LLMs can be efficiently and accurately leveraged for coding open-ended survey responses in German, specifically regarding survey motivation.

Methodology

Data and Coding Scheme

The study utilizes data from the GESIS Panel.pop, a German probability-based mixed-mode panel. The dataset consists of approximately 25,000 open-ended responses to an annual question regarding reasons for survey participation (primary, secondary, and tertiary reasons). For the analysis, a random sample of 5,072 responses was selected. These responses were independently coded by two human survey researchers using a predefined coding scheme comprising 22 categories (including intrinsic, extrinsic, and survey-related reasons, as well as catch-all categories like "No reason" and "Other"). Inter-coder agreement was high (Cohen's kappa = 0.91), with disagreements resolved by a senior expert. These human codes serve as the ground truth for evaluation.

Model Selection

The study compares three state-of-the-art LLM families, selected for their multilingual capabilities:

1. GPT-4o (OpenAI): A proprietary model noted for improvements in non-English languages. It was accessed via AzureOpenAI to ensure data privacy and reproducibility.
2. Llama 3.2-3B-Instruct (Meta): An open-source model, selected as the most capable multilingual option in its suite available in Europe.
3. Mistral-NeMo-Instruct-2407 (Mistral AI): A European open-source model specifically designed for global, multilingual applications, with noted strength in German.

Prompting Approaches

Four distinct prompting strategies were evaluated:

• Zero-shot: The model is given the coding scheme without examples or detailed definitions.
• Zero-shot with Descriptions: The model receives the coding scheme with detailed definitions for each category.
• Few-shot: The model is provided with one example response per category (22 examples total) alongside the coding scheme.
• Fine-tuning: Only applied to GPT-4o due to resource constraints. The model was fine-tuned on 80% of the human-coded data (4,048 responses) and evaluated on the remaining 20% (1,024 responses).

Experimental Configuration

• Temperature: Set to 0 for all models to maximize determinism and reliability.
• Execution: Each response was processed individually with refreshed memory. Queries were run twice per model/approach to assess reliability.
• Evaluation Metrics: The primary metric was the macro F1 score, chosen to account for the imbalanced distribution of categories. Additional metrics included weighted F1, accuracy, and Intraclass Correlation Coefficients (ICC) for reliability.

Key Results

Performance Differences Between Models

• Model Superiority: GPT-4o consistently outperformed the open-source models (Llama and Mistral) across all prompting approaches.
• Absolute Performance: Without fine-tuning, even the best-performing open-source models failed to reach satisfactory performance levels. GPT-4o achieved a macro F1 of approximately 0.70 with few-shot prompting, which was still lower than the performance of supervised methods (e.g., SVMs) reported in previous studies on the same data.
• Fine-tuning Impact: Fine-tuning GPT-4o resulted in a dramatic performance increase, achieving a macro F1 of 0.87. This approach yielded a categorical distribution that perfectly matched the human-coded benchmark, with virtually no missing classifications.

Performance Across Categories

• High Performance: All models performed exceptionally well on frequent, substantive categories such as "incentive," "interest," and "fun" (macro F1 ~0.9).
• Low Performance: Models struggled significantly with non-substantive catch-all categories, specifically "No reason" (e.g., "don't know," "???"), "Non-identifiable/other," and "Other survey characteristics."
• Distributional Shifts: Due to poor performance on specific categories, non-fine-tuned models produced distorted categorical distributions compared to human coders. For instance, open-source models over-assigned "curiosity" and under-assigned "no reason," potentially biasing substantive analyses.

Prompting Approaches

• Zero-shot vs. Few-shot: Few-shot prompting significantly improved performance for open-source models (an increase of ~0.18 in macro F1) but had a marginal effect on GPT-4o.
• Descriptions: Providing detailed category descriptions did not consistently improve performance and, in some cases, degraded it compared to simple zero-shot prompting.
• Reliability: All models exhibited high reliability (ICC > 0.93) across iterations. Notably, Mistral produced identical outputs across iterations when temperature was set to zero, whereas GPT and Llama showed minor variations, suggesting that setting temperature to zero does not always guarantee perfect determinism in all model implementations.

Key Contributions

1. Generalizability Check: The study demonstrates that findings from English-language, non-complex survey tasks do not automatically generalize to German-language, complex, multi-category coding tasks.
2. Model and Strategy Dependency: It establishes that LLM performance is highly conditional on both the specific model architecture and the prompting strategy. No single "off-the-shelf" approach works universally well.
3. Fine-tuning Necessity: The research highlights that for complex, specific domains, fine-tuning on human-coded data is often necessary to achieve performance levels comparable to or exceeding established supervised methods.
4. Distributional Validity: It warns that using non-fine-tuned LLMs can lead to systematic errors in categorical distributions, which may compromise downstream substantive analyses of survey data.

Significance and Claims

The paper concludes that while LLMs hold potential for coding open-ended survey responses, they are not a "one-size-fits-all" solution.

• For Easy Contexts: In simpler contexts (e.g., English, common topics), off-the-shelf LLMs may serve as efficient alternatives to manual coding or traditional ML.
• For Complex Contexts: For specific, complex topics in non-English languages, raw LLM performance is often subpar compared to established methods. Success requires either careful prompt engineering (specifically few-shot prompting) or, most effectively, fine-tuning on human-coded data.
• Human-in-the-Loop: The authors assert that humans must remain in the loop. Fine-tuning requires human-coded benchmark data, and the reliability of non-local models over time is not guaranteed due to potential updates or deprecation.
• Trade-offs: Researchers must weigh the trade-offs between proprietary models (high performance, cost, privacy concerns) and open-source models (privacy, reproducibility, lower performance, high computational requirements).

Ultimately, the study cautions against the uncritical adoption of LLMs for survey coding, emphasizing that rigorous validation against human benchmarks is essential before deployment, particularly when the goal is to preserve the integrity of the data's distribution for substantive analysis.