Semantic Variability of Replies Across LLMs: Implications for Designing Conversation-Based Assessment

Technical Summary: Semantic Variability of LLM-Generated Replies Across LLMs

Problem Statement
The integration of Large Language Models (LLMs) into conversation-based assessments introduces a fundamental tension between adaptivity and standardization. While LLMs enable natural, dynamic interactions necessary for measuring complex skills like communication and collaboration, their generative nature creates variability. Different learners may experience divergent interaction pathways, and identical learner inputs may yield semantically distinct replies depending on the specific LLM model, prompt configuration, deployment setting, or prior chat history.

The core measurement challenge is not the existence of variability itself, but whether this variability remains within acceptable bounds to preserve the validity, reliability, and fairness of the assessment. If uncontrolled, semantic inconsistency across model versions or conversational contexts can introduce construct-irrelevant variance, threatening the comparability of assessment conditions over time, especially given the rapid evolution and frequent retirement of LLMs.

Methodology
This study empirically examined the semantic variability of LLM-generated replies across different models and conversational context conditions using data from real-world dyadic collaborative problem-solving tasks.

• Data Source: The study utilized 61 "focal messages" selected from 99 teams of human participants engaged in online science tasks. These messages were chosen from the late stage of conversations where human replies were coded as highly relevant, ensuring rich prior context and interpretable responses.
• Experimental Conditions: Four LLMs were evaluated: GPT-4o mini, GPT-5.4, GPT-5.4 mini, and GPT-5.4 nano. Each model was tested under two prompting conditions:
  1. No History: The prompt contained only the system instruction and the focal message.
  2. With History: The prompt included the system instruction, the focal message, and all preceding conversational turns.
• Generation Protocol: For each focal message and condition, every LLM generated 100 independent responses. Temperature was set to 0.7 where supported (specifically for GPT-4o mini) to simulate natural response variability.
• Semantic Similarity Metrics: Semantic similarity was quantified using cosine similarity of text embeddings generated by OpenAI's text-embedding-3-large model. The study analyzed:
  • Within-Model Similarity: The mean pairwise similarity among the 100 responses generated by a single model.
  • Cross-Model Similarity: The mean similarity between responses generated by different models.
  • Human Alignment: The similarity between LLM responses and the original human replies.
  • Context Sensitivity: The similarity between responses generated with and without chat history for the same focal message.
• Statistical Analysis: Linear mixed-effects models were fitted to estimate the effects of LLM type and chat history condition on two outcomes: the mean pairwise similarity (semantic consistency) and the standard deviation of pairwise similarity (semantic variability).

Key Results

• Model Dependency: Response consistency is heavily influenced by model architecture and lineage. Responses generated by the same LLM were consistently more similar to each other (within-LLM similarity ranging from 0.715 to 0.795) than to responses generated by different LLMs (between-LLM similarity ranging from 0.443 to 0.604). Models within the same family (e.g., GPT-5.4 variants) showed higher cross-model similarity than models across different generations (e.g., GPT-4o mini vs. GPT-5.4).
• Impact of Chat History: Including conversational history did not universally increase response similarity.
  • In the No History condition, there was a strong positive correlation between the semantic similarity of focal messages and the similarity of the generated replies, indicating responses were anchored to the immediate input.
  • In the With History condition, this correlation weakened, suggesting that chat history introduced contextual information that made replies less dependent on the focal message alone.
  • Statistically, the "With History" condition generally increased the variability (standard deviation) of responses for the reference model (GPT-4o mini). While the interaction terms for GPT-5.4 and GPT-5.4 mini were positive and significant (indicating they were less negatively affected by history than the reference model), GPT-5.4 mini showed a negative main effect coefficient (-0.0014, p=0.924) and a significant positive interaction (0.0505, p=0.017), suggesting its response to history inclusion differs from the reference model but does not uniformly imply higher mean similarity across all conditions.
• Human Alignment: LLM-generated replies were consistently less similar to human replies than they were to each other within the same model. While including chat history slightly improved alignment with human responses, the gap remained significant.
• Cross-History Variability: The semantic similarity between responses generated with history versus those generated without history was moderate (median ~0.40–0.45), indicating that adding context meaningfully alters the semantic content of the output.
• Variation by Model: GPT-5.4 showed significantly higher variation in pairwise similarity compared to the reference model (GPT-4o mini), whereas GPT-5.4 nano showed significantly lower variation.

Significance and Claims
The paper concludes that prompting strategies and conversational context alone are insufficient to ensure semantically consistent responses when the underlying LLM changes. The findings highlight that model choice is a critical variable in assessment design.

The author argues that as LLMs continue to evolve through updates and new releases, maintaining stable assessment conditions becomes an infrastructure challenge rather than merely a prompt-engineering problem. To preserve the validity and fairness of conversation-based assessments, systems require additional mechanisms—such as symbolic rules, response templates, or validation layers—to monitor, benchmark, and control response behavior across model transitions. These strategies are essential to prevent construct-irrelevant variance and ensure that assessment conditions remain comparable over time.