Explain Like I'm 5 or Whatever I Choose: Evaluating the Interactive Potential of Language Model Responses

Technical Summary: Evaluating the Interactive Potential of Language Model Responses

Problem Statement
Current evaluations of Large Language Models (LLMs) in scientific information seeking are increasingly "use-centric," involving live or multi-turn interactions with real users. However, these evaluations predominantly assume a single, static chat interface. As LLMs are integrated into new, malleable interfaces (e.g., reading or writing tools with direct manipulation controls), existing evaluation frameworks fail to account for interface-specific criteria. Specifically, there is a lack of methodology to evaluate whether models can reliably generate multiple distinct responses to a single query that vary along an interpretable axis, such as language complexity, to support user control. While prior work has focused on personalizing a single response to a specific audience, this paper addresses the need to evaluate a model's ability to generate a range of responses that enable users to select their preferred level of complexity dynamically.

Methodology
The authors propose a new evaluation framework grounded in a formative user study and applied to a large-scale model evaluation.

1. Formative User Study (N=16):

   • Design: A within-subjects study comparing a conventional chat interface against an interactive interface featuring a slider to adjust language complexity (Levels 1–5).
   • Task: Participants (primarily STEM researchers) explored unfamiliar STEM topics.
   • Findings: Users valued the agency of interactive complexity but noted that conventional interfaces often provided inconsistent complexity. Participants identified three key drivers of perceived complexity: Jargon (familiarity of words), Information (amount of technical detail), and Length. They also noted that small changes in complexity were often imperceptible, requiring distinct differences between levels.

2. Model Evaluation Framework:

   • Data: The study utilized 98 expert-written scientific queries from ScholarQA-Multi (a subset of ScholarQABench) across five STEM disciplines (Biology, Biophysics, Computer Science, Photonics, Physics). A larger sample of 459 queries generated via a RAG pipeline was also tested to verify robustness.
   • Models: Five recent models were evaluated: GPT-5.1, GPT-5 mini, Claude Sonnet 4.5, Claude Sonnet 4.5 + Thinking, and DeepSeek-V3.1.
   • Generation Protocol: For each query, models were prompted to generate five versions of a response, anchored to specific audiences ranging from "College student" to "Senior researcher" (and alternatively "Child" to "Expert").
   • Complexity Measures: Three metrics were defined based on the user study and prior literature:
     • Jargon: Proportion of words not on the Dale-Chall Word List.
     • Information: Count of independent "atomic facts" identified via an LLM pipeline.
     • Length: Total character count.
   • Performance Criterion: Success is defined not by the quality of a single response, but by the directionality of change between versions. A model performs well if, as the intended complexity level increases, the Jargon, Information, and Length measures consistently increase.

Key Results
The evaluation reveals that current LLMs struggle to reliably generate distinct levels of complexity when prompted to do so:

• Inconsistent Complexity Adjustment: While models vary complexity across responses, the changes are frequently inconsistent. For the best-performing model (Claude Sonnet 4.5), the Jargon measure increased in the correct direction only 46% of the time across the five levels for a single query. The Information measure was even lower at 33%.
• Length vs. Complexity: Models consistently increased Length as complexity levels rose (often >98% of inputs). However, this increase in length did not correlate with increases in Jargon or Information. In many cases, models engaged in "elaborative simplification," adding text that explained concepts in simpler language, thereby increasing length while decreasing technical density.
• Diminishing Returns in Later Levels: Models were significantly better at differentiating the first transition (e.g., College student to Junior Ph.D.) than subsequent transitions. For example, GPT-5 mini correctly increased Jargon for 87.76% of inputs in the 1-to-2 transition, but this dropped to roughly 60% for later transitions.
• Robustness of Findings: These findings held true when increasing the sample size to 459 queries and when altering the audience anchors to a wider range (Child to Expert). While performance metrics improved slightly with wider audience gaps, the models still frequently failed to move complexity measures in the correct direction, often hovering near chance levels.

Significance and Contributions
The paper makes three primary contributions:

1. A New Evaluation Framework: It introduces a method for evaluating LLMs based on their ability to generate multiple, distinct versions of a response across a specific dimension (language complexity), rather than evaluating a single "best" response. This shifts the focus from static personalization to interactive user control.
2. Validated Complexity Measures: It establishes a suite of three measures (Jargon, Information, Length) grounded in user perception and prior work, providing a criterion for evaluating the relationship between response versions.
3. Empirical Evidence of Limitations: The study demonstrates that current state-of-the-art models often fail to reliably adjust complexity measures in the correct direction across multiple versions. This suggests that while models can generate text of varying lengths, they lack the fine-grained control necessary to support direct manipulation interfaces where users expect predictable, monotonic changes in complexity.

The authors conclude that while interactive interfaces offer a promising paradigm for user control, current LLM capabilities are insufficient to support them reliably without significant improvement in generating consistent, multi-level response variations. The findings highlight a gap between the potential of malleable interfaces and the current reality of model behavior.