💬 NLP

STRIVE: Probing Reasoning Limits in Graded Plausibility Generation and Evaluation

The paper introduces STRIVE, an LLM-based framework that automates the generation and evaluation of controlled event sets for psycholinguistic studies by varying plausibility levels, demonstrating that incorporating global reasoning and evaluator-guided refinement significantly improves generation quality and human agreement, though events near plausibility boundaries still require human oversight.

Original authors: Bhiman Kumar Baghel, Anna Chrabaszcz, Tessa Warren, Michael Walsh Dickey, Haley C. Dresang, Xiang Lorraine Li

Published 2026-08-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bhiman Kumar Baghel, Anna Chrabaszcz, Tessa Warren, Michael Walsh Dickey, Haley C. Dresang, Xiang Lorraine Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Science of "Wait, That Doesn't Make Sense"

Imagine your brain is a super-fast movie projector. When you read a sentence like "The chef is chopping vegetables," your brain instantly plays a mental clip of a kitchen. But if you read "The chef is chopping a cloud," your brain hits a sudden, jarring pause. That split-second feeling of "Wait, that doesn't make sense" is a superpower called event knowledge. It's the invisible library of facts we all carry about who usually does what, where, and with what tools.

Scientists who study how we learn and use language (called psycholinguists) love to poke at this library. They want to know exactly how our brains decide if a situation is "real" or "nonsense." To do this, they need to create "trick sentences." They need sentences that are almost right, but just a little bit off, to see how long it takes for our brains to catch the error. The problem? Making these trick sentences by hand is like trying to build a perfect set of Lego bricks where every single piece is slightly different, but the rest of the tower stays exactly the same. It takes forever, and humans get tired and make mistakes. This paper introduces a new way to build these mental traps using Artificial Intelligence, turning a slow, manual job into a fast, automated factory.

Meet STRIVE: The AI That Builds "Almost-Right" Stories

The researchers behind this study, led by Bhiman Kumar Baghel and Xiang Lorraine Li, built a framework they call STRIVE. Think of STRIVE as a very strict, very creative chef who is tasked with making four versions of the same soup recipe. The recipe (the sentence structure) must stay exactly the same, but the main ingredient (the person doing the action) changes to create four different levels of "yuck."

Here is the menu STRIVE tries to create for a verb like "catching a soccer ball":

The Obvious Chef (Plausible-Easy): A goalie. (Everyone agrees: "Yes, goalies catch balls.")
The Maybe Chef (Plausible-Hard): A referee. (Everyone pauses: "Well, referees are there, and they could catch it, but it's not their main job.")
The "Wait, Maybe?" Chef (Implausible-Hard): A hockey player. (This is the tricky one. They are an athlete, they play a sport, they catch pucks... but they don't belong on a soccer field. It's close enough to make you hesitate.)
The Absurd Chef (Implausible-Easy): A ballerina. (Everyone laughs: "No way. Ballet has nothing to do with soccer.")

The goal is to generate these four sentences automatically, ensuring that only the "chef" changes while the rest of the scene (the ball, the field, the hands) stays frozen.

The "Reasoning Scratchpad" Secret Sauce

When the team first asked a powerful AI (GPT-5.1) to just "make these sentences," it was a disaster. The AI would mix up the categories or pick chefs that looked too similar (like a glazier and a home renovation contractor, who both wear generic work clothes). It got the job right only 16.7% of the time.

The paper found that the AI needed a "thinking space" before it started cooking. They gave the AI a reasoning scratchpad—a digital notepad where it had to write down its thought process step-by-step before writing the final sentences. It had to ask itself: "Is this scene realistic? Does this chef actually wear a uniform I can recognize? Is this chef too similar to the others?"

When they added this "think before you speak" step, the success rate jumped to 28.3%. But they didn't stop there. They added a second layer: an AI Critic. After the first AI made the soup, a second AI tasted it and said, "Hey, the hockey player isn't quite right; he belongs in a different category." The first AI then had to go back and fix the recipe.

With this "Generate → Critique → Fix" loop, the success rate skyrocketed to 75.0%. It turns out that for AI, just like for humans, doing your homework and getting a second opinion makes a huge difference.

The "Hard" Part is Still Hard

Even with the best AI, the paper found a stubborn limit. The hardest sentences to get right are the ones in the middle—the "Implausible-Hard" ones (like the hockey player). These are the sentences that sit right on the edge of the cliff between "makes sense" and "doesn't make sense."

The study showed that even the smartest AI evaluators only got these tricky middle cases right 57% of the time. This is a big deal because it means that while AI can do the heavy lifting of creating the bulk of these experiments, humans still need to step in to check the most confusing, borderline cases. The AI is great at the easy stuff and the obvious nonsense, but it still stumbles on the "gray area" where human intuition is needed.

What This Means for the Future

This paper doesn't claim to have solved all of language science. Instead, it offers a powerful new tool. By automating the creation of these controlled sentence sets, STRIVE allows researchers to run experiments much faster and with more variety than ever before. It suggests that the future of studying how we think about the world involves a team-up: AI handles the massive generation and initial sorting, while human experts focus their energy on the subtle, difficult edges where the real mysteries of the mind hide.

In short, the paper proves that if you give an AI a clear set of rules, a notepad to think on, and a friend to critique its work, it can build a library of "almost-right" stories that helps us understand how our own brains work. But for the trickiest riddles, we still need to keep our own human brains in the loop.

Technical Summary: STRIVE – Probing Reasoning Limits in Graded Plausibility Generation and Evaluation

1. Problem Statement

Psycholinguistic research on event knowledge relies on controlled stimulus sets where participants judge the plausibility of events (e.g., "who does what to whom"). To isolate plausibility effects, these studies require matched sentence sets where only one event slot (typically the agent) varies across specific plausibility levels while all other features (verb, patient, instrument, location) remain fixed.

Existing methods for constructing such sets are labor-intensive and manual. Furthermore, current automated approaches often fail to generate graded plausibility sets that span a continuous gradient from "clearly plausible" to "clearly implausible" while maintaining strict control over non-target slots. Specifically, generating the intermediate "hard" conditions (e.g., "somewhat plausible" or "somewhat implausible") is difficult because these cases sit near the semantic boundary, requiring nuanced reasoning about thematic fit and domain overlap without collapsing into obvious categories.

2. Methodology: The STRIVE Framework

The authors introduce STRIVE, an LLM-based framework designed to jointly generate and evaluate controlled event-plausibility stimulus sets. The framework operates on a root verb and produces a set of four sentences ( $S = \{s_{PE}, s_{PH}, s_{IH}, s_{IE}\}$ ) corresponding to a 2x2 design:

Plausibility: Plausible vs. Implausible.
Difficulty: Easy vs. Hard.

2.1 Generator Strategies

STRIVE employs a shared event frame where the Agent slot varies, while the verb form, patient, instrument, and location are held constant. The framework tests four generation strategies:

Reasoning Base (RB): A single-pass generation using a structured prompt that decomposes the task into five subtasks: Scene Design, Agent Selection, Visual Distinctiveness checking, Composition, and Validation. Crucially, this includes a global reasoning scratchpad where the model reasons through the entire procedure before outputting the structured JSON.
Reason-to-Verbalize (R2V): Replaces direct agent selection with verbalized sampling. The model generates a pool of candidate agents with explicit probability scores assigned to specific plausibility ranges (e.g., PE: 0.80–1.00, IH: 0.15–0.35) to prevent cross-grade contamination.
Reason-to-Refine (R2R): An iterative approach where an evaluator provides feedback on the initial RB output. The feedback includes specific diagnoses (e.g., "agent belongs in a different condition," "scene does not support the gradient") which the generator uses to revise only the failing components (agents or scene) rather than regenerating everything.
R2VR: Initializes the refinement process (R2R) with an R2V output.

2.2 Evaluator Mechanism

The evaluator is a separate LLM instance that assesses the generated set against three criteria:

Per-condition Adversarial Verdict: For each agent, the evaluator must articulate the strongest argument for why the agent belongs one level higher and one level lower on the gradient. The verdict is "CORRECT" only if the assigned placement is robust against these counter-arguments. For the "Implausible-Hard" (IH) condition, a specific overlap audit is performed, requiring the agent to share at least two of five dimensions (occupational domain, setting, tools, patient type, physical actions) with the Plausible-Easy agent.
Visual Distinctiveness: Agents must be identifiable by profession from a photograph (e.g., via uniform or gear) and visually distinguishable from one another.
Scene Assessment: Evaluates whether the fixed scene (verb, patient, instrument, location) can support the full plausibility gradient or if it requires redesign.

2.3 Experimental Setup

Models: Six models were tested (GPT-5.1, Claude Sonnet 4.6, and four open-source models: Qwen3-Next-80B, Qwen3-30B, Ministral-14B, Qwen3-4B).
Dataset: 60 English root verbs drawn from psycholinguistic assessments.
Evaluation: The authors use the Alternative Annotator Test (AAT) to compare model-human agreement against human-human agreement.

3. Key Results

3.1 Generation Quality

Iterative Refinement is Critical: The most significant finding is that iterative refinement (R2R/R2VR) drastically improves quality. For GPT-5.1, the rate of high-quality (GOLD) sets rose from 28.3% (RB) to 75.0% (R2R).
Role of Reasoning: Removing the global reasoning scratchpad reduced RB performance to 16.7% and R2R to 40.0%, indicating that explicit reasoning traces are essential for handling the complex constraints.
Token Cost: Iterative methods required 2.7–3.5× more output tokens than single-pass methods, but this cost yielded substantial quality gains.

3.2 Evaluator Validation

LLM-Human Agreement: Using the AAT, GPT-5.1 (with high reasoning effort) achieved model-human agreement ( $\kappa = 0.530$ ) that was statistically non-inferior to human-human agreement ( $\kappa = 0.529$ ). This suggests LLMs can substitute for human annotators in the primary classification task.
Boundary Difficulty: The "Implausible-Hard" (IH) condition remains the most difficult. Even the best evaluator achieved only 57% accuracy on this specific condition. Human annotators also showed high disagreement (SD = 1.03) on these items, indicating that the semantic boundary is inherently ambiguous.
Visual Distinctiveness: Validation of visual distinctiveness was unreliable due to low human-human agreement on text-based descriptions. The authors note that image-grounded validation is required for this specific constraint.

3.3 Slot-Dependent Difficulty

Verb Difficulty is Not Inherent: The difficulty of generating a valid set depends heavily on which slot is varied. Verbs that are "Easy" for agent variation (e.g., catch, throw) often fail when the patient slot is varied, and vice versa.
Disjoint Success Sets: Only 3% of verb-model pairs achieved GOLD status in both agent-varying and patient-varying conditions. This implies that comprehensive stimulus coverage requires varying multiple slots, not just the agent.

4. Contributions and Significance

The paper claims the following contributions:

First Joint Framework: STRIVE is the first framework to jointly generate and evaluate matched, slot-controlled stimulus sets across four ordered plausibility conditions.
Scalable Stimulus Construction: It offers a scalable approach to reducing manual effort in psycholinguistic studies by automating the initial generation and evaluation of event sets.
Insight into Reasoning Limits: The study demonstrates that while reasoning efforts (scratchpads, iterative refinement) significantly improve generation, they do not fully resolve the ambiguity near the plausibility boundary (specifically the IH condition). This highlights a persistent limit in current LLM capabilities regarding fine-grained semantic distinctions.
Methodological Rigor: The work provides a rigorous evaluation protocol (AAT) for LLM-as-a-judge in psycholinguistic contexts, showing that specific model families and reasoning configurations are required to match human performance.

5. Limitations and Ethical Considerations

Text-Based Visual Evaluation: The visual distinctiveness check is currently text-based (relying on descriptions of attire) rather than image-based. The authors acknowledge this as a limitation requiring future image-grounded validation.
Occupational Stereotyping: The reliance on professions with distinctive uniforms (to ensure visual identifiability) may encode demographic and gender stereotypes. The authors flag this as a limitation requiring human review before clinical deployment.
English-Only: The framework and experiments are restricted to English verbs and sentences.
Clinical Scope: STRIVE is designed for research and assessment material development, not for autonomous clinical diagnosis. Generated stimuli require review by qualified clinicians before use with patients.

In summary, STRIVE demonstrates that structured reasoning and iterative refinement can automate the creation of high-quality, graded plausibility stimuli, but it also reveals that the "hard" boundaries of human event knowledge remain a challenging frontier for even the most advanced LLMs.