From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs

1. Problem Statement

Current Large Language Model (LLM) evaluation relies heavily on standardized benchmarks (e.g., MMLU, HumanEval) that report aggregated scores on specific tasks. The authors argue that these scores often fail to capture real-world usefulness, which is highly context-dependent. Users frequently rely on "vibe-testing": an informal, ad-hoc practice where individuals evaluate models through direct interaction using their own prompts, workflows, and subjective criteria (e.g., "Does this feel right for my coding style?").

While prevalent, vibe-testing is currently:

• Unstructured: It lacks a formal framework for analysis.
• Non-reproducible: Insights are scattered across blogs and social media.
• Subjective: Different users prioritize different dimensions (e.g., clarity vs. conciseness).

The core problem is the gap between benchmark performance and user experience, and the lack of a systematic method to bridge this gap by formalizing the vibe-testing process.

2. Methodology

The authors propose a three-stage approach to study, formalize, and automate vibe-testing.

A. Empirical Grounding

To understand how vibe-testing works in practice, the authors collected two empirical resources:

1. User Survey: A survey of 51 technical users (AI/ML experts and practitioners) regarding their evaluation practices.
   • Findings: 82% of respondents vibe-test models. They prioritize workflow fit, clarity, and style over raw benchmark scores. 86% reported a mismatch between benchmark rankings and their real-world experience.
2. "In-the-Wild" Corpus: A curated collection of 40 public model comparison reports from blogs, YouTube, and Reddit.
   • Process: The authors used LLMs to extract specific "vibe-test instances" (a specific task + subjective judgment) and manually verified them. They then annotated these instances to identify recurring patterns.

B. Formalization: The Vibe-Testing Framework

Based on the empirical data, the authors formalize vibe-testing as a two-part process defined by Vibe Dimensions:

1. Input Dimensions (What is tested): How users construct prompts.
   • Examples: Task type, complexity, real-world context, persona-based framing, constraint tightness, and reference material availability.
2. Output Dimensions (How responses are judged): The subjective criteria used for evaluation.
   • Examples: Clarity, cognitive load, style/tone fit, workflow fit, ambiguity handling, reliability, and anthropomorphism.

Definition: Vibe-testing is an interaction-based evaluation where evaluators adapt both the input (via input dimensions) and the judgment criteria (via output dimensions) to capture practical utility.

C. Automated Evaluation Pipeline (Proof-of-Concept)

The authors built a modular pipeline to automate this process:

1. User Profiling: Converts a natural language user description (e.g., "I am a novice Python student") into a structured profile $P$ containing preferred input dimensions ($P_{in}$) and weighted output dimensions ($P_{out}$).
2. Personalized Prompt Generation: Rewrites standard benchmark prompts (e.g., from MBPP+ or HumanEval+) to align with $P_{in}$. This involves adding context, constraints, and persona-specific framing while preserving the core task intent (verified via semantic preservation checks).
3. User-Aware Comparison: Compares model outputs head-to-head using an LLM judge that evaluates responses based on $P_{out}$ (the user's specific weights for dimensions like clarity or workflow fit), rather than just correctness.

3. Key Contributions

1. Empirical Analysis: The first systematic study of "vibe-testing" as a user practice, quantifying its prevalence and identifying the specific dimensions users care about.
2. Formal Framework: A taxonomy of Input and Output dimensions that structures informal user evaluation into a reproducible format.
3. Evaluation Pipeline: A proof-of-concept system that generates personalized prompts and performs subjective, persona-conditioned comparisons.
4. Demonstration of Preference Shifts: Empirical evidence showing that personalizing both the task framing and the evaluation criteria can fundamentally change which model is preferred, even reversing the outcomes of standard benchmarks.

4. Experimental Results

The pipeline was evaluated on coding benchmarks (MBPP+ and HumanEval+) using four user personas (Beginner, Intermediate, Researcher, Advanced) and four model pairs (e.g., GPT-5.1 vs. GPT-OSS-20B).

• Preference Reversals: Personalized prompts significantly altered model rankings.
  • Example: On original benchmark prompts, GPT-5.1 was often preferred by advanced users but underperformed for Beginners compared to GPT-OSS-20B. However, when prompts were personalized for Beginners (emphasizing clarity and explanation), GPT-5.1's win rate jumped from ~3% to ~77%.
  • Control: Neutral paraphrases (rewriting prompts without persona context) did not produce these shifts, confirming the effect is due to personalization, not just rephrasing.
• Dimension-Level Insights: The shift was driven by specific dimensions. For beginners, GPT-5.1 excelled in Clarity and Tone/Style when the prompt explicitly requested it, whereas standard prompts masked these advantages by focusing solely on code correctness.
• Robustness: Results held across different benchmarks (HumanEval+), different prompt generators (Qwen3-32B), and different aggregation methods (weighted vs. unweighted).
• Human Validation: A small-scale human study (6 annotators) showed high agreement (89%) between human and LLM judges on original prompts, though agreement dropped on personalized prompts due to the increased complexity of the task.

5. Significance and Implications

• Bridging the Gap: The work provides a methodology to translate subjective "feelings" into structured metrics, offering a path to evaluate models based on user-centric utility rather than just aggregate scores.
• Context Matters: It demonstrates that "best model" is not a static property; it is a function of the user's profile, the task framing, and the evaluation criteria. A model that scores lower on a benchmark might be superior for a specific user workflow.
• Future Evaluation: The authors suggest that future evaluation suites should move beyond static benchmarks toward dynamic, personalized evaluation pipelines that can simulate diverse user needs and preferences.
• Limitations: The current study is limited to single-turn coding tasks and relies on LLMs for both prompt rewriting and judging, which introduces potential biases. Future work aims to extend this to multi-turn interactions and tool-augmented settings.

In conclusion, the paper argues that formalized vibe-testing is a necessary evolution in LLM evaluation to capture the nuances of real-world deployment, moving the field from "what the model knows" to "how the model works for the user."