Technical Summary: Research-Oriented Human-Centric Evaluation for Foundation Models

Problem Statement

Current evaluations of foundation models predominantly rely on objective benchmarks (e.g., MMLU, SWE-bench) that measure knowledge coverage, reasoning accuracy, and computational efficiency. While effective for standardized task accomplishment, these metrics often overlook the subjective human experience inherent in real-world human-AI collaboration. Existing subjective attempts, such as Chatbot Arena, offer valuable insights but frequently suffer from limited human involvement, coarse-grained criteria, and a lack of systematic disentanglement of different perceptual aspects. There is a critical gap in evaluating how users perceive model behavior in complex, research-oriented, multi-modal contexts, specifically regarding problem-solving efficacy, information quality, and interaction fluency.

Methodology

1. The Human-Centric Evaluation (HCE) Framework

The authors propose a structured framework designed to capture user perceptions across three core dimensions, each decomposed into specific sub-dimensions:

• Problem-Solving Ability:
  • Analytical Accuracy: Precision of reasoning and resolution.
  • Comprehensiveness: Extent to which all relevant problem aspects are considered.
  • Assistance Efficiency: Effectiveness in conserving user time and effort.
• Information Quality:
  • Information Reliability: Factual accuracy, consistency, and currency.
  • Exploration Depth: Level of detail, contextual relevance, and thoroughness of analysis.
• Interaction Experience:
  • Content Relevance: Alignment of responses with user queries.
  • Feedback Adaptability: Ability to adjust responses based on user feedback.
  • Expression Naturalness: Coherence, readability, and linguistic fluency.
  • Response Timeliness: Promptness and efficiency of delivery.

2. Experimental Design

• Task Design: The study utilizes open-ended, time-limited collaborative tasks categorized into two types: Literature Synthesis (organizing domain knowledge) and Innovation-Driven Problems (proposing novel hypotheses). Tasks span eight disciplines: Computer Science, Law, Finance, Medicine, Sociology, Environmental Engineering, Biology, and Education.
• Participants & Models: 82 postgraduate students (experienced in using large models) acted as evaluators. They interacted with four advanced foundation models: DeepSeek-R1, OpenAI o3-mini, Grok-3, and Gemini-2.5.
• Procedure: Evaluators engaged in free-form Q&A interactions for 20 minutes to complete a task. Following the interaction, they independently rated the model on a 5-point Likert scale for each of the nine sub-dimensions.
• Scale: The study generated 604 valid independent dialogue and rating instances. Outliers were removed based on Spearman rank correlation coefficients.

3. Validation and LLM-as-a-Judge

• Statistical Validation: The framework's multidimensional structure was validated using Pearson correlation analysis and Confirmatory Factor Analysis (CFA).
• LLM-as-a-Judge Experiment: To test the feasibility of automated subjective evaluation, several advanced LLMs (including GPT-5.2, Claude-3-7-Sonnet, and Qwen3 variants) were tasked with predicting human ratings (categorized as low, medium, or high) based on interaction logs, without access to the actual human scores.

Key Results

1. Framework Validity

• Discriminative Validity: Pearson correlation analysis revealed low-to-moderate positive correlations (0.1–0.4) among the nine sub-dimensions, confirming they capture distinct aspects of performance.
• Construct Validity: CFA demonstrated strong standardized loadings (≥0.5) for sub-dimensions onto their respective higher-order constructs (Problem-Solving, Information Quality, Interaction Experience), validating the framework's structure.
• Efficiency Nuance: A strong correlation was observed between Assistance Efficiency and Response Timeliness, yet they loaded differently on their constructs, suggesting that while related, time-sensitive process characteristics are distinct from outcome-oriented dimensions in human perception.

2. Subjective Evaluation Outcomes

• Balanced Performance: Overall model rankings were driven by balanced performance across all dimensions rather than exceptional strength in a single area. Grok-3 ranked highest (Avg. 4.30) due to consistent performance across all categories.
• Disciplinary Consistency: While absolute scores varied by discipline, the relative ordering of models remained consistent across the eight domains. This suggests that for general-purpose models, disciplinary context modulates the expression of general reasoning and interaction traits rather than reflecting specialized domain competence.
• Failure Cases: The study identified specific limitations, such as models providing incomplete information (e.g., vague coverage of specific time periods) or failing to adapt to user feedback (e.g., providing irrelevant, overly complex answers when a simple solution was requested).

3. LLM-as-a-Judge Limitations

• Low Accuracy: Even the most advanced model, GPT-5.2, achieved only 51.5% accuracy in replicating human ratings. Other models performed significantly worse, with some approaching random chance (~33%).
• Structural Limitations: The results indicate that current LLMs struggle to simulate human subjective judgment, particularly regarding process-oriented factors like interaction efficiency and dynamic responsiveness. Automated models rely on static text features, whereas human evaluation incorporates temporal awareness and the "feeling" of the collaboration.

Key Contributions

1. Framework Proposal: The introduction and validation of a Human-Centric Evaluation (HCE) framework that systematically assesses foundation models in research-oriented scenarios through three multidimensional, fine-grained perspectives.
2. Dataset Construction: The creation of a dataset comprising 604 human-labeled evaluations across multiple disciplines and models, providing a foundation for future automated subjective evaluation research.
3. Empirical Evidence on Automation: The demonstration via LLM-as-a-judge experiments that current foundation models fail to accurately replicate human subjective ratings, underscoring the irreplaceable value of first-person human assessment for nuanced, process-oriented evaluation.

Significance and Claims

The paper claims that the proposed HCE framework addresses a critical gap in the current evaluation landscape by shifting focus from purely objective task completion to the quality of real human interactions. The study argues that:

• Traditional benchmarks are insufficient for assessing the "assistive capability" of models in complex, open-ended research tasks.
• Model performance in human-centric contexts depends on a balance of capabilities (problem-solving, information quality, interaction) rather than isolated strengths.
• Human evaluation remains essential. The inability of even the most sophisticated LLMs to replicate human subjective judgments highlights that automated evaluation cannot yet replace human assessment for capturing the nuances of collaboration, efficiency, and user experience.

The authors position this work as a complement to traditional benchmarks, offering a user-oriented perspective to inform model assessment and development, particularly for applications where human-AI collaboration is central.