💬 NLP

From Benchmarks to Skills: Low-Rank Factors for LLM Evaluation

This paper proposes a new evaluation paradigm for large language models that applies Factor Analysis to reveal an intrinsically low-rank structure in benchmark performance, demonstrating that a small number of interpretable latent skill factors capture most capabilities and enabling practical tools for identifying redundant tasks, efficiently profiling new models, and selecting models based on specific skill profiles.

Original authors: Aviya Maimon, Amir DN Cohen, Gal Vishne, Shauli Ravfogel, Reut Tsarfaty

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aviya Maimon, Amir DN Cohen, Gal Vishne, Shauli Ravfogel, Reut Tsarfaty

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "One-Number" Trap

Imagine you are trying to judge the quality of 60 different athletes. Currently, the sports world does this by giving every athlete a single, overall score based on how they perform in 44 different events (like running, swimming, lifting weights, etc.).

If an athlete gets a high score, we assume they are a "great all-arounder." But this is misleading.

The Flaw: A gymnast might be amazing at balance but terrible at sprinting. A sprinter might be the opposite. If you just average their scores, they might end up with the same "overall" number, even though their actual abilities are completely different.
The Reality: Current Large Language Models (LLMs) are being judged the same way. We run them on dozens of tests (benchmarks) and give them one average score. The paper argues this hides the truth: some models are great at math but bad at writing, while others are great at conversation but bad at logic. The single score tells us nothing about what they are actually good at.

The Solution: The "Psychologist's Approach"

The authors propose a new way to look at these models, inspired by how psychologists study human personality.

The Analogy: The Big Five Personality Test
When psychologists study humans, they don't just ask, "Are you a good person?" They look at many specific questions to find hidden "dimensions" of personality, like Extroversion or Conscientiousness. They realized that even though there are thousands of ways to act, most behavior boils down to just a few core traits.

The authors applied this same logic to AI:

The Data: They gathered performance data for 60 different AI models across 44 different tasks (like answering medical questions, summarizing news, or solving math problems).
The Magic Tool (Factor Analysis): They used a statistical method called Factor Analysis. Think of this as a "pattern detector." It looks at the messy data and asks: "Do these 44 tests actually measure 44 different things, or are they just measuring a few hidden skills over and over again?"

The Discovery: The "8 Hidden Skills"

The pattern detector found something surprising. The 44 different tests weren't 44 separate things. They were actually just measuring 8 core "skills" that the models possess.

Instead of a long list of scores, the authors created a new "Skill-Based Leaderboard" that rates models on these 8 hidden dimensions:

General NLU: Basic understanding of everyday language and grammar.
Entailment & Bias: Detecting logical connections and spotting bias.
Long-Document Comprehension: Reading and remembering long texts.
Instruction Following: Doing exactly what you ask it to do.
Domain Knowledge: Expertise in specific fields like medicine or law.
Social & Ethical Judgment: Knowing what is polite, safe, or ethical.
Precision & Fidelity: Being exact with numbers and facts (no hallucinations).
Grad-Level Reasoning: Solving complex, graduate-school-level science and logic problems.

The "Aha!" Moment:
The paper shows that two models can have the exact same "overall" score (like a 1320 vs. 1316 on a leaderboard) but be completely different underneath.

Model A might be a genius at math and science but terrible at writing stories.
Model B might be a great storyteller but bad at math.
The old "average score" would say they are equal. The new "skill profile" shows they are totally different tools for different jobs.

Why This Matters: Three Practical Tools

The authors didn't just stop at finding these skills; they built three tools to help people use this new map:

The "Redundancy Detector":
- The Problem: People keep making new tests, but many of them are just re-testing the same old skills.
- The Tool: This tool checks a new test to see if it's actually teaching us something new or if it's just a "copycat" of an existing skill. If a new test is 90% similar to an old one, it's redundant and maybe not worth the effort.
The "Fast-Track Profiler":
- The Problem: Testing a new AI model on all 44 tasks takes a lot of time and money.
- The Tool: You only need to test the new model on a tiny, smartly chosen handful of tasks (about 12). The system then uses the "skill map" to predict how that model would do on the other 32 tasks. It's like guessing a student's final grade by looking at just a few key homework assignments.
The "Best Fit Finder":
- The Problem: You have a specific job to do (e.g., "I need a model to summarize legal contracts"), but you don't know which model to pick.
- The Tool: Instead of guessing, you tell the system what "skills" your job needs. The system then scans the skill profiles of all available models and picks the one that is strongest in those specific areas, rather than just picking the one with the highest overall average.

The Bottom Line

This paper argues that we need to stop treating AI models like a single number on a leaderboard. Instead, we should treat them like a toolbox. Some tools are hammers, some are screwdrivers, and some are wrenches. By using this "Skill-Based" approach, we can finally see exactly what each model is good at, stop wasting time on duplicate tests, and pick the right tool for the job.

Technical Summary: From Benchmarks to Skills: Low-Rank Factors for LLM Evaluation

Problem Statement

Current Large Language Model (LLM) evaluation relies heavily on aggregating scores across a proliferating collection of benchmarks. However, it remains unclear what these aggregate scores reveal about underlying model capabilities. Individual datasets often combine multiple skills, and it is uncertain whether benchmark performance reflects many independent abilities or a small number of shared dimensions. This opacity makes it difficult to determine if new benchmarks offer novel insights or merely redundant measurements, and hinders the ability to select models based on specific downstream requirements rather than a single global score.

Methodology

The authors propose a data-driven paradigm inspired by psychometric theory, specifically Exploratory Factor Analysis (FA). They treat benchmark tasks as test items and LLMs as subjects to uncover latent structures in performance data.

1. Data Construction

The authors constructed a comprehensive performance matrix ( $\Pi$ ) of size $60 \times 44$ , evaluating 60 diverse instruction-tuned LLMs (spanning 24 families, including open-source and proprietary models) on 44 publicly available tasks. These tasks cover classification and generation across short- and long-context settings, including domains like law, medicine, and general reasoning. Scores were harmonized onto a unified 0–10 scale and standardized (zero mean, unit variance) prior to analysis.

2. Factor Analysis via Principal Axis Factoring (PAF)

Instead of standard Principal Component Analysis (PCA), which maximizes total variance (mixing shared and task-specific noise), the authors employ Principal Axis Factoring (PAF).

Model: $\Pi = \Theta\Lambda^\top + \epsilon$ , where $\Theta$ represents the latent skill scores for models, $\Lambda$ represents task loadings, and $\epsilon$ is task-specific noise.
Objective: PAF isolates shared variance across tasks, discarding unique variance. This is hypothesized to yield factors that generalize better to unseen tasks and are more robust to outliers.
Factor Selection: The number of latent factors ( $C$ ) was determined using Kaiser's $\lambda > 1$ rule and an 85% cumulative variance threshold, resulting in 8 latent dimensions.
Interpretation: Factors were labeled as "skills" based on their highest-loading tasks. An LLM-assisted procedure summarized the semantic commonalities of these tasks to assign descriptive names (e.g., "General NLU," "Grad-Level Reasoning").

3. Validation and Robustness

The study validates the latent space through:

Internal Consistency: Measured via Cronbach's $\alpha$ and McDonald's $\omega$ , both exceeding 0.80 for the identified factors.
Robustness Tests: The 8-factor solution was tested against perturbations including model subsampling (removing 30% of models), task removal (leave-one-task-out), and varying factor dimensionality ( $k=6$ to $10$). The core structure remained stable (low principal angles, high Tucker's congruence coefficients).
Comparison with PCA: The authors demonstrate that PCA yields a less interpretable and less reliable latent representation, with components often dominated by single tasks and showing weak correlation with human preference judgments.

Key Results

1. Discovery of 8 Latent Skills

The analysis revealed that LLM performance is governed by eight distinct, stable dimensions:

General NLU: Everyday syntax, semantics, and sentiment.
Entailment & Bias: Fine-grained inference and bias detection.
Long-Document Comprehension: Retrieval, reading, and summarization of long contexts.
Instruction Following & Generation: Truthful, open-ended generation.
Domain Knowledge QA: Expert factual QA in medical, legal, and scientific domains.
Social & Ethical Judgment: Social reasoning and ethical constraints.
Precision & Fidelity: Token-level exactness and quantitative precision.
Grad-Level Reasoning: Graduate-level scientific and legal reasoning.

2. Divergence from Aggregate Scores

The latent skill space reveals that models with similar aggregate rankings (e.g., Chatbot Arena Elo) can possess markedly different skill profiles. For instance, two models with nearly identical Arena scores (1322 vs. 1316) were shown to have complementary strengths across the latent dimensions, a nuance obscured by single-score leaderboards.

3. Alignment with Human Preferences

Aggregating the 8 latent skill scores correlates strongly with Chatbot Arena Elo ( $\rho = 0.73$ ). However, individual skills show heterogeneous alignment:

High Alignment: General NLU and Domain QA are strongly rewarded.
Low/Inverse Alignment: Precision & Fidelity and Grad-Level Reasoning are under-recognized or penalized in preference-based evaluations, which tend to favor fluency and confidence over rigorous verification.

4. Practical Applications

The low-rank structure enables three specific tools:

Task Novelty Diagnosis: Quantifying whether a new benchmark adds novel signal or is redundant (highly correlated with existing factors).
Efficient Model Profiling: Estimating a new model's full skill profile from a small subset of tasks (approx. 12 tasks) with low reconstruction error.
Model Selection: Predicting performance on a new task for all models by inferring the task's skill requirements from a small subset of pilot models.

Significance and Claims

The paper claims to establish Factor Analysis as a principled, interpretable framework for understanding the latent structure of LLM evaluation data. Its primary contributions are:

Paradigm Shift: Moving from opaque aggregate averages to a transparent, skill-centric evaluation framework.
Redundancy Identification: Demonstrating substantial redundancy across existing benchmarks and explaining why many tasks measure overlapping abilities.
Robustness: Showing that the identified skill space is stable across model generations and task perturbations, supporting its use for generalization.
Practical Utility: Providing concrete tools for developers to assess benchmark novelty, profile models efficiently, and select models for specific use cases without exhaustive evaluation.

The authors conclude that this approach offers a more effective, efficient, and transparent method for benchmarking core LLM capabilities, addressing the limitations of current single-score leaderboards. They release their code, leaderboard, and analytical matrices to facilitate further research.