Technical Summary: JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory

Problem Statement

Standard Large Language Model (LLM) evaluation practices typically compress diverse capabilities into single aggregate scores or leaderboard rankings. This approach obscures the inherently multidimensional nature of LLM competence. While profiling methods attempt to group benchmarks by human-defined subject categories (e.g., math, law, biology), these categories reflect educational curricula rather than the internal representational structures of models, which are optimized on mixed-domain corpora without explicit subject boundaries.

Furthermore, traditional Item Response Theory (IRT), which models the interaction between a scalar model ability and item difficulty, fails to capture LLM behavior empirically. Two key patterns challenge the traditional scalar view:

1. Lack of Universal Ordering: A single scalar ability score does not induce a consistent global ordering of models across all items; higher-ability models do not necessarily outperform lower-ability models on every question.
2. Smooth Variation: Model behavior varies smoothly across semantically related items, suggesting that ability is not a single point but a function of directional alignment with the question's semantic content.

Methodology: JE-IRT Framework

The authors propose JE-IRT (Joint Embedding Item Response Theory), a geometric framework that embeds both models and questions into a shared high-dimensional space $\mathbb{R}^d$. Unlike traditional IRT, which treats item parameters as content-agnostic latents, JE-IRT ties difficulty and discrimination directly to question semantics.

Geometric Formulation

In JE-IRT, the probability of a correct response is determined by the geometric interaction between a model embedding $E_{M_i}$ and a question embedding $E_{Q_j}$:
$$P(M_i, Q_j) = \sigma \left( \frac{E_{Q_j} \cdot E_{M_i}}{\|E_{Q_j}\|} - \|E_{Q_j}\| \right)$$

Key geometric interpretations include:

• Direction ($E_{Q_j} / \|E_{Q_j}\|$): Encodes the semantic specialization or topic of the question.
• Norm ($\|E_{Q_j}\|$): Encodes the difficulty of the question. A larger norm corresponds to a harder question.
• Projected Ability: The term $\frac{E_{Q_j} \cdot E_{M_i}}{\|E_{Q_j}\|}$ represents the projection of the model's embedding onto the question's direction, effectively measuring how well the model's specific capabilities align with the question's semantic requirements.

Model Architecture and Training

• Question Embeddings: Generated via a two-stage process: a frozen pretrained text encoder (e.g., ModernBERT-Large or Sentence-Transformer) followed by a trainable adapter (a two-layer MLP). This ensures embeddings are derived from content rather than treated as free parameters.
• Model Embeddings: Learned as trainable vectors $E_{M_i}$ from an embedding table, derived solely from observed response patterns (binary correctness) across questions.
• Optimization: The system is trained end-to-end by minimizing binary cross-entropy loss.

Theoretical Properties

The framework satisfies two critical properties:

1. No Global Ability Ordering: The geometry naturally supports specialized abilities where no single model dominates all others, allowing for cross-over in performance based on topic.
2. Stability: Predicted probabilities change smoothly for semantically similar questions (small angular separation), bounded by the norm difference and angular misalignment.

Key Contributions

1. Empirical Refutation of Scalar Ordering: The authors demonstrate that traditional 2-parameter logistic (2PL) IRT assumptions fail on LLM data. Fits often yield negative or near-zero discrimination parameters, and item-characteristic curves are non-monotone, proving that a single scalar ability cannot explain LLM correctness across diverse benchmarks.
2. JE-IRT Framework: Introduction of a geometric IRT formulation where question orientation captures semantics and norm captures difficulty. This formulation outperforms baselines (including KNN and EmbedLLM) in prediction accuracy while using significantly lower embedding dimensions.
3. Scalable Integration: The framework allows for the data-efficient integration of new LLMs. Once question embeddings are learned, a new model can be added by training only its single embedding vector, requiring as little as 10% of the model's data to reach near-joint-training performance.
4. Interpretable Geometry:
   • Difficulty: Embedding norms reliably correlate with question difficulty (higher norms = lower accuracy).
   • Taxonomy: Clustering of question embeddings reveals an LLM-centric taxonomy that only partially aligns with human-defined subjects (e.g., MMLU categories), suggesting models organize knowledge by latent ability requirements rather than curricular labels.
   • Cross-Subject Abilities: Linear probes (difference-of-means) can recover interpretable ability directions, such as an "arithmetic axis," which correctly identifies quantitative reasoning requirements in non-math subjects (e.g., virology, global facts) where raw semantic embeddings fail.

Experimental Results

• Prediction Accuracy: JE-IRT achieves test accuracies of ~75% on the EmbedLLM dataset (112 models, 10 benchmarks), surpassing baselines like EmbedLLM (74.09%) and KNN (71.52%).
• Dimensionality: High performance is achieved with low-dimensional embeddings (peaking around $d=256$), indicating the predictive structure is effectively low-dimensional.
• Out-of-Distribution (OOD) Generalization: Leave-one-out experiments show that benchmarks with high directional alignment to the training pool (e.g., LogiQA, MathQA) suffer smaller performance drops when excluded. Benchmarks with low alignment (e.g., PIQA) show larger drops, validating directional alignment as a diagnostic for generalizability.
• Routing: In a cost-agnostic routing task, JE-IRT improves selection accuracy by ~10 percentage points over a "best-average" baseline and surpasses a "per-benchmark best" baseline, demonstrating the utility of the learned geometry for downstream applications.

Significance and Claims

The paper positions JE-IRT as a unified and interpretable geometric lens for connecting LLM abilities with question structure. Its primary significance lies in:

• Moving Beyond Aggregates: It offers a distinct perspective on model evaluation that moves away from single-number rankings toward a multidimensional view of specialization and difficulty.
• Mechanistic Understanding: The learned geometry provides a compact representation of how capabilities transfer across tasks, revealing that models organize knowledge according to their own representational structures rather than human-defined subjects.
• Practical Utility: The framework supports practical tasks such as routing queries to the most capable model and generalizing to new models with minimal data, addressing the rapid pace of LLM release.

The authors conclude that JE-IRT establishes a foundation for developing more mechanistic accounts of LLM generalization, though they note limitations regarding binary labels and the need for future work on graded scores and multi-objective evaluation signals.