Technical Summary: Uncovering Competency Gaps in Large Language Models and Their Benchmarks

1. Problem Statement

The evaluation of Large Language Models (LLMs) currently relies heavily on standardized benchmarks that report aggregated scores (e.g., accuracy, pass@k). While useful, this aggregation obscures two critical issues:

1. Model Gaps: Specific sub-areas or concepts where models systematically underperform, which are masked by high-level aggregate metrics.
2. Benchmark Gaps: Imbalances in the coverage of concepts within the benchmarks themselves. Existing semantic annotations (e.g., hand-curated topics) are often coarse-grained, manually curated, and difficult to scale, leading to a lack of fine-grained understanding of model capabilities and benchmark distributions.

Current methods often fail to provide a scalable, fine-grained view of how specific concepts, contexts, and reasoning patterns relate to real-world usage, risking the systematic over-testing of certain concepts while overlooking others.

2. Methodology: Competency Gaps (CG)

The authors propose Competency Gaps (CG), an automated, unsupervised method to identify both model and benchmark gaps using Sparse Autoencoders (SAEs).

Core Mechanism

The method leverages SAEs to decompose an LLM's internal representations into sparse, interpretable feature vectors (SAE concept activations).

• Concept Dictionary: Each dimension of the SAE vector corresponds to a distinct concept (e.g., "coding syntax," "polite rejection"). These concepts are assigned textual labels via autointerpretability methods, forming a fixed "concept dictionary" (often containing thousands to millions of concepts).
• Activation Scoring: For a given data point (token sequence $x_i$), the activation of a concept $c$ is calculated by summing the SAE activation scores across all tokens and normalizing by the sequence length to prevent verbosity bias.

Metrics

The method defines two primary metrics grounded in the SAE concept space:

1. Benchmark Gaps (Coverage):

   • Per-Benchmark Coverage ($\chi^{(b,c)}_{bench}$): Measures the activation of concept $c$ in benchmark $b$ relative to the average activation of all concepts in that benchmark.
   • Cross-Benchmark Coverage ($\bar{\chi}^{(c)}_{bench}$): The mean coverage of concept $c$ across a suite of benchmarks.
   • Gap Identification: Concepts with coverage below a threshold (e.g., bottom decile or near-zero) are flagged as underrepresented or missing.

2. Model Gaps (Performance):

   • Per-Benchmark Model Performance ($\chi^{(b,c)}_{model}$): A weighted average of the model's performance on data points where concept $c$ is active.
   • Cross-Benchmark Model Performance ($\bar{\chi}^{(c)}_{model}$): The mean performance across benchmarks for concept $c$.
   • Gap Identification: Concepts where the model performs poorly (low $\bar{\chi}^{(c)}_{model}$) are identified as model gaps.

Workflow

The proposed workflow involves:

1. Selecting a suite of benchmarks.
2. Computing CG scores (coverage and performance) for all concepts in the SAE dictionary.
3. Identifying gaps (low coverage or low performance).
4. Using an LLM to cluster these concepts and interpret the findings (e.g., grouping "time" concepts or "boundary setting" concepts).

3. Key Contributions

• Automated Gap Identification: A systematic, scalable method to uncover both model weaknesses and benchmark coverage gaps without manual annotation.
• Grounded Evaluation: By using model-internal SAE representations, the method ensures that identified gaps reflect the model's actual internal state rather than external, potentially misaligned embeddings.
• Cross-Model and Cross-Benchmark Applicability: The method was demonstrated on five popular open-source models (Gemma2, Llama3.1, Mistral, Qwen3, DeepSeek-R1) and over a dozen diverse benchmarks (including MATH, GSM8K, MMLU, GPQA, and arena-style benchmarks).
• Open-Source Tooling: Release of the method's code and an interactive web application for exploring model capabilities and benchmark balance.

4. Results

The authors applied CG to five models and ten static benchmarks, yielding several key findings:

Benchmark Gaps

• Skewed Representation: Benchmark suites exhibit strong right-skewed coverage distributions. A small number of concepts (e.g., "English Premier League football discussions," "new conversation boundaries") are over-represented, while many others are under-tested.
• Missing Core Concepts: The analysis identified 314 concepts (1% of the dictionary) entirely missing from the benchmark suite. These included critical meta-cognitive concepts (e.g., "assistant explaining limitations," "regulatory compliance") and legal concepts.
• Specific Benchmark Deficiencies: Individual benchmarks were found to miss concepts central to their stated goals. For example, AGI Eval lacked concepts related to "thorough and objective assessment of evidence," and Social IQA lacked concepts regarding "instructions on how someone should behave."

Model Gaps

• Performance Distribution: Models showed high variance in performance across concepts.
• Strengths: Models performed best on STEM tasks (coding, data handling) and helpful behaviors (positive sentiment, commitment to help).
• Weaknesses (Validated): The method recovered known weaknesses, such as poor performance on time-related concepts (e.g., "cooking time durations"), image manipulation instructions, and palindrome reasoning.
• Weaknesses (Novel): The method identified previously unstudied gaps, such as "appeals to intuition in reasoning" and "grokking/deep intuitive understanding."
• Sycophancy vs. Boundaries: A recurring theme was that models performed well on sycophantic/helpful concepts but poorly on concepts requiring polite rejection or maintaining professional boundaries.

Robustness

• SAE Independence: The method yielded qualitatively similar results when using an SAE trained on a different model (e.g., using Gemma's SAE to analyze Llama), suggesting the approach is robust to the specific SAE used, though model-specific SAEs offer higher precision.
• Stability: Scores remained stable under random subsampling of data (low standard deviation) and responded predictably to adversarial perturbations (removing high/low performing data points shifted aggregate scores as expected).

5. Significance and Claims

The paper claims that the Competency Gaps method provides a necessary complement to existing benchmarks by offering a concept-level decomposition of model behavior.

• For Benchmark Developers: The method helps identify "benchmark gaps"—core concepts that should fall within a benchmark's scope but are currently missing. This allows for the iterative improvement of benchmark design to ensure balanced coverage.
• For Model Developers: It surfaces specific "model gaps" (weaknesses) that may be overlooked by aggregate scores, particularly those related to safety, boundaries, and meta-cognition.
• Systemic Insight: The authors highlight an entanglement between benchmark and model gaps: benchmarks that over-represent certain concepts (e.g., instruction following) may inadvertently incentivize models to optimize for those specific concepts while neglecting others (e.g., calibrated refusal), creating a feedback loop of narrow competence.

The authors emphasize that while the method is powerful, it is limited to the SAE concept space; it cannot detect gaps for concepts the model does not internally represent. Furthermore, they note that concept activation does not equate to capability; the method relies on the joint diagnosis of activation and benchmark performance scores.

The work concludes by offering a tool for the community to move beyond aggregated scores toward a more transparent, interpretable, and fine-grained understanding of LLM competencies.