Technical Summary: XCOMPS – A Multilingual Benchmark of Conceptual Minimal Pairs

Problem Statement

While Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding (NLU), a fundamental question remains: do these models genuinely understand semantics, or do they rely on shallow statistical correlations? Specifically, it is unclear whether LLMs possess a universal conceptual reasoning mechanism that transcends linguistic boundaries. Previous work, such as the COMPS dataset, evaluated semantic reasoning using minimal pairs but was limited to English, leaving the question of cross-linguistic generalization open. Furthermore, existing evaluation methods often rely on metalinguistic prompting, which assesses a model's ability to produce correct outputs (performance) rather than its underlying internal representations (competence).

Methodology

Dataset Construction: XCOMPS

The authors introduce XCOMPS, a multilingual extension of the COMPS dataset covering 17 languages (including analytic, inflectional, and agglutinative types).

• Concept Selection: The dataset utilizes 521 concepts and their negative samples from the original COMPS dataset. Negative pairs are categorized into four types based on their relationship to the positive concept:
  1. Taxonomic: Negative concepts share a broad category but differ in key properties (e.g., toaster vs. coffee maker).
  2. Overlap: Negative concepts share semantic properties but lack the specific property under evaluation.
  3. Co-occurrence: Negative concepts appear in similar contexts but do not share the target property.
  4. Random: Negative concepts are entirely unrelated to the target property (e.g., toaster vs. wren).
• Construction Pipeline: To ensure high-quality translations, the authors employed a human-LLM interactive translation pipeline:
  1. Seed Translation: Human experts manually translated concepts and properties from English into German and Chinese to serve as disambiguation seeds (e.g., distinguishing "bat" as sports equipment vs. animal).
  2. Expansion: LLMs (GPT-4o) translated the seed data into the remaining 15 languages.
  3. Human Review: Native experts manually reviewed and corrected translations to ensure cultural and semantic accuracy, particularly for low-resource languages.
  4. Sentence Generation: LLMs generated fluent positive and negative sentence pairs based on the verified concepts and properties.

Evaluation Framework

The paper evaluates LLMs using three distinct methods to distinguish between performance and competence:

1. Metalinguistic Prompting (Meta): The model is explicitly asked to choose the more acceptable sentence in a minimal pair. This measures surface-level performance.
2. Direct Probability Measurement (Direct): The model's log-probabilities for the acceptable vs. unacceptable sentences are compared without explicit prompting. This captures implicit preferences.
3. Neurolinguistic Probing (Neuro): Diagnostic classifiers are trained on the hidden states of the model's internal layers (using last-token pooling) to determine if the model inherently encodes conceptual-property relationships. This directly measures competence.

Experimental Setup

• Model: The experiments utilize Llama-3.1-8B-Instruct.
• Metrics: For prompting and probability, accuracy is measured. For probing, the F1 score (averaged over five cross-validation folds) is used.

Key Results

1. Cross-Linguistic Variability: Conceptual understanding is not consistent across languages. While models perform well on English tasks, their reasoning ability deteriorates significantly in low-resource languages. The degree of deterioration varies even among low-resource languages, suggesting that semantic reasoning is neither universally stable nor equally supported by training corpora.
2. Distinct vs. Subtle Contrasts: Models excel at distinguishing concept-property pairs that are visibly different (e.g., Random pairs like "toaster" vs. "wren"). However, performance drops markedly when negative pairs share subtle semantic similarities (e.g., Taxonomic or Overlap pairs like "toaster" vs. "coffee maker"). This suggests models may rely on conspicuous cues rather than deep conceptual reasoning.
3. Morphological Complexity: There is a negative correlation between morphological complexity and concept-reasoning scores. Languages with higher complexity (Agglutinative > Inflected > Analytic) yield lower scores and require deeper layers for successful conceptual reasoning.
4. Convergence of Methods: Direct probability measurements and neurolinguistic probing results show high correlation, indicating that direct probability measurements closely track the models' internal representations.

Significance and Contributions

The paper claims that XCOMPS provides a robust foundation for assessing the multilingual capabilities of LLMs. The primary contribution is the demonstration that LLMs lack a universal semantic reasoning mechanism that transcends language boundaries.

• Dataset: It provides the first multilingual conceptual minimal pair dataset covering 17 typologically diverse languages.
• Evaluation Insight: By combining prompting, probability, and probing, the work highlights the gap between a model's ability to perform a task (performance) and its actual internal understanding (competence), particularly across different languages.
• Implications: The results suggest that conceptual capabilities learned in English do not transfer seamlessly to languages with different typologies or weaker representation in training data. This challenges the assumption that LLMs possess language-agnostic conceptual understanding.

The authors acknowledge limitations, noting that the dataset does not cover all linguistic families and that the human-LLM construction process may introduce minor inconsistencies. However, the work establishes a critical benchmark for future research into multilingual semantic reasoning.