Technical Summary: A Dataset of Rated Conceptual Arguments

1. Problem Statement

The rapid advancement of Large Language Models (LLMs) in domains with verifiable ground truth (e.g., coding, mathematics) contrasts sharply with their performance on "conceptual questions." Conceptual questions are defined as inquiries lacking a realistically accessible ground truth answer and lacking a widely accepted methodology for resolution (e.g., ethical theories, the nature of consciousness, decision-theoretic paradoxes).

The primary obstacle to improving LLMs in this domain is the absence of ground truth. It is difficult to train or evaluate models on whether a specific normative ethical theory is "correct" or whether a model is "conscious." This paper posits that while bottom-line conclusions on such topics are hard to evaluate, the arguments supporting them can be evaluated. Specifically, the authors argue that it is easier to evaluate contextualized arguments (placed in the context of existing theories and claims) and to evaluate them along specific dimensions (e.g., centrality, strength) rather than holistically.

2. Methodology

Dataset Construction

The authors constructed a dataset comprising 951 rated critiques across 442 positions. The data generation process involved:

• Positions: Sourced from diverse origins including hand-written texts (some with LLM assistance), adapted books, philosophy coursework, magazines/blogs, and model-generated text. Topics span AI safety, decision theory, normative ethics, philosophy of mind, and politics.
• Critiques: Generated via a mix of human writing and an active learning-like approach using LLMs. LLM-generated critiques were filtered through expert judges to identify those that were difficult for both humans and models to rate, as well as those that were highly rated (both correctly and erroneously).
• Rating Dimensions: Human experts rated each critique on seven dimensions (0–1 scale):
  1. Centrality: How central the attacked issue is to the position.
  2. Strength: How successfully the critique attacks the identified issues.
  3. Correctness: The proportion of claims that are factually or logically true.
  4. Clarity: The unambiguity of the critique after careful reading.
  5. Dead Weight: The extent of content unrelated to the critique's argument.
  6. Single Issue: Whether the critique focuses on a single issue.
  7. Overall: A holistic rating.

Evaluation Metrics

To assess model performance, the authors developed two scoring functions to compare model ratings against human expert ratings:

1. Weighted Pairwise Ranking Error Rate: A ranking-based loss where the model's preference between two critiques of the same position is compared to human preference. Errors are weighted by the magnitude of the difference in human ratings.
2. Custom Weighted Loss: A point-wise loss based on the absolute error of rubric items. Notably, strength and centrality are combined into a single metric (their product) to resolve ambiguity in allocating scores between the two. This metric also accounts for clarity; if a critique is rated as very unclear by humans, only clarity and overall scores are used in the loss calculation.

Experimental Setup

The authors evaluated several modern LLMs (including GPT-5, Claude Opus 4, and Gemini 2.5) using a simple baseline prompt without few-shot examples. They specifically investigated whether "thinking" or "reasoning" modes (Chain of Thought) improved performance on this task.

3. Key Results

• Model Correlation: The ranking of models on this dataset aligns well with their general capabilities and other benchmarks. Newer models generally outperform older ones within the same family (e.g., GPT-5 > GPT-4.1), and higher-tier models outperform lower-tier ones (e.g., Opus > Sonnet).
• Reasoning Limitations: Contrary to expectations that "thinking" models would excel at conceptual reasoning, the authors found that enabling "thinking" or "reasoning" modes did not typically improve performance. In some cases, it slightly hurt performance. The authors hypothesize that current reasoning training is heavily optimized for domains with verifiable answers (math, coding) and does not generalize well to the subjective, conceptual issues addressed in this dataset.
• Human vs. Model Performance: Top models (e.g., GPT-5) approach but do not yet fully saturate human expert performance. In validation tests, human experts (specifically Emery Cooper and Caspar Oesterheld) showed high agreement after discussion, with a "ceiling" performance against initial ratings that top models are beginning to approach but still fall short of.
• Dataset Limitations: The authors note distributional biases, such as model-generated critiques being disproportionately weak and a lack of positions with a wide spread of both high-quality and low-quality critiques, which complicates ranking-based evaluation.

4. Significance and Claims

The paper claims that its primary contribution is a dataset and evaluation framework for conceptual reasoning, addressing the "ground truth" problem by shifting focus from evaluating conclusions to evaluating arguments.

• Thesis Validation: The project supports the thesis that arguments on philosophical and conceptual topics can be evaluated more reliably than bottom-line conclusions, particularly when contextualized.
• AI Safety and Alignment: The work is framed within the agenda of "differential acceleration," aiming to improve AI specifically for tasks like AI safety, science, and multi-agent cooperation, which rely heavily on conceptual reasoning.
• Debate Infrastructure: The dataset is intended to support research into AI safety via debate, where AI agents debate conceptual issues to help humans make better decisions. The authors argue that while debate on ground-truth topics is well-studied, the lack of ground truth in philosophy makes the ability to evaluate arguments (rather than just outcomes) critical.

The authors remain modest about their findings, acknowledging that subjectivity in ratings cannot be fully eliminated and that the current dataset has limitations regarding the diversity of critique quality. They suggest that future iterations should include more double-rated critiques and more difficult arguments to ensure that ceiling performance remains above that of the best models.