Technical Summary: CROWDMATH: A Dataset of Crowdsourced Mathematical Research Discussions

Problem Statement

Current benchmarks for Large Language Models (LLMs) in mathematical reasoning predominantly evaluate performance on well-specified problems with definitive final answers, step-by-step derivations, or complete proofs (e.g., competition-style problems or formal theorem proving). While these datasets demonstrate strong capabilities in producing correct solutions to closed-form problems, they fail to capture collaborative open-problem solving. In this setting, mathematical progress is distributed across multiple participants and posts, involving partial arguments, error identification, reasoning repair, and the gradual synthesis of incremental contributions into a complete proof. Existing models struggle to track the functional significance of individual contributions within this evolving, multi-turn discourse.

Methodology

Dataset Construction: CROWDMATH

The authors introduce CROWDMATH, a dataset comprising 164 expert-annotated progress chains derived from the MIT PRIMES–Art of Problem Solving (AoPS) CrowdMath program (2016–2025). This program is a collaborative research initiative where high school and undergraduate participants solve original, unsolved problems in discrete mathematics and combinatorics, often leading to peer-reviewed publications.

• Data Source: The dataset is drawn from public forum threads where participants discuss open problems.
• Annotation Schema: Each post is annotated with functional roles relative to the evolving solution process. The schema includes:
  • Progress: Partial ideas, lemmas, or strategies.
  • Proof: Completion of a proof or the final missing step.
  • Erroneous: Claims containing fatal mathematical errors.
  • FindError: Identification of errors in prior posts.
  • Question/Answer: Clarifying inquiries and responses.
  • Result: New mathematical findings not previously known.
• Process: Annotation occurred in two passes: (1) identifying all mathematical results established in a thread, and (2) labeling individual posts based on their contribution to those results. Posts were anonymized, and the data was structured into "progress chains" that trace a specific result from the problem statement to the final proof, preserving temporal dependencies.
• Statistics: The dataset covers 164 chains across 10+ mathematical topics (e.g., pursuit-evasion games, metric dimension of graphs). Chain lengths vary from 1 to 25 posts, with a mean of 2.7 posts per result.

Evaluation Tasks

The authors evaluated six frontier models (GPT-5.4, Claude Opus 4.6, Grok 4.20, DeepSeek V3.2, Gemini 2.5 Flash, Qwen3-235B) on two tasks derived from the annotations:

1. Post-Role Classification: Given a thread prefix and a target post, the model must classify the post into one of four functional roles: Progress, Result (Proof completion), Erroneous, or FindError. This tests the model's ability to understand the functional significance of a contribution.
2. Next-Post Prediction: Given a thread prefix, the model must predict the actual next post from a set of four candidates (one true, three distractors sampled from other threads). This tests the model's ability to follow the local flow and logical continuity of the discussion.

Key Results

1. Difficulty in Identifying Functional Significance

Models performed poorly on Post-Role Classification, achieving macro-F1 scores between 0.33 and 0.42.

• Specific Failure Mode: Models consistently struggled to distinguish between Progress (partial contributions) and Result (proof completion). While precision for the Result class was relatively high (models rarely mislabeled non-Proof posts as Proof), recall was low. Models frequently labeled posts that completed a proof merely as "Progress," failing to recognize the completion of the argument.
• Implication: Current models lack the discourse-level reasoning to judge the significance of a post within the broader collaborative trajectory.

2. Success in Local Flow Prediction

Models performed significantly better on Next-Post Prediction, achieving accuracies between 82.7% and 87.7% (compared to a 25% random baseline).

• Implication: Frontier models can effectively follow the surface flow of mathematical discourse and predict the immediate next step in a conversation, suggesting they understand the local context and logical progression of a single thread.

Significance and Claims

The paper claims that CROWDMATH exposes a critical gap between solving well-specified mathematical problems and understanding collaborative mathematical progress.

• Process vs. Product: While models can often produce correct final answers or follow a linear derivation, they struggle to understand mathematical progress as an evolving, non-linear, collaborative process involving errors, corrections, and partial ideas.
• Benchmarking Shift: The authors argue that shifting evaluation from final-answer correctness to "progress-aware reasoning" is necessary to benchmark mathematical intelligence more holistically.
• Limitations: The authors note that the dataset is result-centric (excluding abandoned ideas or dead ends), relies on coarse-grained post-level labels, and reflects the discourse of student participants rather than professional researchers.

In conclusion, the paper posits that current frontier models have not yet fully learned the discourse-level reasoning required to track how mathematical ideas develop over time in a collaborative setting, despite their proficiency in solving isolated, well-defined problems.