Categorizing Mathematical Concepts with LLM Voting Ensembles in Mathswitch

Technical Summary: Categorizing Mathematical Concepts with LLM Voting Ensembles in Mathswitch

Problem Statement
The Mathswitch project aims to connect mathematical concepts across disparate resources (e.g., Wikidata, Wikipedia, MathWorld, Agda-Unimath) by importing records and linking those referring to the same idea. However, because Mathswitch approximates the set of mathematical concepts via SPARQL queries over the collaboratively edited Wikidata graph, the imported data is inherently noisy. This noise manifests in two ways: items that are clearly non-mathematical but match query patterns due to incorrect tagging, and genuinely ambiguous terms that possess both mathematical and non-mathematical meanings. The paper addresses the challenge of filtering this noise to ensure the reliability of the cross-reference network without requiring a manually labeled training dataset, which is difficult to produce given the specialized domain expertise required for edge cases.

Methodology
The authors propose and evaluate an automated categorization pipeline consisting of three stages:

1. Data Retrieval: SPARQL queries are executed against Wikidata to fetch candidate items based on property paths (e.g., items that are instances of topics studied by mathematics). The system also fetches associated English Wikipedia article text and external identifiers (MathWorld, nLab, etc.).
2. LLM Voting Ensemble: Instead of training a dedicated classifier, the system employs a voting ensemble of Large Language Models (LLMs) acting as "judges."
   • Input: Each candidate item is presented to multiple independent LLMs with a structured prompt containing the item's name, description, extracted keywords, and article text.
   • Task: Each judge answers a binary question ("Is the given concept a mathematical concept?") and provides a confidence score (0–100).
   • Ensemble: The system uses a heterogeneous pool of models (specifically DeepSeek-R1 14B, Gemma-3 12B, and Qwen-2.5 14B, hosted locally via Ollama). A majority vote determines the final classification.
   • Feedback Loop: Items classified as non-mathematical by a majority are automatically appended to the SPARQL exclusion list for subsequent imports, creating a self-refining data pipeline.
3. Network Construction: Filtered items are grouped into unified concepts using a Union-Find algorithm to merge records from different sources.

Key Results
The system was evaluated on a dataset of 16,385 items imported in April 2026.

• Positive Control: On a sample of 1,000 items known to have MathWorld identifiers (assumed to be mathematical), the ensemble achieved an accuracy of 98.2%.
• Robustness to Surface Markers: When references to "MathWorld" were removed from the prompt context, accuracy dropped by only ~1 percentage point, indicating the models rely on semantic content rather than database identifiers as shortcuts.
• Negative Control: On a sample of 500 physics concepts, the ensemble correctly classified 86% as non-mathematical. The few items classified as mathematical (4.8%) were identified as legitimate boundary cases (e.g., differential equations, symmetry groups) where physics and mathematics overlap.
• Error Analysis: The authors analyzed 48 items where the ensemble disagreed with the MathWorld label, categorizing the failures into three distinct types:
  1. Degenerate Descriptions: Items with tautological or empty descriptions (e.g., "Wiener sausage" described only as "mathematical concept") caused models, particularly Gemma, to misclassify based on surface names.
  2. Narrow Scope Bias: One model (Gemma) frequently rejected items with applied contexts (e.g., Pauli matrices in quantum mechanics), viewing them as applications rather than pure mathematics.
  3. Editorial-Scope Mismatches: Items unanimously rejected by judges but present in MathWorld (e.g., signal processing tools) suggest MathWorld's editorial scope is broader than the judges' working definition of "mathematical concept."

Significance and Claims
The paper claims that an LLM voting ensemble serves as a practical and effective filter for noisy mathematical concept data imported from Wikidata. The significance of the work lies not in achieving perfect accuracy, but in demonstrating that:

• A zero-shot ensemble approach can achieve high reliability (98.2%) without labeled training data.
• The specific failure modes of the ensemble are separable and actionable. The authors identify that most errors stem from upstream data quality issues (degenerate descriptions) or definitional mismatches (editorial scope), rather than fundamental flaws in the LLM approach.
• The system provides a feedback loop where categorization results directly refine the data ingestion process, improving the quality of future imports.

The authors remain modest, noting that the evaluation relies on MathWorld identifiers as a proxy for ground truth (which is itself noisy at the boundaries) and that the 1500-item sample without identifiers lacks a ground truth for calculating false positive rates. They conclude that while the ensemble is a viable filter, future work should focus on enriching degenerate descriptions and refining the definition of mathematical concepts to better align with the reference corpus.