Technical Summary: Croissant Tasks: A Metadata Format for Reproducible Machine Learning Evaluations

Problem Statement

Reproducibility remains a critical bottleneck in machine learning (ML), particularly within benchmarks and competitions that drive community research. Current practices suffer from underspecified execution details (e.g., hyperparameters, data preprocessing, prompts), brittle software environments, and the withholding of source code. While existing remedies like checklists, model cards, and reproducibility badges improve transparency, they rely on human-centric verification and lack a formal, standardized representation of execution flows. Consequently, reproducing results often requires labor-intensive, ad-hoc setups rather than automated execution, creating a gap between high-level conceptual contributions and the low-level details required for execution.

Methodology

The authors introduce Croissant Tasks, a declarative, machine-actionable metadata format designed to abstract low-level implementation details into high-level specifications. Built on Semantic Web technologies (extending schema.org and the Croissant Datasets vocabulary), the format uses JSON-LD to represent benchmarks as structured data.

The core methodology involves two distinct but linked components:

1. Problem-Solution Decoupling: The format formally separates the Task Problem (the definition of the challenge, including inputs, expected output schemas, and evaluation metrics) from the Task Solution (the concrete implementation, execution environment, and results). This allows for standardized comparisons across diverse approaches against an immutable problem definition.
2. Agentic Automation: The authors propose a workflow where autonomous agents (LLMs) utilize these specifications. The workflow consists of:
   • Extraction: An agent retrofits existing benchmark papers into the Croissant Tasks format, generating TaskProblem and TaskSolution files.
   • Synthesis: A second agent ingests the TaskProblem specification to generate functional code implementations from scratch, aiming to reproduce the reported results without access to the original source code.

This approach shifts the goal from "technical replication" (running the exact original code) to "conceptual reproducibility" (verifying scientific claims via independently generated implementations).

Key Contributions

The paper makes three primary contributions:

1. The Croissant Tasks Specification: A formal vocabulary that decouples task problems from solutions. It defines core classes such as cr:TaskProblem (defining inputs, output specs, and evaluation specs) and cr:TaskSolution (providing concrete implementations and results). It supports hierarchical structures via cr:subTask for complex benchmarks.
2. An Automated LLM Pipeline: A system that automatically extracts Croissant Task descriptions from research papers. This pipeline includes validation against SHACL shapes to ensure structural correctness and iterates to fix errors.
3. Empirical Validation: Experiments demonstrating that autonomous agents can ingest these specifications to generate functional, accurate reproduction pipelines. The study evaluates five diverse benchmarks from the NeurIPS 2025 Datasets and Benchmarks track (Absence Bench, CoRe, MedSG-Bench, NOVA, SAGE-Eval).

Experimental Results

The authors evaluated the format's expressivity and the feasibility of automated reproduction:

• Expressivity and Extraction: In generating Croissant Task descriptions from five benchmark papers, the agent achieved an average field coverage of 97.4%. The single lower score (87% on CoRe) was attributed to the extraction process missing specific hyperparameters in the source text, not a limitation of the format itself.
• Automated Reproduction: Agents were tasked with generating implementations using only the Croissant Task file (without the original paper PDF) versus using the PDF directly.
  • Croissant Tasks Only: Achieved an average success rate of 97.1% in correctly implementing evaluation metrics.
  • PDF Only: Achieved an average success rate of 90%.
  • Comparison: The structured Croissant Tasks format provided a clearer blueprint, reducing context window overload and preventing premature agent termination seen in PDF-based attempts.
  • Failure Modes: In the complex NOVA benchmark, agents sometimes substituted complex logic with simplified proxies or failed to utilize specific libraries without explicit guidance. However, providing detailed metric descriptions in the Croissant format mitigated these issues.

Significance and Claims

The paper positions Croissant Tasks as a foundational step toward conceptual reproducibility. By providing a machine-readable specification that links claimed outcomes to the necessary inputs and metrics, the format enables:

• Interoperability: Benchmarks defined on one platform can be seamlessly imported and executed on others (e.g., moving from Codabench to HELM).
• Evolution and Reuse: Separating problems from solutions allows researchers to test new methods on existing tasks without rewriting evaluation protocols.
• Discoverability: Leveraging schema.org enables structured search and indexing of tasks and benchmarks for both humans and autonomous agents.

The authors modestly claim that this work demonstrates the feasibility of automated reproduction via high-level specifications. They acknowledge that full adoption requires overcoming barriers related to ease of use, institutional incentives (e.g., conference mandates), and the fidelity of metadata generation. The ultimate vision is to shift the ML community away from brittle technical replication toward a system where scientific claims are verified through distinct, independently generated implementations derived from standardized, machine-actionable specifications.