Technical Summary: PlantMarkerBench

Problem Statement

Cell-type-specific marker genes are fundamental to plant biology, enabling the identification of cellular states across tissues and developmental stages. While numerous plant marker databases and single-cell atlases exist, they primarily rely on curated lists or high-throughput expression data without explicitly modeling the supporting evidence found in scientific literature. A critical gap remains in the ability of automated systems to interpret full-text biological papers to determine whether a specific sentence provides valid evidence for a gene–cell-type association. Current resources focus on entity extraction or marker lookup but fail to evaluate whether a model can:

1. Correctly interpret heterogeneous literature evidence (e.g., distinguishing direct expression from indirect functional associations).
2. Determine the validity of a gene–cell-type claim amidst noise, spurious aliases, and ambiguous contexts.
3. Classify evidence into specific biological regimes (expression, localization, function, indirect, or negative).

Consequently, the reliability of modern Large Language Models (LLMs) in performing literature-grounded biological evidence attribution remains unquantified.

Methodology

The authors introduce PlantMarkerBench, a multi-species benchmark constructed from full-text scientific literature. The dataset and evaluation framework were developed using a reproducible, modular "agentic" curation pipeline.

Dataset Construction Pipeline

The pipeline integrates five stages to generate 5,550 sentence-level evidence instances across four plant species (Arabidopsis thaliana, maize, rice, and tomato):

1. Literature Collection & Filtering: Papers are retrieved from PubMed/PMC using species- and cell-type-oriented queries. Full-text XML is parsed, retaining only sections likely to contain biological evidence (abstracts, results, discussions) while excluding methods and references.
2. Species Assignment: Articles are assigned to a primary species based on weighted mentions in titles and abstracts to prevent cross-species contamination.
3. Biological Grounding:
   • Gene Matching: Species-specific matchers map gene mentions to canonical identifiers (e.g., TAIR for Arabidopsis, RAP/MSU for rice, MaizeGDB for maize, SGN for tomato), handling aliases and synonyms.
   • Cell-Type Vocabulary: Controlled vocabularies are constructed from developmental biology literature and single-cell atlases.
4. Hybrid Retrieval & Candidate Generation: A retrieval agent decomposes articles into sentence-centered windows. A hybrid scoring strategy combines keyword matching, BM25, dense embeddings, and evidence-cue detection to identify candidate gene–cell-type pairs.
5. Evidence Grading & Aggregation: An LLM-based agent evaluates candidates, assigning structured labels for validity, evidence type, and support strength. A human review protocol involving computational biology experts validates difficult cases (e.g., spurious aliases, wrong-gene grounding).

Benchmark Tasks

The benchmark defines two primary evaluation tasks:

1. Marker-Evidence Validity Prediction: A binary classification task determining if a candidate sentence provides valid evidence supporting a gene as a marker for a target cell type.
2. Evidence-Type Classification: A multi-class task categorizing evidence into five regimes: Expression (direct expression/reporter activity), Localization (spatial localization), Function (perturbation/phenotype links), Indirect (biologically related but not direct marker evidence), and Negative/Noise (irrelevant or unsupported).

The dataset includes substantial "hard negatives" and weak-support examples, reflecting the realistic difficulty of extracting reliable markers from literature. Approximately two-thirds of instances are invalid, weak, indirect, or ambiguous.

Key Contributions

1. First Multi-Species Literature-Grounded Benchmark: PlantMarkerBench is the first benchmark designed to evaluate literature-grounded plant marker evidence attribution from full-text scientific papers across multiple species.
2. Reproducible Modular Pipeline: The authors release a pipeline integrating biological grounding, hybrid retrieval, structured evidence grading, and targeted human review, enabling future extension and auditing.
3. Fine-Grained Evidence Regimes: The benchmark moves beyond binary retrieval to define biologically meaningful evidence categories (expression, localization, function, indirect, negative) and support strength annotations (strong, medium, weak).
4. Comprehensive LLM Benchmarking: The paper provides a systematic evaluation of both closed-source and open-weight LLMs across species and prompting strategies, analyzing failure modes through evidence-type and error-taxonomy analysis.

Results

The authors evaluated diverse models (including GPT-5.4, Qwen2.5, Llama3, and DeepSeek-R1) on the benchmark. Key findings include:

• Performance Gap: While frontier models achieve moderate success on binary validity prediction (F1 ~0.5–0.7), performance drops substantially on fine-grained evidence-type classification.
• Evidence-Type Difficulty: Models perform relatively well on Expression evidence but struggle significantly with Localization, Functional, and Indirect evidence. For many models, the macro-F1 for evidence-type classification remains below 0.4.
• Dominant Failure Modes:
  • Evidence-Type Confusion: The most common error is misclassifying the type of evidence (e.g., confusing indirect associations with direct expression).
  • False Positives in Open-Weight Models: Smaller and open-weight models exhibit elevated false-positive rates, often over-predicting marker evidence from weak co-occurrence patterns or ambiguous contexts.
  • Species-Specific Challenges: Performance varies by species. Rice showed the strongest overall performance, while maize and tomato proved more challenging, particularly for localization reasoning.
• Prompting Effects: Few-shot prompting improved binary validity prediction but did not consistently resolve fine-grained evidence attribution. Direct prompting often yielded the best evidence-type macro-F1 for larger models.

Significance and Claims

The paper claims that PlantMarkerBench provides a challenging and reproducible evaluation framework for literature-grounded biological evidence attribution. The authors assert that current LLMs, even frontier models, remain far from solving the problem of reliable marker evidence attribution. The benchmark highlights that models often recognize biologically relevant context without accurately grounding gene–cell-type relationships or distinguishing mechanistic evidence categories.

The significance of this work lies in:

• Establishing a standard for evaluating AI-assisted plant biology beyond simple entity extraction.
• Revealing that "evidence-type confusion" is a dominant failure mode, suggesting that models lack the deep contextual biological inference required to distinguish between direct and indirect evidence.
• Providing a foundation for future research into trustworthy scientific information extraction and AI-assisted discovery in plant biology.

The authors remain modest, noting limitations such as potential pretraining overlap, the current focus on specific cell types (root/developmental), and the reliance on LLM-assisted grading which may contain residual noise. They position the benchmark as a step toward more robust, evidence-grounded scientific reasoning rather than a solved problem.