Comparative benchmarking of AI research agents for omics data interpretation

Technical Summary: Comparative Benchmarking of AI Research Agents for Omics Data Interpretation

Problem Statement
The rapid acceleration in the generation of omics data (proteomics, transcriptomics, and metabolomics) has created a bottleneck in data analysis and interpretation, shifting research toward data-driven rather than hypothesis-driven modes. While AI research agents are increasingly deployed to assist or replace human bioinformaticians in analyzing these datasets, there is a lack of systematic approaches to assess their utility or determine which agent is best suited for specific tasks. Currently, many scientists rely on general-purpose Large Language Models (LLMs) like ChatGPT, while specialized agents such as K-Dense, Finch, and Biomni are emerging. The authors identify a pressing need for careful, standardized benchmarking of these agents to provide the community with an auditable basis for selection.

Methodology
The study employed a rigorous, quantitative benchmarking framework comparing four AI systems: three specialized biomedical agents (K-Dense, Finch, and Biomni) and one general-purpose agent (ChatGPT).

• Datasets and Modalities: The agents were evaluated on three distinct omics modalities using standardized, de-identified datasets:
  • Metabolomics: LC-MS/MS data regarding atrial fibrillation predictors.
  • Proteomics: Label-free mass spectrometry data from Alzheimer's disease brain tissue.
  • Transcriptomics: Bulk RNA-seq data from the TCGA-BRCA project.
• Experimental Design: Each agent was run eight times per modality (96 total runs) using identical, modality-specific prompts designed to reflect real-world query variations (from open-ended exploration to prescriptive workflows).
• Evaluation Framework: The authors introduced a multi-dimensional scoring system:
  • AgentScore: A process-anchored composite score defined as $AgentScore = P \times (0.5 + 0.3 \frac{D}{100} + 0.5 \frac{R}{100})$, where $P$ is process quality, $R$ is result reporting, and $D$ is interpretation depth. Crucially, $P$ acts as a multiplicative gate; outputs failing to document the analytical process cannot achieve a high score regardless of result quantity.
  • DomainScore: An independent, domain-aware ordinal rubric (0–2 scale) based on 16 modality-specific criteria (e.g., batch correction, normalization, pathway interpretation).
  • Reproducibility: Measured via the mean pairwise cosine similarity of sentence-transformer embeddings across independent runs.
  • Robustness Checks: The study utilized bootstrap resampling (10,000 iterations) to quantify ranking uncertainty and tested rankings against seven alternative scoring definitions to control for output volume, component weighting, and reproducibility bias.

Key Results

• No Universal Winner: No single agent dominated across all modalities. Performance was highly modality-dependent.
  • Metabolomics: K-Dense ranked first (92.66), closely followed by Finch (90.20). Biomni was third, and ChatGPT last.
  • Proteomics: Finch ranked first (86.72), with K-Dense and Biomni in a statistically non-significant tier below. ChatGPT was last.
  • Transcriptomics: Finch ranked first (86.72) with a narrow confidence interval, followed by K-Dense. Biomni occupied a stable mid-tier, while ChatGPT was last.
• Agent Characteristics:
  • K-Dense: Produced the most reproducible outputs (cosine similarity > 0.90 in all modalities) and the broadest domain coverage, but occasionally required user re-prompts due to internal verification failures.
  • Finch: Produced highly organized, templated reports with the highest consistency in transcriptomics. It excelled in proteomics and transcriptomics but showed weaker coverage of upstream preprocessing steps.
  • Biomni: Offered strong biological interpretation but frequently omitted quality control and preprocessing steps. It experienced operational failures (stalls) during computationally intensive bootstrap steps in transcriptomics.
  • ChatGPT: Without specialized scaffolding, it managed only basic statistics and finished last in all modalities. It exhibited "silent failures" (zero parsable output) in several runs and high run-to-run variability in transcriptomics.
• Scoring Sensitivity: Rankings were robust to alternative scoring definitions regarding output volume and component weighting, except in proteomics where the top three agents were closely clustered, making the choice of rubric a significant determinant of the final rank.

Significance and Claims
The paper claims that agent architecture (the "harness") is more critical than base model capability in determining output quality. The interaction between the agent's scaffolding (e.g., iterative loops, tool use, workflow constraints) and the specific analytical steps of each omics modality dictates performance, rather than the underlying LLM alone.

The study emphasizes that:

1. Process-anchored evaluation is essential; an agent cannot be considered successful if it produces plausible interpretations without documenting the analytical process.
2. Reproducibility and accuracy are distinct metrics. High reproducibility does not guarantee high accuracy (as seen with ChatGPT's consistent but shallow outputs), and high accuracy does not guarantee stability.
3. Multi-rubric reporting is necessary. When agents are closely clustered in performance, the choice of scoring rubric becomes the primary driver of rankings, necessitating the reporting of multiple parallel metrics (AgentScore, DomainScore, and reproducibility) to provide a complete picture.

The authors conclude that while AI agents can automate complex omics workflows, their outputs should be treated as preliminary implementations requiring human oversight. Future pipelines will likely require hybrid systems combining specialized sub-agents rather than relying on a single general-purpose model.