Benchmarking Agentic Bioinformatics Systems for Complex Protein-Set Retrieval: A Coccolithophore Calcification Case Study

This study benchmarks three agentic bioinformatics systems on a complex protein-retrieval task involving coccolithophore calcification, revealing that while some agents generate larger protein sets, the Codex system achieves superior performance through a better balance of sensitivity and specificity, higher relevance, and greater repeatability.

The Gist

Imagine you are a chef trying to cook a very specific, complex dish: Coccolithophore Calcification.

In the real world, this is how tiny ocean algae build their tiny, intricate limestone shells. To do this, the algae need a massive team of proteins (the "ingredients" and "tools") working together. Some transport carbon, some handle calcium, some build the shell structure, and others send signals to tell the cell when to start.

The problem? There are millions of proteins in the world's biggest protein database (UniProt), and finding the exact ones needed for this specific algae is like finding a needle in a haystack, where the haystack is made of other needles that look almost the same.

This paper is a cooking competition to see which "AI Chef" (Agentic System) is best at gathering the right ingredients for this specific recipe.

The Three Contestants

The author set up a challenge for three different AI systems to download the correct list of proteins. Here is how they performed, using a grocery shopping analogy:

1. Codex (The Precision Shopper)

• The Style: Codex is like a strict, highly trained personal shopper who reads your recipe card word-for-word.
• The Result: It brought back a moderate-sized bag of groceries (2,118 items).
• The Quality: Almost everything in the bag was exactly what you needed. 92% of the items were perfect matches. It didn't bring you a whole crate of generic "salt" when you only needed "sea salt."
• The Vibe: Reliable, consistent, and high-quality. If you asked it to shop again tomorrow, it would bring back the exact same bag.

2. DeerFlow (The Enthusiastic Scout)

• The Style: DeerFlow is like a scout who loves to explore. It brings back a huge bag (6,255 items).
• The Quality: It found everything Codex found, plus a lot of extra stuff. Some of the extras were great "bonus" ingredients (like a specific type of glue for the shell), but a lot of it was just "maybe useful" generic stuff (like generic kitchen knives when you needed a specific scalpel). About 44% of its bag was a bit too broad.
• The Vibe: Great for finding hidden gems, but you have to do a lot of sorting to separate the good stuff from the clutter. It was a little inconsistent; if you asked it to shop twice, the second bag looked quite different from the first.

3. Biomni (The Over-Excited Broadcaster)

• The Style: Biomni is like a radio host who hears "calcium" and starts listing every calcium-related thing in the universe. It brought back a massive truckload (8,752 items).
• The Quality: While it found the right ingredients, it also filled the truck with generic calcium sensors, random enzymes, and things that only vaguely relate to the job. Nearly 70% of its bag was "low relevance" junk.
• The Vibe: It's great if you want to brainstorm ideas, but terrible if you need a specific list for a recipe. If you asked it to shop twice, the two bags would be completely different, making it hard to trust.

The Big Lessons (The "Aha!" Moments)

The paper teaches us three main things about using AI for science:

1. Volume isn't Victory
Just because an AI gives you a bigger list doesn't mean it's better. In fact, a huge list often means the AI is guessing too much. Codex won because it was precise, not because it was loud.

2. Consistency is King
If you ask an AI to do a task twice and it gives you two totally different answers, you can't trust it. Codex was almost identical every time it ran. Biomni changed its mind wildly. For science, you need the robot that doesn't get confused.

3. The "Hybrid" Strategy is Best
The author realized that the perfect solution wasn't just one AI. It was a mix:

• Use Codex as the main backbone because it's so accurate.
• Use DeerFlow as a "supplement" to check if it found any cool, specific extras that Codex missed (like specific shell-glue proteins).
• Ignore Biomni for this specific job because it was too messy.

The Takeaway for Everyone

Think of these AI agents like different types of search engines.

• Biomni is like a wild Google search that gives you 10 million results, including ads and unrelated blogs.
• DeerFlow is like a specialized library catalog that finds the right books plus some related magazines.
• Codex is like a librarian who knows your exact needs and hands you the three perfect books, no questions asked.

The Conclusion: When doing complex scientific work, you don't want the AI that talks the most or finds the most things. You want the one that understands the context, sticks to the rules, and gives you a list you can trust without spending hours cleaning it up.

The Golden Rule: Don't just ask the AI for "everything." Break the big problem into small, specific steps (like "find carbon transporters" then "find shell builders"), and always check if the AI gives you the same answer twice before you trust it.

Technical Summary

1. Problem Statement

The paper addresses the challenge of evaluating Large Language Model (LLM) agents in complex bioinformatics tasks. While LLM agents are increasingly used for multi-step retrieval workflows involving external databases (like UniProt), practical evaluations are limited, particularly for prompts where the target protein set is large and biologically heterogeneous.

The specific problem is retrieving proteins related to coccolithophore calcification. This biological process is not defined by a single ontology term but is distributed across diverse mechanisms: inorganic carbon acquisition, calcium transport, pH regulation, organic matrix assembly, polysaccharide metabolism, vesicle trafficking, and signaling. The difficulty lies in the fragmented and unevenly curated nature of UniProt annotations for these specific processes, making exhaustive yet precise retrieval difficult for AI agents.

2. Methodology

The study benchmarked three distinct agentic AI systems on a standardized, difficult retrieval task.

• The Benchmark Task:

  • Goal: Download all UniProt proteins related to coccolithophore calcification.
  • Constraints: Restrict search to coccolithophores only; organize outputs into six mechanistic categories; provide evidence for selections.
  • Categories: (1) Inorganic carbon acquisition, (2) Calcium/pH homeostasis, (3) Organic matrix/adhesion, (4) Polysaccharide biosynthesis, (5) Vesicle trafficking, (6) Signaling/regulation.
  • Repetition: Each system was run twice to assess within-system stability (repeatability).

• Systems Compared:

  1. Codex + Claude Scientific Skills: A general multi-agent desktop environment extended with a domain-specific scientific skill library.
  2. DeerFlow 2: An open-source "super-agent" harness using default skills only.
  3. Biomni Lab: A biomedical AI agent focused on retrieval-augmented planning and code execution (tested via online output).

• Evaluation Metrics:

  • Normalization: All outputs were normalized to UniProt accession numbers.
  • Overlap Analysis: Three-way Venn decomposition was performed for each category to identify shared cores and unique expansions.
  • Relevance Scoring: A heuristic relevance assessment classified subsets as High, Medium, Low-Medium, or Low relevance based on alignment with known calcification literature (e.g., penalizing generic kinases or transcription factors).
  • Stability Metrics: Repeatability was measured using Jaccard similarity (mean category and micro-Jaccard) between Run 1 and Run 2.
  • Provenance: Analysis of output artifacts (scripts, manifests, query terms) to assess reproducibility.

3. Key Contributions

• Novel Benchmarking Framework: Establishes a protocol for evaluating agentic bioinformatics systems that goes beyond simple accuracy, incorporating sensitivity-specificity trade-offs, repeatability, and provenance artifacts.
• Taxonomic and Mechanistic Decomposition: Demonstrates that complex biological queries must be decomposed into mechanistic bins (e.g., separating transport from signaling) to achieve high retrieval quality.
• Stability as a Critical Metric: Introduces the concept that repeatability is a crucial indicator of agent quality; unstable agents produce high-variance outputs that increase downstream curation costs.
• Hybrid Workflow Recommendation: Proposes a "best-practice" workflow combining a specificity-oriented agent for the backbone and a broader agent for targeted supplementation, rather than relying on a single system's output.

4. Results

A. Sensitivity vs. Specificity Trade-off

• Codex: Retrieved 2,118 proteins. It achieved the best balance: 92.4% of its proteins fell into "High" or "Medium" relevance subsets. It showed the highest precision.
• DeerFlow: Retrieved 6,255 proteins. It was more exhaustive but less specific: 56.2% High/Medium relevance, with significant expansion into "Low-Medium" relevance.
• Biomni: Retrieved 8,752 proteins (largest set). It had the weakest specificity: only 30.5% High/Medium relevance, with 69.5% falling into "Low" or "Low-Medium" categories due to broad expansion into generic calcium sensors, kinases, and transcription factors.

B. Repeatability (Stability)

• Codex: Extremely stable. Mean category Jaccard = 0.982; Micro-Jaccard = 0.974. It produced nearly identical results across runs.
• DeerFlow: Intermediate stability. Mean category Jaccard = 0.795; Micro-Jaccard = 0.571. Stable in transport categories but varied significantly in signaling.
• Biomni: Least stable. Mean category Jaccard = 0.412; Micro-Jaccard = 0.319. Results varied wildly between runs, particularly in broad categories like signaling.

C. Category-Specific Insights

• Transport/Enzyme Categories (e.g., Carbon Acquisition): All systems converged well on a core of carbonic anhydrases and transporters.
• Matrix/Adhesion Categories: Codex uniquely identified pentapeptide-repeat candidates, while DeerFlow and Biomni found lectin/fibronectin candidates.
• Signaling Categories: This was the least specific area. All agents struggled to avoid generic kinases and transcription factors, though Codex remained the most constrained.

D. Provenance and Artifacts

• Codex provided the most useful artifacts for downstream curation, including retrieval scripts, test files, and accession-level evidence tables.
• DeerFlow was explicit about query generation (structured JSON with UniProt terms).
• Biomni provided narrative evidence but lacked the structured query transparency of the others.

5. Significance and Conclusion

The study concludes that for complex protein-family retrieval, agent quality depends less on raw output volume and more on prompt decomposition, taxonomic scoping, and query precision.

• Codex is recommended as the primary source for establishing a high-quality "backbone" set due to its balance of specificity and stability.
• DeerFlow is valuable as a targeted supplement, particularly for matrix and glycan categories, provided its broader expansions are filtered.
• Biomni, while capable of high recall, behaves more like an exploratory tool and is less suitable for precise, category-resolved exports without heavy manual curation.

Key Takeaway: High-quality AI-assisted bioinformatics requires a hybrid workflow: using a specificity-oriented agent for the core set, a broader agent for supplementation, and rigorous accession-level overlap analysis combined with repeated-run stability checks to validate the final dataset. The paper emphasizes that "repeatability" is a critical, often overlooked dimension of AI agent reliability in scientific research.