Evaluating Agentic Bioinformatics through Function, Evidence, and Validation

Technical Summary: Evaluating Agentic Bioinformatics through Function, Evidence, and Validation

Problem Statement
Large language model (LLM) agents are increasingly capable of planning, executing, and interpreting biological analyses. However, the field currently lacks a unified framework to establish the scientific credibility of these systems. Existing reviews primarily categorize agents by application domain, architecture, or high-level capability, but they fail to jointly operationalize the accountability of the workflows these agents generate. A system may successfully execute a tool call or produce a fluent response that matches a benchmark, yet still rely on inappropriate data, unjustified analytical assumptions, or untraceable evidence. Consequently, "final-answer correctness" is insufficient; the scientific community requires a shift toward assessing "workflow correctness"—ensuring the process supporting a conclusion is appropriate, inspectable, traceable, replayable, and scientifically defensible.

Methodology: The Function–Evidence–Validation (FEV) Framework
To address this gap, the authors introduce the Function–Evidence–Validation (FEV) framework. This framework treats the inspectable workflow trajectory—the sequence from a biological objective to an evidence-supported conclusion—as the primary unit of analysis, rather than the system architecture or final output alone. The framework consists of three complementary, non-interchangeable dimensions:

1. Function (Workflow Operations): This dimension records the demonstrable operations a system performs. It is architecture-agnostic and focuses on six observable capabilities:

   • F1: Planning and task decomposition (translating objectives into ordered, revisable plans).
   • F2: Role specialization and coordination (managing multi-agent or modular handoffs).
   • F3: Tool selection and workflow execution (invoking tools, managing parameters, and parsing outputs).
   • F4: Workflow-state maintenance and trace capture (preserving context, intermediate artifacts, and execution logs).
   • F5: Self-evaluation and repair (detecting failures and modifying plans or code).
   • F6: Verification and escalation (assessing output sufficiency and triggering review).

2. Evidence (Traceable Support): This dimension characterizes the sources used to inform actions and support claims. It distinguishes between six categories of traceable sources, emphasizing that access to a source does not equate to validation:

   • E1: Scientific literature.
   • E2: Structured biological knowledge (ontologies, databases).
   • E3: Biological measurements and metadata (sequencing reads, clinical variables).
   • E4: Software and statistical outputs (QC reports, model fits).
   • E5: Scientific-model outputs (predictions, simulations).
   • E6: Experimental or clinical observations (empirical readouts).

3. Validation (Assurance Stages): This dimension defines the level of assurance established for a specific use case through five cumulative stages (V0–V4):

   • V0 (Illustrative Output): Explanations or plans without demonstrated execution.
   • V1 (Demonstrated Execution): Successful tool invocation or workflow completion.
   • V2 (Replayable Computation): Sufficient information (inputs, parameters, versions, traces) to reconstruct the workflow.
   • V3 (Scientifically Evaluated Computation): Evaluation against task-appropriate scientific checks (baselines, statistical diagnostics, expert review) beyond mere execution.
   • V4 (Prospective Empirical Evaluation): Direct prospective testing of an agent-generated output (e.g., a designed protein or guide RNA) using empirical evidence aligned with the principal claim.
   • Qualifiers: The framework also records specific forms of assessment (e.g., Benchmark [B], Human/Expert [H], Statistical [S], Robustness [R], External [X], Prospective [P], Closed-loop [C]) that accompany the V-stage.

Evidence Map and Results
The authors applied the FEV framework to map 109 agentic or agent-adjacent systems and 28 benchmark/evaluation resources, representing 128 unique publications across genomics, single-cell and spatial omics, protein science, drug discovery, computational pathology, and general bioinformatics automation.

Key findings from the synthesis include:

• Capability Distribution: Planning (F1) and tool-mediated execution (F3) are widely represented. However, explicit verification/escalation (F6) and role specialization (F2) are less common.
• Evidence Profiles: Profiles are dominated by software outputs (E4), structured knowledge (E2), and retrospective biological data (E3). Experimental or clinical observations (E6) are comparatively rare.
• Validation Gaps: Most systems are classified at V3 (scientifically evaluated computation). Only seven systems reach V4 (prospective empirical evaluation), and none are assigned V0.
• Domain Differences:
  • Genomics/CRISPR: Strong in planning and execution, but prospective testing is rare.
  • Single-cell/Spatial: High prevalence of biological measurements (E3) and external validation (X).
  • Protein/Drug Discovery: Heavy reliance on scientific-model outputs (E5); prospective testing is concentrated here but remains limited.
  • Pathology: Strong in tool execution and trace capture, with growing use of expert assessment (H).
• The "Gap": There is a significant disparity between the breadth of operational capabilities (Function) and the depth of scientific assurance (Validation). Broader functionality (more agents, tools, or memory) does not imply stronger evidence grounding or replayability. The largest recurring gaps are in prospective empirical testing and closed-loop empirical refinement.

Significance and Contributions
The paper claims four primary contributions:

1. Redefining the Unit of Analysis: It establishes the inspectable workflow trajectory, rather than architecture or final output, as the primary metric for evaluating agentic bioinformatics.
2. The FEV Framework: It introduces a structured, accountability-oriented framework that separates demonstrated operations (Function), traceable support (Evidence), and use-case-specific assurance (Validation). This prevents the conflation of operational breadth with scientific reliability.
3. Cross-Domain Evidence Map: It provides a structured mapping of 109 systems and 28 benchmarks, offering a comparative view of reported capabilities and assurance levels across diverse biological domains.
4. A Workflow-Correctness Agenda: It outlines a path forward prioritizing inspectable shared state, provenance-preserving handoffs, explicit verification gates, and claim-aligned prospective testing.

Conclusion
The authors argue that the next stage of agentic bioinformatics depends less on maximizing autonomy or the number of tools and more on strengthening scientific accountability. Reliable systems must preserve inspectable workflow trajectories, distinguish between observations and predictions, expose uncertainty, and apply validation appropriate to the specific claims being made. The ultimate goal is to shift evaluation from "what an agent can do" to "what scientific conclusions its reported workflow can support."