🧬 biology

GenCellAgent: Generalizable, Training-Free Cellular Image Segmentation via Large Language Model Agents

GenCellAgent is a training-free, multi-agent framework that leverages large language models to dynamically orchestrate specialist tools and vision-language models, achieving state-of-the-art, adaptable cellular image segmentation across diverse modalities and novel organelles without the need for retraining.

Original authors: Xi Yu, Yang Yang, Qun Liu, Yonghua Du, Sean McSweeney, Yuewei Lin

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xi Yu, Yang Yang, Qun Liu, Yonghua Du, Sean McSweeney, Yuewei Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are a biologist looking at thousands of microscopic photos of cells. Your goal is to draw outlines around specific parts, like the nucleus or the mitochondria (the cell's power plants). In the past, this was like trying to find a needle in a haystack using a single, rigid metal detector. If the needle was made of a different metal (a different type of microscope image), your detector wouldn't work, and you'd have to buy a new one or spend hours manually drawing the outlines yourself.

GenCellAgent is a new, "training-free" digital assistant that solves this problem. Think of it not as a single tool, but as a smart project manager that runs a team of specialists and generalists to get the job done without needing to be retrained for every new job.

Here is how it works, broken down into simple concepts:

1. The "Smart Project Manager" (The Multi-Agent System)

Instead of one AI trying to do everything, GenCellAgent uses a team of three AI "agents" that talk to each other:

The Planner: This is the boss. When you upload an image and say, "Find the mitochondria," the Planner looks at the picture and asks, "Who is the best expert for this specific photo?" It might choose a specialist tool trained on that exact type of microscope, or a general tool if the image is unusual.
The Executor: This is the worker. It runs the tool the Planner chose to draw the outlines.
The Evaluator: This is the quality inspector. It looks at the drawing the Executor made and asks, "Is this accurate? Does it look like a real mitochondria?" If the answer is "No," it sends the drawing back to the Executor to try again with better instructions.

2. The "Memory Book" (Long-Term Learning)

Most AI tools forget everything once you close the window. GenCellAgent has a long-term memory.

Self-Evolution: If the system struggles with a new type of cell structure (like the Golgi apparatus), it tries to solve it. Once it gets it right (perhaps with a little help from a human), it saves that success in its "Memory Book."
The Result: The next time you ask it to find that same structure, it doesn't start from scratch. It opens its Memory Book, finds the previous example, and says, "I've seen this before! I know how to do it." It literally learns from its own past mistakes and successes without needing a human to retrain it.

3. The "Chameleon" (Adapting to New Conditions)

Imagine you have a tool that is great at taking photos in bright sunlight, but you suddenly need to take photos in a dark cave. A normal tool would fail.

GenCellAgent acts like a chameleon. If the image looks different from what its tools expect (a "domain shift"), it doesn't just give up. It grabs a few "reference images" (like a cheat sheet) and uses them to instantly adapt its strategy. It can switch from a specialist tool to a flexible, general-purpose tool on the fly to handle the weird lighting or texture.

4. The "Text-to-Image" Translator

Sometimes, you want to find a cell part that no one has ever programmed a tool to find yet.

You can simply type a description: "Find the organelle that looks like a stack of pancakes."
GenCellAgent reads your text, understands the visual description, and uses a powerful vision model to hunt for those "pancake stacks." If it misses a spot, it reads its own feedback, rewrites its instructions, and tries again until it gets it right.

5. The "Human Co-Pilot" (Human-in-the-Loop)

Sometimes, even the best AI gets confused. GenCellAgent has a special mode where a human expert can step in.

Instead of starting over, you can click a few dots or draw a quick line to correct the AI's mistake.
The Magic: The system doesn't just accept the correction; it learns from it. It saves your correction in its Memory Book. Next time, it will remember, "Oh, the user prefers this specific way of drawing the edge," and it will adjust its future behavior to match your style.

The Bottom Line

GenCellAgent is a universal, self-improving assistant for cell biology.

It doesn't need retraining: It works on new images immediately.
It gets smarter over time: It remembers what worked before.
It adapts: It switches tools automatically if the image changes.
It collaborates: It lets humans fix mistakes easily, and then remembers those fixes for next time.

The paper claims this system can handle a wide variety of cell images (from 7 different benchmarks) better than using any single tool alone, and it can even find new cell parts just by reading a text description, all while reducing the amount of time scientists spend manually drawing lines on images.

Technical Summary: GenCellAgent

Problem Statement

Cellular image segmentation is a cornerstone of quantitative biology, yet it faces significant hurdles due to the heterogeneity of imaging modalities, morphological variability of cells and organelles, and the scarcity of annotated data. Existing solutions, ranging from classical handcrafted pipelines to specialized deep learning models (e.g., Cellpose, SAM derivatives, MitoNet, ERNet), often struggle to generalize across different laboratories, microscopes, or labeling strategies. Retraining models for new domains introduces latency and cost, while current Large Language Model (LLM) agent systems for bioimage analysis typically rely on static planning, lack adaptive tool selection, and do not possess long-term memory to learn from user feedback. Consequently, there is a need for a robust, adaptable segmentation framework that operates without retraining, handles out-of-distribution (OOD) data, and integrates expert human guidance efficiently.

Methodology

The authors present GenCellAgent, a training-free, multi-agent framework that orchestrates specialist segmenters and generalist vision-language models (VLMs) through a structured planner-executor-evaluator loop supported by long-term memory. The system comprises three coordinated LLM agents:

Planning Agent ( $A_{plan}$ ): Acts as the central coordinator. It interprets user queries, retrieves relevant prior knowledge from memory, and designs workflows. It dynamically selects tools based on image-style similarity and task requirements. If a specific tool is unavailable or underperforms, it escalates to in-context adaptation or text-guided generation.
Execution Agent ( $A_{exe}$ ): Orchestrates the invocation of tools. This includes:
- Specialized Tools: Routing to state-of-the-art models like Cellpose, $\mu$ SAM, CellSAM, MitoNet, and ERNet based on the target organelle and image modality.
- Generalist Tools: Utilizing agentic SAM3 (for text-guided segmentation) and SegGPT (for in-context learning) when no dedicated tool exists or when domain shifts occur.
- Human-in-the-Loop (HITL): Facilitating user corrections via a graphical interface using positive/negative points and polygons.
Evaluation Agent ( $A_{eval}$ ): A vision-language model that assesses segmentation quality against specific criteria (e.g., boundary precision, morphological accuracy). It provides a numerical score and textual feedback to refine prompts for iterative improvement.

Key Mechanisms:

Intelligent Tool Selection: The system computes style-similarity correlations between input images and anchor datasets to route images to the best-performing specialist tool.
In-Context Adaptation: When specialist tools fail on OOD data (e.g., different microscopy modalities), the system retrieves similar image-mask pairs from memory and uses SegGPT to adapt the segmentation without retraining.
Text-Guided Segmentation: For novel objects without dedicated tools, the system uses an iterative pipeline where a VLM (Gemini) interprets textual descriptions to generate bounding boxes, which then prompt SAM3 for mask generation. This loop refines prompts based on evaluation feedback.
Memory-Driven Self-Evolution: Successful segmentations and user preferences are stored in a long-term memory module. This allows the system to retrieve reference examples for few-shot learning and adapt to individual user workflows over time.

Key Contributions

Training-Free Framework: GenCellAgent achieves high-performance segmentation across diverse modalities without requiring task-specific model retraining.
Multi-Agent Orchestration: It is the first application of LLM agents to cellular image segmentation, linking language-guided reasoning to domain-specific vision tools via a dynamic planning-execution-evaluation loop.
Adaptive Routing and Enhancement: The system automatically selects the optimal specialist tool for each image and employs in-context learning to recover performance on distribution shifts where dedicated tools fail.
Novel Object Segmentation: It enables fully automated, text-guided segmentation of organelles not covered by existing models (e.g., Golgi apparatus) through iterative refinement.
Personalization and Self-Evolution: By integrating HITL corrections into memory, the system learns from user feedback, enabling personalized workflows and continuous performance improvement over time.

Results

The authors evaluated GenCellAgent across seven cell-segmentation benchmarks comprising 4,718 images from diverse microscopy modalities (phase-contrast, fluorescence, electron microscopy, etc.).

Tool Selection Accuracy: GenCellAgent achieved 87.2% tool-selection accuracy, consistently matching or exceeding the performance of the best individual specialist tool on every dataset.
Overall Performance: The system outperformed all baselines (Cellpose, $\mu$ SAM, CellSAM) in overall segmentation accuracy (mean IoU) across the full test set.
Out-of-Distribution Robustness: On OOD data (e.g., FIB-SEM images for ER and Golgi), where specialist models (ERNet, MitoNet) failed or degraded significantly, GenCellAgent substantially outperformed them by leveraging in-context adaptation with SegGPT.
Novel Object Segmentation: In zero-shot settings for mitochondria, ER, and Golgi (without dedicated training data), GenCellAgent outperformed standalone zero-shot baselines (SAM3, CLIPSeg) and approached the performance of supervised models trained on the specific domain.
Self-Evolution: In continuous learning scenarios, GenCellAgent improved its performance as the number of reference examples increased, outperforming SAM-based models that were continually fine-tuned on the same data.
Human-in-the-Loop: Minimal human corrections (points/polygons) significantly boosted segmentation quality, and the system effectively learned from these corrections to improve future runs.

Significance

The paper claims that GenCellAgent provides a practical path to robust, adaptable cellular image segmentation without retraining. By reducing the annotation burden and matching user preferences through self-evolving memory, it makes advanced analysis broadly accessible to biological researchers. The framework bridges the gap between agentic language-model control and domain-specific vision tools, offering a scalable solution for the heterogeneous and evolving landscape of biological imaging. The authors position this work as a foundational step toward training-free, generalizable bioimage workflows in everyday laboratory settings.