MyeGPT: an AI agent for Multiple Myeloma

MyeGPT is an AI-powered browser application that enables multiple myeloma researchers to validate hypotheses and visualize results from the complex CoMMpass dataset by converting natural language queries into automated, data-driven analyses without requiring programming skills.

The Gist

Imagine you have a massive, incredibly detailed library of medical records for a specific type of blood cancer called Multiple Myeloma. This library, known as the CoMMpass study, contains the "life stories" of 1,143 patients. It includes their symptoms, treatment history, genetic makeup, and how long they lived. However, there's a catch: this library is written in a secret code (complex computer programming languages) that only a few specialized librarians (bioinformaticians) can read. For a regular doctor or researcher, trying to find a specific answer—like "Do patients with a certain gene mutation live longer?"—is like trying to find a needle in a haystack while wearing blindfolded gloves.

Enter MyeGPT.

What is MyeGPT?

Think of MyeGPT as a super-smart, bilingual librarian who has memorized the entire CoMMpass library. You don't need to know the secret code to talk to it. You can simply ask it in plain English, just like you would ask a human: "Show me the survival rates for patients who relapsed after their first treatment," or "Compare the overall health of patients with high versus normal levels of a specific protein."

MyeGPT doesn't just guess the answer. It acts like a detective that:

1. Understands your question.
2. Digs into the database to find the exact pages of data needed.
3. Does the math to calculate the answer.
4. Draws a picture (a chart or graph) to show you the results.

How Does It Work?

The researchers built this "librarian" using a type of Artificial Intelligence called Agentic AI. Unlike a standard chatbot that just chats, an "agent" can actually do things.

• The Brain (LLM): This is the part that understands your language and plans the steps to solve the problem.
• The Tools: MyeGPT has a set of digital tools. It has a "search tool" to find the right definitions in the library's manual, a "database tool" to run complex queries, and a "drawing tool" to create graphs.
• The Memory (Knowledge Base): Before it starts, the team fed MyeGPT a 52-page instruction manual that explains exactly what every column in the database means. This ensures it doesn't get confused about what "treatment response" or "copy number" actually refers to.

Did It Work? (The Test Drive)

The researchers didn't just build it; they put it through a rigorous driving test to see if it was safe and accurate.

1. The Trivia Test: They asked MyeGPT 20 tricky questions that required complex math and data digging. They compared its answers against a "gold standard" created by human experts.

   • The Result: The best version of MyeGPT got a score of 79.4 out of 100. It was good enough to be useful, though not perfect. Interestingly, a smaller, cheaper AI model performed almost as well as the giant, expensive ones, making it a cost-effective choice.

2. The "Replay" Test: They asked MyeGPT to recreate famous studies that had already been published by humans.

   • The Result: MyeGPT successfully reproduced the findings of two major studies. It calculated survival rates and gene expression patterns that matched the original human researchers' work almost perfectly.

3. The "Human vs. Machine" Test: They asked both MyeGPT and human experts to classify patients into risk groups (High Risk vs. Standard Risk).

   • The Result: The agreement between the AI and the humans was incredibly high (96.5%). This means the AI "thinks" very similarly to a human expert when looking at this data.

What Can It Do Right Now?

The paper shows MyeGPT handling three types of tasks:

• Simple: "How many patients took Drug X?" (One quick look-up).
• Medium: "Do patients with Gene Y respond better to Drug Z?" (Comparing two different sets of data).
• Complex: "Scan the entire genetic code of all 1,143 patients to find any gene that might be linked to treatment failure." (A massive, genome-wide search).

Important Limitations (The Fine Print)

The authors are very clear about what MyeGPT is not:

• It is not a doctor. You cannot ask it, "Should I give this specific patient this specific drug?" The paper explicitly states that MyeGPT is for research and hypothesis generation only. Because AI can make mistakes and the data comes from an observational study (not a controlled clinical trial), its advice should never be used to make real-life medical decisions for patients.
• It needs a research setting. It is designed to help scientists explore data on their laptops or even smartphones, but it is not a clinical tool for hospitals yet.

The Bottom Line

MyeGPT is a proof-of-concept tool that bridges the gap between complex, massive medical datasets and the researchers who want to use them. It turns a library that requires a PhD in computer science to navigate into a conversational chat interface. While it's not ready to replace doctors, it acts as a powerful "co-pilot" for scientists, helping them ask questions and find answers in the vast ocean of Multiple Myeloma data much faster than before.

Technical Summary

Technical Summary: MyeGPT – An AI Agent for Multiple Myeloma

Problem Statement
The MMRF CoMMpass study represents a landmark resource for multiple myeloma research, comprising comprehensive clinical and molecular data for 1,143 patients. However, the dataset's complexity—featuring 763 clinical fields alone alongside multi-omics data (WGS, WES, RNA-seq, FISH)—creates a steep learning curve for researchers. Existing tools, such as the MMRF Virtual Lab or R libraries (MMRFBiolinks, MMRFVariant), offer limited analysis scopes that cannot easily accommodate cross-modal hypothesis testing or custom queries. Furthermore, general-purpose AI agents face barriers in this domain: they lack pre-configured access to controlled-access datasets like CoMMpass, often requiring users to upload sensitive data, and they struggle with the specific field nomenclature and semantic relationships inherent to the CoMMpass schema.

Methodology
The authors developed MyeGPT (Myeloma Generative Pretrained Transformer), a specialized agentic AI system designed to function as an "AI bioinformatician" for multiple myeloma.

• Architecture: MyeGPT utilizes the ReAct (Reasoning and Acting) paradigm. It operates in alternating cycles where a Large Language Model (LLM) reasons about a user's natural language query, formulates a plan, and executes actions using specific tools.
• Data Infrastructure: The system is backed by a PostgreSQL database containing 17 interlinked tables derived from CoMMpass release IA22 (and IA16a for specific translocations). This includes clinical demographics, treatment responses, survival data, exome variants, copy number variations, and derived metrics like chromothripsis calls and mutational signatures.
• Knowledge Base: To bridge the gap between natural language queries and database schema, the authors implemented a Retrieval-Augmented Generation (RAG) system. A 52-page documentation set describing the CoMMpass fields is indexed as a text-embedding vector store. The agent retrieves relevant documentation pages based on query similarity before attempting to generate SQL queries.
• Tooling: MyeGPT employs a suite of tools:
  • Document Search: Retrieves relevant schema documentation via vector similarity.
  • LangChain SQL Query: Generates and tests SQL queries with a LIMIT clause to prevent token bloat during debugging.
  • Python SQL Query: Executes validated SQL queries in a sandboxed Python environment, saving full results to disk.
  • Python REPL: Performs statistical analysis, plotting, and calculations.
• Benchmarking: The authors constructed the CoMMpass Quantitative Test Set (CQTS), comprising 20 numeric response questions categorized by difficulty (Simple, Intermediate, Complex). Evaluation metrics included a Reasoning-Accuracy Composite (RAC) score (weighted for both method and answer correctness) and the Net Promoter Score (NPS). They also evaluated text-embedding models for document retrieval accuracy.

Key Contributions

1. Specialized Dataset Agent: MyeGPT serves as a proof-of-concept for a "dataset curator" agent that possesses deep, pre-configured knowledge of a specific, complex biomedical dataset, eliminating the need for users to upload data or manually map field names.
2. Cost Optimization Strategies: The authors detail significant engineering optimizations to reduce the cost of agentic AI. By employing RAG to retrieve only necessary documentation pages and enforcing a two-step SQL process (debugging with LIMIT followed by full execution), they reduced the cost per benchmark run from ~$10 to ~$0.10 (a ~100-fold reduction).
3. Evaluation Framework: The paper introduces a rigorous benchmarking protocol (CQTS) with scoring criteria that rewards logical reasoning steps, not just final answers, allowing for the differentiation of models that fail due to recursion limits versus those that fail due to lack of knowledge.

Results

• Benchmark Performance: In the Numeric Response Benchmark, Anthropic's Claude Opus 4.5 achieved the highest Reasoning-Accuracy Composite (mRAC) score of 0.794 and an NPS of +11, with zero zero-score responses. Among OpenAI models, the budget-friendly GPT-5 mini performed competitively (mRAC 0.681, NPS +6), offering a favorable balance of performance and cost.
• Embedding Models: For document retrieval, Google's gemini-embedding-001 and OpenAI's text-embedding-003-large (3072 dimensions) demonstrated superior performance (first-attempt accuracy ~67-68%) compared to lower-dimensional models.
• Reproduction of Known Findings:
  • Functional High Risk (FHR) Classification: When tasked with classifying patients into Standard Risk (SR), Genomic High Risk (GHR), and FHR categories, MyeGPT (using various LLMs) showed high inter-rater reliability with human experts. The highest agreement was between Claude Sonnet 4.5 and a human rater (Cohen's κ = 0.965).
  • PR Subtype Analysis: MyeGPT successfully reproduced survival analyses and differential gene expression results from the Skerget et al. (2024) CoMMpass IA22 publication. While minor discrepancies existed (attributed to batch correction methods and censored date handling), the effect sizes and p-values closely matched the original study.
• Novel Scenarios: The system successfully executed tasks ranging from simple univariate queries (e.g., distribution of treatment termination reasons) to complex genome-wide screens (e.g., identifying mutations enriched in FHR patients vs. SR patients and performing gene-wise Cox regression).

Significance and Claims
The paper positions MyeGPT not as a clinical decision-support tool, but as a research acceleration engine. The authors explicitly state that due to the non-deterministic nature of LLMs and the observational design of the CoMMpass study, the tool is restricted to exploratory data analysis and hypothesis generation.

The primary significance claimed is the ability to lower the technical barrier for myeloma researchers, enabling "on-the-go" analysis via a smartphone browser without requiring programming skills. MyeGPT demonstrates that specialized agentic AI can effectively navigate complex, multi-modal biomedical datasets, transforming natural language queries into reproducible statistical analyses and visualizations. The authors encourage the creation of similar specialized agents for other datasets (e.g., TCGA) to democratize access to large-scale biomedical data.