💬 NLP

Joint Optimization of Reasoning and Dual-Memory for Self-Learning Diagnostic Agent

The paper proposes SEA, a self-learning diagnostic agent featuring a cognitively inspired dual-memory module and a reinforcement training framework that jointly optimizes reasoning and memory management to significantly outperform baselines in diagnostic accuracy and continual learning by effectively transforming clinical experience into reusable knowledge.

Original authors: Bingxuan Li, Simo Du, Yue Guo

Published 2026-04-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bingxuan Li, Simo Du, Yue Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a new doctor.

In the old way of doing things with AI, we would just feed the computer a giant textbook of medical facts and say, "Memorize this." If the computer gets a diagnosis right, we say "Good job." If it gets it wrong, we say "Try again." But here's the problem: real doctors don't just memorize textbooks. They get better because they see patients. They remember, "Oh, last week I saw a guy with a headache and a rash, and it turned out to be something rare. I should keep that in mind."

The paper you shared introduces SEA (Self-Learning Diagnostic Agent). Think of SEA not as a robot that just reads books, but as a smart medical apprentice that learns how to learn.

Here is how it works, broken down into simple ideas:

1. The Two-Notebook System (Dual-Memory)

Most AI models have a short-term memory (like a sticky note) that they forget as soon as the conversation ends. SEA is different. It has two notebooks, inspired by how human brains work:

The "Sticky Note" Notebook (Short-Term Memory): This holds the last few patients the doctor just saw. It's very specific. "Patient A had a fever and a cough yesterday." This helps the doctor stay grounded in the immediate situation.
The "Rule Book" Notebook (Long-Term Memory): This is the magic part. When the "Sticky Note" notebook gets full, SEA doesn't just throw the old notes away. Instead, it summarizes them. It asks: "What did I learn from these patients?" and writes a general rule in the Rule Book.
- Example: Instead of remembering every single patient with a specific rash, it writes a rule: "If a patient has a rash AND a fever AND no cough, check for Disease X."

This is like a chef who doesn't just remember every single meal they cooked, but writes down a perfect recipe card so they can cook it again later without tasting every ingredient from scratch.

2. The Coach Who Gives Two Types of Feedback (Joint Optimization)

Usually, when we train AI, the coach only cares about the final score: "Did you guess the disease correctly? Yes/No."

SEA has a smarter coach. This coach gives feedback on two things at once:

The Diagnosis: "Did you get the right answer?"
The Memory Management: "Did you write down a useful rule? Did you throw away a useless note?"

If the AI just dumps all its notes into the long-term memory without thinking, the coach says, "Stop! That's too much clutter. Summarize it first!" If the AI forgets a crucial detail, the coach says, "Write that down!"

This forces the AI to learn how to organize its experience, not just how to guess the answer.

3. The "Cold Start" Problem

Imagine you are a new doctor on your first day. You have no experience. If we only reward you for getting the diagnosis right immediately, you might panic because you don't know anything yet.

SEA's training is like a video game with levels:

Early Levels: The coach cares mostly about you collecting notes and writing rules. "Don't worry about being perfect yet; just learn the patterns!"
Later Levels: Once you have a good Rule Book, the coach starts caring more about getting the diagnosis right.

This helps the AI grow from a confused beginner into an expert without getting stuck.

4. The Results: Why It Matters

The researchers tested this "Apprentice" (SEA) against other AI models:

The "Textbook Only" models: They were okay, but they couldn't learn from new experiences.
The "Naive Memory" models: These tried to remember everything but got confused by too much information (like a student trying to memorize the whole library instead of the key chapters).
SEA: It crushed the competition.
- In standard tests, it was 20% more accurate than the best previous models.
- In long-term tests (where it sees hundreds of patients one after another), it kept getting better and better, while others stayed the same or even got worse.

The Bottom Line

The paper shows that to build a truly smart medical AI, we can't just make it smarter at reading. We have to teach it how to remember.

SEA is like a doctor who keeps a journal, writes down lessons learned from every case, and turns those lessons into simple rules. It doesn't just "know" medicine; it learns from experience, just like a human expert does. This makes it safer, more reliable, and capable of handling rare or tricky diseases that textbooks might not cover perfectly.

1. Problem Statement

Current Large Language Model (LLM) based diagnostic agents face two critical limitations:

Independent Case Processing: Most approaches treat clinical cases as isolated events, failing to accumulate experience or adapt over time. They lack the ability to learn from past successes and failures, unlike human clinicians who develop "experience-shaped cognitive structures."
Ineffective Memory Management: While general-purpose memory mechanisms exist, they struggle in the medical domain. Clinical cases are information-dense and noisy. Naïve storage of every past case is infeasible due to context limits, while aggressive compression risks losing rare but critical diagnostic signals.
The Gap: Existing methods focus on teaching agents what to diagnose (acquiring knowledge) rather than how to improve through accumulated experience. There is a lack of frameworks that jointly optimize diagnostic reasoning with the management of long-term memory (abstraction of rules) and short-term memory (recent cases).

2. Methodology: The SEA Framework

The authors propose SEA (Self-learning diagnostic agent with dual-memory), a system designed to mimic human cognitive learning through a reinforcement learning (RL) framework.

A. Dual-Memory Architecture

Inspired by human cognition, SEA utilizes two coupled memory clusters:

Short-Term Memory ( $M^S_t$ ): Stores a bounded set ( $K$ ) of recent, concrete patient cases with outcomes (symptoms, predictions, feedback). It operates on an append-only basis. When capacity is reached, cases are evicted.
Long-Term Memory ( $M^L_t$ ): Stores abstract diagnostic rules distilled from the evicted short-term cases. These rules represent reusable heuristics (e.g., symptom-disease associations, failure patterns) rather than verbatim case logs.

B. Agent Actions and Rollout

The agent does not just output a diagnosis; it explicitly chooses memory operations as part of its action space:

list: Retrieve current memory contents.
append: Add a new case to short-term memory.
pop: Evict specific cases when capacity is near limit.
consolidate: Summarize evicted cases into abstract rules and insert them into long-term memory.

C. Reinforcement Learning Framework

SEA is trained using Group Relative Policy Optimization (GRPO) with a specialized reward structure:

Joint Reward Function: The total reward $r_t$ $r_{t}$ is a weighted sum of a Diagnostic Reward ( $r_{diag}$ $r_{d ia g}$ ) and a Memory Reward ( $r_{mem}$ $r_{m e m}$ ).
- Diagnostic Reward: High positive reward for correct diagnosis, negative for incorrect.
- Memory Reward: Penalizes the size of the short-term memory cluster to encourage timely eviction and consolidation, preventing uncontrolled growth.
Round-Wise Reward Shaping: To address the "cold-start" problem in long-horizon tasks, the reward weights are dynamically scheduled. Early rounds emphasize memory formation (learning to store/consolidate), while later rounds emphasize diagnostic accuracy.
Advantage Estimation: The framework computes advantages separately for each round position to handle the non-stationary nature of the shaped rewards, ensuring stable gradient updates.

3. Key Contributions

Dual-Memory Design: A novel architecture that separates concrete case storage from abstract rule induction, allowing the agent to balance immediate context with generalized knowledge.
Joint Optimization: The first framework to jointly optimize diagnostic reasoning and memory management decisions via RL, rather than treating memory as a passive retrieval tool.
Self-Learning Mechanism: The agent learns to decide when to evict cases and how to distill them into rules, transforming raw experience into reusable knowledge without modifying the base model's weights during deployment.
Comprehensive Evaluation: Evaluation across both static (standard accuracy) and dynamic (long-horizon streaming) settings, including expert validation of the induced rules.

4. Experimental Results

A. Standard Evaluation (MedCaseReasoning Dataset)

Setting: Static diagnostic competence where cases are solved independently.
Performance: SEA (Qwen-8B) achieved 92.46% accuracy, outperforming the strongest baseline (SFT + Memory) by +19.6%.
Insight: Joint optimization significantly outperforms outcome-only RL or naïve memory integration. Naïve memory integration (ReAct + Short-term) actually degraded performance compared to zero-shot, highlighting the need for structured management.

B. Long-Horizon Evaluation (ER-Reason Dataset)

Setting: Streaming deployment where the agent receives feedback after each case and must adapt over time.
Performance:
- Final Accuracy: SEA achieved 0.7214, the highest among all methods.
- Improvement: SEA showed the largest gain over 100 rounds (+0.35 $\Delta$ Acc@100), whereas baselines showed marginal gains or degradation.
Insight: Effective adaptation requires explicit mechanisms for experience accumulation and reuse; standard training alone cannot sustain improvement in streaming environments.

C. Expert Evaluation

Method: A professional physician evaluated the induced rules and reasoning traces.
Scores (5-point Likert):
- Overall Trust: 4.216
- Rule Usefulness: 3.946
- Clinical Correctness: 3.865
Conclusion: The induced rules are clinically sound, reliable, and practically meaningful, confirming that the agent learns valid medical heuristics rather than superficial patterns.

5. Significance

Paradigm Shift: Moves beyond static "knowledge acquisition" to continual, experience-driven learning for medical agents.
Deployment Viability: Demonstrates that agents can improve "on-the-job" during deployment without requiring expensive retraining of base model parameters, addressing privacy and safety constraints in healthcare.
Interpretability: By explicitly generating and storing diagnostic rules, SEA offers a transparent mechanism for understanding how the agent learns, bridging the gap between black-box LLMs and clinical trust.
Generalizability: The dual-memory module acts as a plug-and-play component that improves performance across different backbone models (Qwen, GPT-5.2), suggesting a scalable solution for long-horizon agentic tasks.

In summary, SEA proves that by structurally separating memory management from reasoning and training them jointly via reinforcement learning, diagnostic agents can achieve human-like continual learning, significantly outperforming current state-of-the-art methods in both accuracy and adaptability.