MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution

1. Problem Statement

Current Medical Vision-Language Models (VLMs) face a fundamental cognitive misalignment with clinical expertise.

• Discrete Token Limitation: Standard VLMs rely on discrete tokens for reasoning (e.g., Chain-of-Thought or CoT). This leads to quantization loss, inability to represent continuous pathological features (e.g., gradual lesion density changes), and information dissipation over long reasoning chains.
• Static External Dependencies: Existing solutions like Retrieval-Augmented Generation (RAG) or prompt injection rely on static external knowledge that is not causally verified or dynamically adapted to the specific image context.
• Hallucinations: The autoregressive nature of discrete tokens often leads to "pseudo-logical" hallucinations where models generate plausible-sounding but factually incorrect reasoning chains that lack grounding in physical visual evidence.
• Goal: The authors aim to simulate the Implicit Diagnostic Memory of human experts—where diagnosis is an instant, experience-driven pattern recognition process—by evolving a continuous, latent memory within the model rather than relying on verbose, explicit text-based reasoning.

2. Methodology: MedSynapse-V Framework

MedSynapse-V introduces a three-stage progressive training paradigm to evolve Latent Diagnostic Memory. The framework shifts from external knowledge injection to autonomous internalization.

Stage I: Meta Query for Prior Memorization (MQPM)

• Objective: Condense structured anatomical priors into compact implicit memory vectors.
• Mechanism:
  • A frozen Anatomical Encoder (MedSAM3) extracts spatial features from the input image.
  • A learnable Diagnostic Memory Sampler ($P_\phi$) uses $N$ learnable Meta-Query Probes to attend to specific pathological patterns (e.g., boundaries, textures) within the encoder's feature map.
  • These probes aggregate features into a set of $N$ compact Diagnostic Implicit Memory vectors ($\mathcal{M}$).
  • These vectors are injected directly into the VLM's hidden state stream (between question encoding and answer generation).
• Warmup: A semantic alignment warmup phase ensures the injected memory vectors are semantically coherent with the VLM's hidden space before reinforcement learning begins.

Stage II: Causal Counterfactual Refinement (CCR)

• Objective: Optimize the memory vectors to ensure they contain only causally relevant diagnostic information, pruning redundancies.
• Mechanism:
  • Uses Group Relative Policy Optimization (GRPO) to refine the policy.
  • Composite Reward Function:
    1. Accuracy Reward ($r_{acc}$): Standard correctness of the final diagnosis.
    2. Causal Counterfactual Reward ($r_{causal}$): The core innovation. The model generates a counterfactual scenario by masking diagnostically critical regions (using region masks from the anatomical encoder) to create an "intervened" memory ($\mathcal{M}'$). The reward is the log-likelihood difference between the original output and the output generated with the intervened memory.
  • Effect: If the model's performance drops significantly when critical regions are masked, the memory is deemed causally relevant. This forces the model to rely on genuine visual evidence rather than spurious correlations or shortcuts.

Stage III: Intrinsic Memory Transition (IMT)

• Objective: Internalize the diagnostic capability so the model can generate memory autonomously at inference, removing the need for the heavy anatomical encoder.
• Mechanism:
  • Privileged-Autonomous Dual-Branch Distillation:
    • Teacher Branch: Uses the full pipeline (Anatomical Encoder + Sampler) to generate "privileged" memory ($\mathcal{M}_{pri}$).
    • Student Branch: Uses a lightweight Autonomous Memory Module ($A_\psi$) that takes only the VLM's own visual encoding features to generate "intrinsic" memory ($\mathcal{M}_{auto}$).
  • Training Objective: Minimizes the Jensen-Shannon Divergence (JSD) between the next-token distributions of the Teacher and Student branches across the full vocabulary.
  • Result: At inference, the anatomical encoder and sampler are removed. The model generates its own diagnostic memory vectors with negligible computational overhead, achieving near-lossless performance compared to the encoder-dependent version.

3. Key Contributions

1. First Latent Memory Evolution Framework: Proposes the first method to evolve diagnostic implicit memory in the latent space for medical VLMs, moving away from static external knowledge to progressive, autonomous internalization.
2. Causal Counterfactual Refinement (CCR): Introduces a novel reinforcement learning strategy using counterfactual interventions to quantify and enforce the causal contribution of memory elements, ensuring clinical fidelity.
3. Intrinsic Memory Transition (IMT): Develops a privileged-autonomous distillation paradigm that eliminates auxiliary modules (like heavy segmentation encoders) at inference while retaining expert-level diagnostic patterns.
4. Superior Efficiency and Accuracy: Demonstrates that compact latent memory can replace verbose Chain-of-Thought reasoning, achieving higher accuracy with significantly lower latency and token counts.

4. Experimental Results

The framework was evaluated on seven medical multimodal benchmarks (including VQA-RAD, SLAKE, PathVQA, MMMU Health, and MedXpertQA-MM).

• Performance:
  • MedSynapse-V (8B) achieved an average accuracy of 61.4% (with encoder) and 59.6% (encoder-free/IMT), outperforming all baselines.
  • It surpassed the strongest RL-based CoT baseline (MMedExpert-R1, 55.7%) by +3.9 percentage points without any auxiliary modules at inference.
  • It showed the largest gains on visual-grounding benchmarks (e.g., +9.0% on VQA-RAD), where discrete CoT tokens typically fail due to information dissipation.
• Efficiency:
  • Latency: MedSynapse-V (IMT) generates diagnoses in ~2.6 seconds per sample, comparable to zero-shot Qwen3-VL-8B (2.8s) and significantly faster than CoT methods (5.2–5.8s).
  • Token Count: It produces concise, correct answers in ~34–44 tokens, whereas CoT baselines generate 185–238 tokens of verbose reasoning, often containing hallucinations.
• Ablation Studies:
  • Removing the causal reward ($r_{causal}$) caused performance to drop by 4.1 pp, confirming that causal calibration is essential to prevent the model from treating memory as inert padding.
  • Removing the MQPM warmup caused a collapse to near zero-shot performance, highlighting the need for semantic alignment.
  • The t-SNE visualizations showed that the evolved memory space forms distinct, clinically coherent clusters for different modalities and disease subtypes.

5. Significance

• Paradigm Shift: MedSynapse-V challenges the prevailing "Chain-of-Thought" paradigm in medical AI. It demonstrates that continuous latent reasoning guided by causal rewards is more effective, efficient, and robust than discrete token-based reasoning for complex diagnostic tasks.
• Clinical Reliability: By enforcing causal alignment through counterfactual interventions, the model reduces hallucinations and ensures diagnoses are grounded in specific visual evidence, a critical requirement for clinical deployment.
• Deployment Viability: The Intrinsic Memory Transition mechanism allows the model to operate as a standard, lightweight VLM at inference (no external encoders or RAG), making it highly practical for real-world medical applications where latency and resource constraints are paramount.

In summary, MedSynapse-V successfully bridges the gap between static visual features and dynamic clinical intuition by evolving a compact, causally verified, and self-contained diagnostic memory within the model's latent space.