Domain Fine-Tuning vs. Retrieval-Augmented Generation for Medical Multiple-Choice Question Answering: A Controlled Comparison at the 4B-Parameter Scale

1. Problem Statement

Practitioners deploying small, locally-runnable Large Language Models (LLMs) for medical applications face a critical design decision: should they invest resources in domain fine-tuning (encoding knowledge into model weights) or Retrieval-Augmented Generation (RAG) (injecting knowledge at inference time)?

While both approaches are known to improve performance individually, they are rarely compared head-to-head under identical conditions. Existing studies often vary in backbone architecture, evaluation datasets, and retrieval pipelines, making it difficult to determine which intervention offers the best accuracy gain for a fixed parameter budget. This paper addresses the specific question: At a small scale (4B parameters), which strategy—fine-tuning or RAG—delivers superior accuracy for medical multiple-choice question answering?

2. Methodology

The authors conducted a strictly controlled 2×2 experimental design to isolate the variables of interest.

• Experimental Factors:
  1. Model Type: General-purpose (Gemma 3 4B) vs. Domain-adapted (MedGemma 4B).
  2. Context Strategy: No RAG (Question only) vs. RAG (Question + Retrieved Passages).
• Controlled Variables:
  • Model Size: Both models are 4B parameters.
  • Quantization: Both served as 4-bit quantized GGUF builds via Ollama.
  • Temperature: Fixed at 0.1 for deterministic sampling.
  • Prompting: Identical medical-expert persona and machine-readable answer contract ({"answer":"A|B|C|D"}).
  • Evaluation Set: The full MedQA-USMLE 4-option test split (1,273 questions).
  • Repetitions: Each question was queried 3 times per setup (Total: 15,276 LLM calls).
• Retrieval Pipeline:
  • Corpus: Explanation fields from the MedMCQA training split (ensuring no data leakage, as MedMCQA is disjoint from MedQA).
  • Embedding: nomic-embed-text.
  • Ranking: Hybrid scoring (60% dense cosine similarity + 40% lexical) with a cosine distance cutoff of 0.3.
  • Context: Top 3 chunks inserted into the prompt.
• Statistical Analysis:
  • Aggregation: Majority vote across the 3 repetitions.
  • Significance Testing: McNemar's exact paired test to compare accuracy between setups on the same 1,273 items.

3. Key Contributions

1. Controlled Head-to-Head Comparison: The first study to hold the backbone, prompt, and evaluation set constant while toggling both fine-tuning and RAG at the 4B scale.
2. Statistical Evidence: Provides rigorous statistical proof (via McNemar's test) that domain fine-tuning yields significant gains, whereas a competitive RAG pipeline does not at this scale.
3. Reproducible Pipeline: Released a fully open-source, deterministic pipeline (Ollama + ChromaDB) with 15,276 raw JSONL traces to support replication.

4. Results

The study evaluated four configurations:

1. Gemma 3 4B (General) + No RAG: 46.4% accuracy.
2. Gemma 3 4B (General) + RAG: 47.3% accuracy.
3. MedGemma 4B (Domain) + RAG: 51.4% accuracy.
4. MedGemma 4B (Domain) + No RAG: 53.3% accuracy (Best Performer).

Key Findings:

• Impact of Fine-Tuning: Switching from the general Gemma 3 backbone to the domain-adapted MedGemma backbone resulted in a +6.8 percentage point (pp) gain in accuracy (53.3% vs. 46.4%). This difference is highly statistically significant (p < 10⁻⁴).
• Impact of RAG: Adding RAG did not produce a statistically significant improvement for either model.
  • For the general model, the gain was marginal (+0.9 pp, p = 0.56).
  • For the domain-tuned model, the point estimate was actually slightly negative (-1.9 pp, p = 0.16), suggesting RAG may introduce noise or interference when the model already possesses the relevant knowledge.
• Consistency: All setups showed high consistency across repetitions (≥0.99 agreement), indicating that the results are robust against decoding noise.

5. Significance and Discussion

• Weights vs. Context: At the 4B parameter scale, domain knowledge encoded in the weights dominates domain knowledge supplied in the context. For small, locally deployed models, fine-tuning is a more reliable lever for accuracy than building a retrieval pipeline.
• Why RAG Failed: The authors propose four potential reasons for RAG's lack of efficacy in this specific context:
  1. Reasoning vs. Retrieval: USMLE questions require complex reasoning and integration of findings, which a retrieved text snippet cannot easily shortcut.
  2. Corpus Quality: The retrieval corpus (MedMCQA explanations) is broad but not authoritative, potentially diluting the model's prior knowledge.
  3. Capacity Limits: 4B models may lack the context window capacity or reasoning ability to effectively ground and integrate retrieved chunks while solving complex problems.
  4. Knowledge Redundancy: MedGemma likely already absorbed the textbook knowledge present in the retrieval corpus, leaving little marginal value for RAG to add, and potentially causing interference.
• Practical Implication: For practitioners with limited engineering budgets deploying small medical LLMs, prioritizing domain fine-tuning over RAG is the recommended strategy. RAG is not harmful but is not a substitute for in-weight knowledge at this scale.

6. Limitations

• Single Benchmark: Results are specific to MedQA-USMLE; RAG might perform better on open-book or verbatim retrieval tasks.
• Single Scale: Findings apply to 4B models and may not extrapolate to larger frontier models (e.g., 70B+) where RAG gains are often more pronounced.
• Quantization: Both models were 4-bit quantized, which may introduce slight accuracy shifts, though the paired comparison isolates the relative effects.