💬 NLP

Evaluating Prompt Engineering Techniques for RAG in Small Language Models: A Multi-Hop QA Approach

This paper presents a large-scale empirical study demonstrating that optimizing prompt engineering techniques significantly enhances the multi-hop reasoning performance of Small Language Models in Retrieval Augmented Generation systems, achieving up to 84.5% gains on the HotpotQA dataset compared to standard approaches.

Original authors: Amir Hossein Mohammadi, Ali Moeinian, Zahra Razavizade, Afsaneh Fatemi, Reza Ramezani

Published 2026-02-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Amir Hossein Mohammadi, Ali Moeinian, Zahra Razavizade, Afsaneh Fatemi, Reza Ramezani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly young, assistant named Alex. Alex is great at chatting and knows a lot of general facts, but if you ask him a tricky question that requires connecting dots from three different books, he might get confused or make things up. This is what happens with Small Language Models (SLMs) in the world of AI. They are fast and cheap to run, but they struggle with complex puzzles.

To help Alex, you give him a library of books (external knowledge) to look at before answering. This setup is called RAG (Retrieval-Augmented Generation). But here's the catch: How you ask Alex to use those books matters just as much as the books themselves.

This paper is like a massive cooking competition where the researchers tested 24 different ways of giving instructions (prompts) to two different versions of Alex (the AI models Qwen and Gemma) to see which instruction style helped them solve the hardest puzzles (Multi-Hop Questions) best.

Here is the breakdown of their findings, using some everyday analogies:

1. The Problem: The "Overwhelmed Intern"

Think of the AI models as interns.

The Big Models are like senior experts who can figure out complex problems on their own.
The Small Models (SLMs) are like bright but junior interns. They are fast and cheap, but if you just hand them a stack of papers and say, "Find the answer," they might get lost in the noise or miss the connection between two facts.

The researchers wanted to know: What is the best way to tell the intern how to read those papers?

2. The Experiment: 24 Different "Instruction Manuals"

The team didn't just guess. They tested 24 different "instruction manuals" (prompts) on the HotpotQA dataset (a test full of tricky questions that require linking facts, like "Who is the wife of the actor who starred in the movie directed by the person who won the Oscar for Best Director?").

They grouped the instructions into three types:

The Standard Way: "Here are the books, give me an answer." (The Baseline)
The "Textbook" Way: Instructions taken from existing AI research (like "Think step-by-step" or "Check your work").
The "New Hybrid" Way: Custom instructions the researchers invented, mixing different techniques to create a super-guide for the intern.

3. The Results: Speed vs. Accuracy (The "Fast Food vs. Fine Dining" Dilemma)

The study revealed a classic trade-off, like choosing between a fast-food burger and a slow-cooked gourmet meal.

Option A: The "Fast Food" Prompts (Literature-Derived)

What they are: Short, simple instructions like "Just give me the answer."
The Result: These were super fast. The AI answered in about 1 second.
The Catch: The answers were okay, but not perfect. It's like grabbing a quick burger; it fills you up, but it's not the best meal possible.
Best for: Chatbots where you need an instant reply, even if it's slightly less accurate.

Option B: The "Gourmet" Prompts (Novel Hybrid)

What they are: Long, detailed instructions that tell the AI: "First, read this paragraph. Then, find the connection to that paragraph. Then, double-check your logic. Finally, synthesize the answer."
The Result: These produced much better answers (up to 6% more accurate). The AI got the complex puzzles right.
The Catch: It took 8 to 10 times longer to get the answer. The AI was "thinking" hard, which slowed things down.
Best for: Medical diagnosis, legal research, or scientific discovery where being right is more important than being fast.

4. The Twist: Not All Interns Are the Same

The researchers tested two different AI models, and they behaved differently:

Model A (Qwen 2.5 - 3B): This is the "Junior Intern." It needed a lot of hand-holding. To get the best results, you had to give it the long, detailed "Gourmet" instructions. It couldn't figure out the complex puzzle on its own.
Model B (Gemma 3 - 4B): This is the "Senior Intern." It is slightly bigger and smarter.
- The Surprise: For this model, the researchers found a "Magic Prompt" called Expert Synthesis.
- The Magic: They told the model: "You are an expert. Synthesize the information and give me the best answer."
- The Result: This model was both fast AND accurate. It didn't need the long, step-by-step hand-holding. It could "just know" what to do. This broke the usual rule that says "you have to be slow to be smart."

5. The Big Takeaways (The "Cheat Sheet")

If you are building an AI system, here is what this paper tells you to do:

There is no "One Size Fits All": You can't just copy-paste a prompt from one AI to another. What works for the "Junior Intern" (Qwen) might be too confusing or too slow for the "Senior Intern" (Gemma).
Choose your goal:
- Need Speed? Use the short, simple prompts.
- Need Perfection? Use the long, complex "Hybrid" prompts.
Bigger isn't always just "smarter," it's "different": A slightly larger model (Gemma) handled complex instructions much more efficiently than the smaller one, sometimes even beating the "slow and accurate" rule.
Don't rely on simple math: The researchers found that standard computer tests (like counting matching words) were bad at judging these answers. They had to use a "Judge AI" (GPT-4o-mini) to read the answers and give them a human-like grade based on logic and truth.

In Summary

This paper is a guidebook for anyone trying to use small, efficient AI models for hard tasks. It proves that how you ask the question is just as important as the AI itself. By tweaking the instructions, you can turn a fast-but-dumb AI into a slow-but-genius one, or find the sweet spot where a smarter AI can be both fast and genius.

The Bottom Line: If you want the best results, don't just throw data at the AI. Give it a clear, well-structured map on how to navigate that data, and pick the map that fits the specific "brain" you are using.

1. Problem Statement

While Retrieval-Augmented Generation (RAG) is well-studied for Large Language Models (LLMs), its optimization for Small Language Models (SLMs) remains a critical research gap. SLMs (e.g., 3B–4B parameter models) are computationally efficient but struggle with complex reasoning tasks, particularly Multi-Hop Question Answering (QA), which requires synthesizing information from multiple disjointed documents.

The core problem addressed is that standard RAG prompts often fail to guide SLMs through the necessary reasoning steps to connect facts across retrieved documents. While prompt engineering is known to influence performance, there is a lack of systematic, large-scale empirical data on:

Which specific prompt strategies work best for SLMs in multi-hop scenarios.
The trade-offs between accuracy, latency, and efficiency when using advanced prompting techniques on resource-constrained models.
Whether "one-size-fits-all" prompting strategies exist or if they are model-specific.

2. Methodology

The authors conducted a large-scale empirical study involving 18,720 test instances from the HotpotQA dataset (a benchmark for multi-hop QA).

Experimental Setup

Models Tested: Two state-of-the-art SLMs:
- Qwen2.5-3B-Instruct (3 billion parameters)
- Gemma3-4B-It (4 billion parameters)
Prompt Variants: A total of 24 distinct prompt templates were evaluated, categorized into:
1. Baseline: 1 Standard RAG prompt (standard_context_aware).
2. Literature-Derived: 9 established techniques (e.g., Chain-of-Thought, Few-Shot, Role-Playing, Self-Refine).
3. Novel Hybrid: 14 custom-designed prompts combining techniques like decomposition, hierarchical synthesis, and expert personas specifically for SLMs.
Pipeline: The RAG pipeline used pre-retrieved documents from the RAGBench dataset to isolate the impact of prompt engineering from retrieval quality. The generator (SLM) produced answers, which were then evaluated.

Evaluation Metrics

The study moved beyond simple string matching (which failed to capture semantic correctness in reasoning-heavy tasks) by employing a Multi-Dimensional Evaluation Framework:

Answer Quality ( $S_{LLM}$ ): Instead of RAGAS (which relied on lexical overlap), the authors used an LLM-as-a-Judge approach (using GPT-4o-mini). The judge scored answers on five weighted dimensions:
- Semantic Alignment (35%)
- Information Accuracy (25%)
- Logical Coherence (20%)
- Response Completeness (15%)
- Practical Utility (5%)
Processing Time: Measured as wall-clock latency (seconds).
Efficiency Score: A composite metric defined as $Efficiency = S_{LLM} / Processing Time$ , balancing quality against speed.

3. Key Contributions

Novel Taxonomy of Prompt Strategies: The paper introduces a comprehensive taxonomy of 24 prompts, including 14 novel hybrid variants designed specifically to scaffold multi-hop reasoning in SLMs.
Rigorous Multi-Dimensional Analysis: It provides a holistic view of performance, quantifying the trade-offs between accuracy, latency, and efficiency, rather than focusing solely on accuracy.
Model-Specific Insights: The study demonstrates that optimal prompting is highly dependent on the underlying model architecture and parameter count, debunking the "plug-and-play" assumption.
Actionable Guidelines: It offers a practical decision matrix for practitioners to select prompts based on their specific constraints (e.g., real-time latency vs. high-stakes accuracy).

4. Key Results

The study revealed significant performance gains and distinct behavioral patterns between the two models.

Performance Gains

Accuracy Improvements: Novel hybrid prompts improved accuracy significantly over the standard baseline:
- Qwen2.5-3B: Increased from 76.8% (baseline) to 83.0% (best hybrid).
- Gemma3-4B: Increased from 78.5% (baseline) to 84.6% (best hybrid).
Efficiency Trade-off: Generally, higher accuracy came at a steep cost. Novel hybrid prompts increased processing time by 8x to 10x compared to the baseline (e.g., from ~1.2s to ~10s on Qwen).

Model-Specific Findings

Qwen2.5-3B (Smaller Model):
- Struggled with verbose reasoning chains.
- Best for Accuracy: optimized_multi_hop_prompting (83.0%).
- Best Balance: multi_hop_compact_prompting offered near-top accuracy (82.5%) with more manageable latency.
- Best for Speed: instruction_tuned (literature-derived) was the efficiency champion.
Gemma3-4B (Larger Model):
- Demonstrated superior performance across all metrics due to larger parameter count and architecture.
- Breakthrough Finding: The prompt expert_synthesis_prompting achieved the highest accuracy (84.6%) while simultaneously being the fastest (6.98s) and most efficient among all novel prompts. This suggests that for capable SLMs, high-level "expert persona" instructions are more effective than explicit, step-by-step Chain-of-Thought, breaking the typical accuracy-latency trade-off.

Comparative Analysis

Literature-Derived Prompts: Consistently outperformed in speed and efficiency (e.g., instruction_tuned), making them ideal for real-time applications.
Novel Hybrid Prompts: Consistently outperformed in accuracy, making them suitable for high-stakes domains (medical, legal, scientific) where precision outweighs latency.

5. Significance and Implications

This research provides a critical roadmap for deploying RAG systems in resource-constrained environments:

Strategic Prompt Selection: It proves that prompt engineering is not just a minor tweak but a decisive factor in SLM performance. The choice of prompt must be dictated by the specific model and the application's tolerance for latency.
Breaking the Trade-off: The discovery that expert_synthesis_prompting works exceptionally well on Gemma3-4B suggests that as SLMs scale, they may not require verbose, explicit reasoning scaffolding, allowing for both high accuracy and low latency.
Practical Deployment: The paper offers a clear guide for practitioners:
- Use short, literature-derived prompts for real-time chatbots and edge devices.
- Use complex hybrid prompts for offline, high-precision tasks.
- Tailor prompts to the model: What works for a 3B model may not work for a 4B model.

In conclusion, the paper establishes that while SLMs have inherent reasoning limitations, strategic prompt engineering can unlock latent capabilities, enabling them to compete with larger models in complex multi-hop reasoning tasks when the right prompt architecture is applied.