Generative AI-Based Virtual Assistant using Retrieval-Augmented Generation: An evaluation study for bachelor projects

1. Problem Statement

The Department of Advanced Computing Sciences (DACS) at Maastricht University faced a significant administrative bottleneck due to the expansion of its bachelor's program, which doubled the number of project groups (from 36 to 72). This increase led to an overwhelming volume of repetitive student inquiries regarding academic rules, regulations, attendance policies for sessions, and examination details.

• Core Challenge: Students often struggle to interpret complex regulations, leading to time-consuming searches or unnecessary contacts with project coordinators.
• LLM Limitations: Although Large Language Models (LLMs) can generate human-like text, they suffer from hallucinations (inventing facts), a lack of context-specific knowledge (particularly for niche rules in the academic domain), and temporal constraints (static training data may not reflect current regulations).
• Goal: Development of a Virtual Assistant (VA) that provides precise, context-aware, and reliable answers to student inquiries without overloading staff.

2. Methodology

The authors propose a Retrieval-Augmented Generation (RAG) system enhanced by Multi-Query mechanisms and a Self-Reflection loop. The architecture consists of three main phases:

A. Retrieval Pipeline

• Data Preprocessing: Documents were manually split into relevant sections to minimize noise.
• Multi-Query Retrieval: To address the difficulty students have in formulating precise questions, the system uses an LLM to generate multiple variations of the user query. This captures different perspectives and broadens the search scope.
• Vector Database & Embeddings: Documents and queries are converted into dense vector representations using embedding models (e.g., Google, OpenAI, Mistral, BGE).
• Ranking & Fusion:
  • Reciprocal Rank Fusion (RRF): Combines results from multiple query variations to re-rank documents and highlight those that consistently appear high in the lists.
  • Reranking: A Cross-Encoder model or LLM embedder is used to calculate precise similarity scores between the query and the retrieved documents.
  • Few-Shot Learning: An existing Q&A dataset from coordinators is stored in the vector store to provide the generative model with few-shot examples for correct answers.

B. Generation Pipeline

• Prompt Engineering: The system uses structured XML prompts and a low temperature setting (0.2) to reduce randomness and ensure conservative, fact-based answers.
• Output: The LLM generates an answer strictly based on the retrieved context.

C. Self-Reflection (Fallback Mechanism)

This is a novel component designed to validate the output before it reaches the user:

1. Hallucination Check: An LLM evaluates whether the generated answer is based on the retrieved facts. If hallucinations are detected, the system regenerates the answer.
2. Answer Relevance Check: The system verifies whether the answer actually resolves the user's query.
3. Query Rewriting: If the answer is unsatisfactory or the query is ambiguous, the system attempts to rewrite the query or poses clarification questions to the user.
4. Exceeding Attempts: If the loop fails after a set number of attempts, the system resorts to a safe answer or escalation.

3. Main Contributions

• Specialized RAG Architecture: The integration of Multi-Query generation and a Self-Reflection loop, specifically adapted for academic regulatory domains.
• Robust Evaluation Framework: The study employs a hybrid evaluation approach with:
  • Automated Metrics: RAGAS framework metrics (Context Precision, Context Recall, Answer Relevancy, Faithfulness) and a custom precision metric.
  • Human Evaluation: A controlled experiment with 64 bachelor's students, divided into two groups (A and B), to test the VA against human coordinators in a "crossover" design.
• Real-World Validation: Unlike many theoretical studies, this work was tested with actual students facing real academic scenarios, providing empirical data on user behavior and system utility.

4. Results

Automated Evaluation (RAGAS Metrics)

Comparison of GPT-3.5 and Gemini 1.0 Pro:

• Gemini 1.0 Pro: Slightly higher Context Precision (89% vs. 88%).
• GPT-3.5: Superior performance in Context Recall (42% vs. 41%), Answer Relevancy (57% vs. 37%), and Faithfulness (43% vs. 32%).
• Conclusion: GPT-3.5 was selected as the primary generator as it offers a better balance between relevance and factual grounding.

Human Evaluation (Student Survey)

• Improvement in Accuracy: In most scenarios, the percentage of correct answers increased when students used the VA.
  • Example (Scenario 1): Correct answers rose from 32.4% (without VA) to 46.7% (with VA).
  • Example (Scenario 6): Correct answers rose from 50.0% to 73.5%.
• Reduction in Uncertainty: Responses with "I don't know" decreased significantly when using the VA (e.g., Scenario 2 dropped from 32.4% to 6.7%).
• Exceptions: Scenarios 5 and 7 showed a performance decline. Analysis revealed this was due to missing information in the training data (distinction between sessions and examinations) and unforeseen edge cases not covered by the rules.
• User Satisfaction:
  • Helpfulness: Rated highly for most scenarios (average 3.5–4.5/5).
  • Response Time: Average of 10.045 seconds (SD 2.39 s), which was perceived as acceptable by students.
  • Preference: Despite the VA's usefulness, 71% of students initially preferred contacting a human coordinator, indicating a trust gap that the VA aims to bridge over time.

5. Significance and Future Work

• Educational Policy Implications: The VA successfully demonstrated the potential to reduce the administrative workload for staff while enabling students to have immediate, precise access to complex regulations.
• Technical Insights: The study highlights that while RAG improves accuracy, data quality (manual splitting, comprehensive training data) is critical. Gaps in source documents lead directly to system failures.
• Identified Limitations:
  • Inability to access personal student data (e.g., specific coordinator assignment) without integration into the Learning Management System (LMS).
  • Difficulties in distinguishing similar concepts (e.g., "skill courses" vs. "lectures").
  • Challenges with long context inputs.
• Future Directions:
  • Integration with the university LMS for personalized answers.
  • Implementation of Reverse HyDE and contrastive learning to improve retrieval.
  • Use of semantic entropy for more robust hallucination detection.
  • Addition of specific templates (e.g., for force majeure) and FAQs based on student feedback.

In summary, the paper validates that a well-structured RAG system with self-correction mechanisms can effectively serve as a specialized educational tool, provided the underlying knowledge base is rigorous and the system is continuously refined based on real-world feedback.