A Locally Deployed RAG-Based Academic Advising System for Course Selection

This paper proposes a locally deployed, privacy-preserving RAG-based system that leverages large language models and structured syllabus data to assist students in overcoming information overload and institutions in addressing resource limitations by generating optimal, prerequisite-aware course sequences for personalized academic planning.

The Gist

Imagine you are a student trying to build the perfect schedule for your degree. You have a massive library of course descriptions (syllabi) in front of you. The problem? There are so many books, and they are written in a confusing way. You don't know which class you need to take before another, or if a specific class fits your goals. Trying to find the answers yourself is like looking for a needle in a haystack while wearing blindfolds. Meanwhile, the human advisors who usually help are overwhelmed, tired, and can't talk to everyone at once.

This paper introduces a solution called Syllabot. Think of Syllabot as a super-smart, privacy-focused librarian who lives entirely on your own computer (not on the internet). Its job is to read all those course syllabi and answer your questions about which classes to take, without ever sending your personal data to a cloud server.

Here is how the paper breaks down the system and what it found, using simple analogies:

1. The Problem: The "Overwhelmed Student" vs. The "Busy Advisor"

Students often get stuck because they have too much information and don't know how to connect the dots. They might pick a class just because the title sounds cool, missing the fact that they need a prerequisite class first. Human advisors want to help, but they are stretched too thin.

2. The Solution: Syllabot (The Local Librarian)

The researchers built a system that combines two things:

• A Search Engine: It scans the syllabus documents to find the exact pages you need.
• A Local Brain (LLM): It reads those pages and writes a clear answer for you.

The "Local" Part: Usually, AI systems send your questions to big servers in the cloud. Syllabot runs entirely on the university's own computers. This is like having a private tutor in your own study room rather than calling a stranger on a public phone line. It keeps your data safe and private.

3. How the Search Works (The Three Strategies)

The researchers tested three different ways for the system to find the right information, similar to how you might search for a book:

• The Keyword Hunter (BM25): This looks for exact word matches. If you ask, "What is the textbook for Math 101?", it finds the page that literally says "Math 101" and "textbook." It's great for specific terms but bad if you ask the same question in a different way.
• The Meaning Matcher (Semantic/Embedding): This understands the idea behind your words. If you ask, "What book do I need for the intro math class?", it knows that "intro math class" means "Math 101" even if those exact words aren't there. It's like a librarian who understands your intent, not just your vocabulary.
• The Hybrid Approach: This tries to use both the Keyword Hunter and the Meaning Matcher at the same time, hoping to get the best of both worlds.

4. The Experiments: Testing the Librarian

To see which search method worked best, the researchers created a "test drive" with 100 questions. They categorized the questions into four types:

• The Easy One: "What is the textbook?" (Exact words match).
• The Paraphrase: "What book do I need?" (Same meaning, different words).
• The Puzzle: "What do I need to know before taking Class A and Class B?" (Requires finding info in multiple places).
• The Trap: "Can I take a class that doesn't exist?" (The system should say "I don't know").

5. What They Found

• The "Meaning Matcher" was the MVP: Surprisingly, the system that understood the meaning of the questions (Semantic) worked better than the one that just looked for exact words (Keyword), even for simple questions. The "Hybrid" approach didn't always beat the "Meaning Matcher" on its own; sometimes, mixing them just added noise.
• More Context isn't Always Better: When the researchers gave the AI too many pages to read at once (increasing the "Top-k" limit), the quality of the answers actually dropped. It's like giving a student a whole encyclopedia to answer a simple question; they get confused and start making things up (hallucinations).
• The "Refusal" Skill: When asked about things the syllabus didn't cover, the system was good at saying "I don't know." However, if the researchers gave it too much irrelevant information to read, it started guessing answers instead of refusing. It's like a student who, when overwhelmed with too much extra reading, starts making up an answer just to finish the test.

6. The Bottom Line

The paper concludes that for academic advising, a system that understands the meaning of a student's question is crucial. However, you have to be careful not to feed the AI too much extra information, or it might get confused and give you a confident but wrong answer.

The researchers suggest that future versions should be smarter about which pieces of information to pick, rather than just picking everything that sounds similar. They also note that while their system works well for this specific set of Japanese course syllabi, it's a proof-of-concept for how universities can build private, helpful AI tools without risking student privacy.

Technical Summary

Technical Summary: A Locally Deployed RAG-Based Academic Advising System for Course Selection

Problem Statement

Students often struggle to construct optimal course sequences due to information overload, recognition limitations, and the complexity of prerequisites found in extensive syllabi. Conversely, educational institutions face challenges in providing adequate, personalized academic advising due to limited resources and the heavy caseloads of human advisors. While AI-powered chatbots offer a potential solution, standard Generative AI models lack access to institution-specific, up-to-date syllabus data, and training dedicated models requires computational resources beyond the reach of many organizations. Furthermore, existing Retrieval-Augmented Generation (RAG) systems often employ fixed-size chunking strategies that fail to preserve the semantic integrity of semi-structured syllabus documents (e.g., tables, metadata, section titles), leading to degraded retrieval quality.

Methodology

The authors propose Syllabot, a locally deployed RAG system designed to assist students in course selection using syllabus information while preserving data privacy.

System Architecture

The system operates on a four-stage workflow:

1. Preprocessing: Syllabus documents are converted into header-based chunks enriched with metadata (source path and concise syllabus info).
2. Indexing: Chunks are indexed for retrieval.
3. Retrieval: Given a student query, the system retrieves relevant chunks using one of three strategies:
   • BM25: Lexical retrieval using Janome for Japanese tokenization.
   • Semantic Retrieval: Embedding-based retrieval using multilingual-e5-large and Pinecone.
   • Hybrid Retrieval: A combination of lexical and semantic results fused via Reciprocal Rank Fusion (RRF).
4. Generation: A locally executed Large Language Model (LLM), specifically RakutenAI-7B-instruct, generates answers based only on the retrieved context. The temperature is set to 0.0 to ensure consistency.

Evaluation Framework

To rigorously assess performance, the study introduces a retrieval-oriented query taxonomy based on retrieval complexity (defined by the number of required "gold chunks" and lexical overlap):

• Lexical Match Queries: Single gold chunk, high keyword overlap.
• Semantic Paraphrase Queries: Single gold chunk, low keyword overlap.
• Multi-evidence Synthesis Queries: Multiple gold chunks required.
• Out-of-scope Queries: No gold chunks (used to test refusal behavior).

Evaluation queries were automatically generated from predefined gold chunks using an LLM. The system was evaluated using:

• Retrieval Metrics: Gold Chunk Precision@K and Recall@K.
• Generation Metrics: Faithfulness (grounding in context) and Answer Relevance.
• Boundary Metric: Refusal Rate on out-of-scope queries.

Key Contributions

1. System Implementation: A privacy-preserving, locally deployed RAG system for academic advising that reduces reliance on external cloud APIs.
2. Taxonomy & Generation: A novel retrieval-oriented query taxonomy and an automated method for generating evaluation queries with gold chunk labels, allowing for granular analysis of retrieval complexity.
3. Evaluation Metrics: The proposal of Gold Chunk Precision@K and Recall@K to directly measure the retriever's ability to collect necessary evidence.
4. Empirical Analysis: A comprehensive evaluation demonstrating that retrieval strategy effectiveness varies significantly based on query type and complexity.

Experimental Results

The study utilized a corpus of 97 Japanese university courses and a 100-query evaluation set.

• Baseline: An LLM-only baseline (without RAG) refused 93.3% of answerable queries, confirming the necessity of external syllabus data for this domain.
• Retrieval Performance:
  • Lexical Match: Dense (semantic) retrieval performed competitively, achieving the highest Precision@K, challenging the assumption that BM25 is superior for keyword-heavy queries in this corpus.
  • Semantic Paraphrase: As expected, dense and hybrid retrieval significantly outperformed BM25.
  • Multi-evidence Synthesis: Hybrid retrieval did not show a clear advantage over dense retrieval; both outperformed BM25 in Recall, though BM25's Precision dropped rapidly as Top-k increased.
  • Hybrid Limitation: Fixed-weight hybrid fusion (RRF) did not consistently outperform dense retrieval, suggesting that non-gold chunks from one strategy may dilute the other.
• Generation Quality: As the number of retrieved chunks (Top-k) increased, Faithfulness decreased after Top-10, indicating that excessive context introduces noise that degrades the lightweight local LLM's performance.
• Boundary Capability: Refusal rates for out-of-scope queries declined as Top-k increased. The model became more prone to hallucinating plausible but unsupported answers when provided with larger, potentially superficially related contexts.

Significance and Claims

The paper claims that Syllabot successfully demonstrates the viability of locally deployed RAG systems for academic advising, offering a privacy-conscious alternative to cloud-based solutions. The study highlights that retrieval complexity dictates strategy effectiveness; a "one-size-fits-all" retrieval approach is suboptimal. Specifically, the authors argue that while dense retrieval performed well in their syllabus corpus, fixed hybrid strategies may not be necessary or optimal.

The authors modestly acknowledge limitations, noting that lexical retrieval (BM25) retains advantages for documents with distinctive terms and that current evaluation metrics like "Answer Relevance" may penalize correct refusals or reward plausible but incorrect answers. Consequently, the paper concludes that future work should focus on evidence-oriented retrieval (identifying chunks that contribute evidence rather than just semantic similarity) and evidence-based generation evaluation to further reduce hallucination risks in academic advising systems.