TAMUSA-Chat: A Domain-Adapted Large Language Model Conversational System for Research and Responsible Deployment

This paper introduces TAMUSA-Chat, a research-oriented framework that enables academic institutions to build responsible, domain-adapted conversational AI systems through supervised fine-tuning, retrieval-augmented generation, and systematic evaluation, while providing a publicly available codebase to support reproducible experimentation and ethical deployment.

The Gist

Imagine you have a brilliant, well-read librarian who has read almost every book in the world. This librarian is incredibly smart and can chat about anything from cooking recipes to quantum physics. However, there's a catch: this librarian has never visited your specific university.

If you ask them, "What time does the library close at Texas A&M University-San Antonio (TAMUSA)?" they might guess based on general knowledge, or worse, they might confidently make up an answer because they want to be helpful. This is the problem with standard AI chatbots: they are generalists, not specialists.

TAMUSA-Chat is the solution the authors built to fix this. Think of it as taking that world-famous librarian and giving them a specialized training camp specifically for your university.

Here is how the paper explains this process, broken down into simple, everyday concepts:

1. The Problem: The "Generalist" vs. The "Specialist"

Standard AI models are like a tourist who has visited 100 countries but doesn't know the local bus schedule in your town. If you ask them about a specific university major or a scholarship deadline, they might give you a "plausible-sounding" answer that is actually wrong. In a university setting, getting facts wrong can be a big deal (imagine a student missing a deadline because the AI guessed wrong!).

2. The Solution: The "University Boot Camp"

The authors created a system called TAMUSA-Chat. Instead of building a new librarian from scratch (which would take forever and cost a fortune), they took an existing smart AI and gave it a "boot camp" using only information from TAMUSA.

• The Data Collection (The Scavenger Hunt):
  Imagine a team of robots sent out to the university website. They don't just read the front page; they dive into PDFs, course catalogs, policy handbooks, and news articles. They collect everything public about the university, just like gathering every piece of a puzzle.
• The Cleaning (The Tidy-Up):
  The robots then clean this data. They throw out ads, broken links, and confusing formatting. They turn messy web pages into neat, organized cards that the AI can easily read.
• The Training (The Flashcards):
  This is the magic part. The system takes those clean documents and turns them into Flashcards.
  • Front of card: "What are the requirements for the Biology major?"
  • Back of card: "You need a 3.0 GPA and to pass Bio 101."
    The AI practices these flashcards thousands of times until it memorizes the answers perfectly. This is called Supervised Fine-Tuning.

3. The Safety Net: The "Fact-Checker" (RAG)

Even after training, there's a risk the AI might forget something or get confused. To fix this, the system uses a trick called Retrieval-Augmented Generation (RAG).

Think of this as giving the librarian a magic index card while they are talking to you.

• When you ask a question, the system first quickly searches its database of university documents (the index cards).
• It finds the exact page that answers your question.
• It hands that page to the AI and says, "Read this first, then answer."
• This ensures the AI isn't just guessing from memory; it's reading the official rules right in front of it. If the answer isn't in the documents, the AI is trained to say, "I don't know," instead of making things up.

4. The Result: A Responsible, Transparent Assistant

The paper emphasizes that this isn't just a cool toy; it's built with responsibility in mind.

• No Secrets: The system keeps a record of where every answer came from (like a citation in a research paper).
• Privacy Guard: It's programmed to ignore private student data and only use public information.
• Open Source: The "recipe" for this chatbot is published online for free. Other universities can look at the code, learn from it, and build their own versions without starting from zero.

Why Does This Matter?

The authors argue that universities shouldn't just "drop in" a generic chatbot. It's like hiring a generic tour guide for a specific city; they might get lost. TAMUSA-Chat is a custom-built guide who knows every corner of the campus, every rule in the handbook, and is honest about what they don't know.

In a nutshell:
The paper describes a recipe for turning a "smart but clueless" AI into a "smart and knowledgeable" university assistant by feeding it the school's specific books, teaching it how to study, and giving it a reference manual to check its work before it speaks.

Technical Summary

Here is a detailed technical summary of the paper "TAMUSA-Chat: A Domain-Adapted Large Language Model Conversational System for Research and Responsible Deployment."

1. Problem Statement

The paper addresses the critical gap between general-purpose Large Language Models (LLMs) and the specific needs of academic institutions. While off-the-shelf models (e.g., ChatGPT) are powerful, they suffer from several limitations when applied to university contexts:

• Hallucinations & Inaccuracy: They often generate plausible but factually incorrect information regarding specific academic programs, prerequisites, or policies (e.g., inventing non-existent majors).
• Lack of Contextual Awareness: They are unaware of institutional nuances, such as catalog years, transfer credit rules, and evolving departmental guidelines.
• Liability & Trust: Deploying generic chatbots in education poses risks of misinformation and liability if the AI provides authoritative-sounding but incorrect guidance.
• Reproducibility Gap: Existing solutions often prioritize application deployment over reproducible research workflows, lacking transparency in data provenance, evaluation metrics, and governance.

2. Methodology

The authors propose TAMUSA-Chat, a modular, research-oriented framework designed to adapt open-source LLMs to the Texas A&M University–San Antonio (TAMUSA) context. The system architecture is divided into five functional layers:

A. Data Acquisition & Governance

• Crawling: Uses Playwright to scrape public institutional websites (handling dynamic JavaScript rendering) and extract documents (PDFs, Word, Excel).
• Preprocessing: Implements rigorous cleaning, including HTML parsing, text normalization, duplicate removal, and PII (Personally Identifiable Information) detection.
• Governance: Strictly limits collection to public content, employs human review for sensitive data, and maintains version control for historical tracking.

B. Data Processing & Embedding

• Dataset Construction: Converts raw content into structured JSON and generates Instruction-Response (I-R) pairs via:
  • Template-based question generation from content.
  • Extraction of existing FAQs.
  • Synthetic generation using LLMs followed by human validation.
  • Reformulation/augmentation to increase linguistic diversity.
• Retrieval-Augmented Generation (RAG) Setup:
  • Documents are chunked (256–512 tokens) and embedded using Sentence Transformers (e.g., all-MiniLM-L6-v2).
  • Embeddings are indexed using FAISS with HNSW (Hierarchical Navigable Small World) for efficient approximate nearest neighbor search.

C. Model Fine-Tuning

• Base Models: Evaluates open-source models like LLaMA-2, Mistral, and Falcon. The primary experiments utilize SmolLM-135M-Instruct for its efficiency on edge devices and CPUs.
• Training Strategy: Employs Supervised Fine-Tuning (SFT) on the institutional I-R dataset.
• Optimization: Supports both full fine-tuning and Parameter-Efficient Fine-Tuning (PEFT) via LoRA (Low-Rank Adaptation) to reduce memory and compute requirements.
• Infrastructure: Designed to run on single high-end GPUs or distributed clusters (e.g., TAMUSA's CAMSA HPC) using PyTorch DistributedDataParallel.

D. Inference & RAG Pipeline

• Hybrid Approach: Combines parametric knowledge (from fine-tuning) with non-parametric knowledge (from retrieval).
• Workflow:
  1. User query is embedded and searched against the FAISS index (top-k=3 chunks).
  2. Retrieved context is concatenated with a system prompt emphasizing accuracy and honesty.
  3. The fine-tuned model generates a response using controlled decoding (Temperature=0.7, Top-p=0.9).
  4. Citations: The system automatically links answers to source documents to mitigate hallucinations.

E. Deployment & Utilities

• Modularity: Components (data, training, inference) are decoupled to allow independent experimentation.
• Deployment: Supports FastAPI/Flask REST services, Docker containerization, and cloud integration.
• Configuration: Centralized config.yaml for reproducible hyperparameter management.

3. Key Contributions

1. Open Research Framework: Unlike many application-focused chatbots, TAMUSA-Chat is designed as a reproducible research platform. All code, training scripts, and evaluation protocols are open-sourced (GitHub).
2. Hybrid Architecture: It effectively integrates Supervised Fine-Tuning (SFT) for domain adaptation with Retrieval-Augmented Generation (RAG) for factual grounding, balancing model fluency with verifiable accuracy.
3. Responsible AI Governance: The framework embeds ethical considerations directly into the pipeline, including PII screening, bias monitoring, transparency in data provenance, and clear disclaimers regarding system limitations.
4. Scalable & Efficient Design: Demonstrates that effective domain adaptation is possible with smaller models (135M parameters) and parameter-efficient methods, making it accessible to institutions with limited computational budgets.

4. Results & Corpus Statistics

The paper details the construction of the first institutional corpus for TAMUSA:

• Data Volume: 3,847 web pages and 412 documents (PDFs, etc.) crawled.
• Token Count: ~2.4 million tokens after cleaning.
• Instruction Pairs: 8,932 unique instruction-response pairs generated.
• Performance: The system demonstrates the ability to answer nuanced institutional questions (e.g., admission prerequisites) with citations, reducing the risk of hallucination compared to generic models.
• Evaluation: While specific benchmark scores are not fully detailed in the abstract, the methodology emphasizes a dual approach: automated metrics and human assessment protocols to evaluate alignment, factual grounding, and response quality.

5. Significance

• Academic Utility: Provides a blueprint for universities to deploy AI assistants that are safe, accurate, and aligned with institutional policies, moving beyond "black box" chatbots.
• Research Advancement: Contributes to the field of Domain Adaptation by showing that small, efficient models can be effectively adapted to specific domains without prohibitive costs.
• Ethical Deployment: Sets a standard for Responsible AI in education, prioritizing transparency, auditability, and user trust over raw performance metrics.
• Community Resource: The public availability of the codebase allows other institutions to replicate, audit, and improve upon the framework, fostering open collaboration in educational AI.

In summary, TAMUSA-Chat is not just a chatbot but a methodological framework that bridges the gap between general AI capabilities and the rigorous, safety-critical requirements of higher education institutions.