Stan: An LLM-based thermodynamics course assistant

This paper presents Stan, a locally deployed, privacy-preserving AI system for an undergraduate thermodynamics course that utilizes open-weight models to simultaneously provide grounded, reference-backed tutoring for students and generate actionable teaching insights for instructors from a shared corpus of lecture transcripts and textbook data.

The Gist

Imagine you are in a massive, high-speed lecture hall for a difficult subject like Thermodynamics. The professor talks fast, writes complex equations on the board, and uses words like "fugacity" and "entropy" that sound like alien languages.

Now, imagine two different people in that room: a student trying to keep up, and the professor trying to remember what actually happened in class.

This paper introduces Stan, a smart digital assistant built specifically to help both of them. Think of Stan not as a magic robot that knows everything, but as a super-organized librarian and a reflective mirror that lives entirely on the computer in your pocket, not in the cloud.

Here is how Stan works, broken down into simple concepts:

1. The "Local Library" (No Cloud, No Privacy Worries)

Most AI tools today are like renting a car from a big rental company. You have to drive to their office (the cloud), pay by the mile (API fees), and they can see your destination (your data).

Stan is different. It's like buying your own car and keeping it in your garage.

• How it works: The authors built Stan to run entirely on local hardware (like a standard laptop or a powerful desktop).
• Why it matters: It costs nothing per question, your class notes and recordings never leave your computer (total privacy), and it works even if the internet goes down. It uses open-source "brains" (AI models) that anyone can inspect and tweak.

2. For the Student: The "Smart Flashlight"

When a student asks, "What is fugacity?", a normal AI might guess an answer based on what it read on the internet. If it guesses wrong, that's dangerous in engineering.

Stan acts like a flashlight in a dark library.

• The Process: Instead of guessing, Stan first looks at the student's question, finds the exact keywords (like "fugacity"), and then shines its light directly onto the textbook's index and the lecture recording.
• The Result: It doesn't just give an answer; it gives the answer plus the page number and the specific moment in the lecture where the professor explained it. It says, "Here is the definition, and here is exactly where you can read more about it."
• The Safety Net: It uses a "dual-path" system. One part is a fast, rigid rule-finder (like a spell-checker), and the other is a smart AI. If the smart AI gets confused, the rule-finder ensures the student still gets something useful.

3. For the Professor: The "Time-Traveling Mirror"

This is the most unique part of the paper. Usually, AI is just for students. But Stan also helps the teacher.

Imagine the professor could rewind the semester and watch a highlight reel of their own teaching.

• The "Confusion Map": Stan listens to the recordings and spots moments where students sounded confused or asked the same question three times. It tells the professor: "Hey, on Tuesday, 15 minutes in, the class seemed lost on 'Entropy.' You might want to review that."
• The "Story Collector": Professors often use funny stories or real-world analogies to explain hard concepts. Stan catalogs these. It creates a searchable list: "Show me every time I used a 'metal cube' analogy to explain heat." This helps the teacher remember their best teaching moments.
• The "Gap Finder": It can tell the professor, "You talked about this concept verbally, but it never made it into the homework or slides." This helps them fix the course materials.

4. The "Brainy Librarian" vs. The "Hallucinating Artist"

The paper admits that AI can sometimes "hallucinate" (make things up confidently). To stop this, Stan uses a few clever tricks:

• The "No-Imagination" Rule: The AI is strictly told, "You can only use the information I just gave you from the textbook. Do not make up new facts."
• The "Two-Pass" System: When analyzing a long lecture, the AI first just grabs all the questions (like a net catching fish), and then a second pass sorts them to find the important ones. This prevents the AI from getting overwhelmed and making up fake answers.
• The "Prompting" Trick: Because the AI is a general model, it might not know engineering terms. The professor gives it a "cheat sheet" of technical words (like "Peng-Robinson equation") before it starts listening. This is like giving the librarian a specific dictionary so they don't confuse "fugacity" with "GASB."

5. The Future: A "Glass Box"

The authors plan to make Stan even smarter. They want to add "glass box" tools—simple computer programs that solve thermodynamics problems step-by-step. Instead of just telling the student the answer, Stan will be able to say, "Here is the code that solves this, and here is how it works," allowing students to peek under the hood of the math.

The Big Picture

Stan is not trying to replace the teacher or the textbook.

• It doesn't grade papers.
• It doesn't write the lecture.

Instead, it acts as a bridge. It connects the messy, fast-paced reality of a live lecture to the structured, reliable world of the textbook. It gives students a way to find answers without getting lost, and it gives teachers a way to look back at their teaching, learn from it, and make it better for the next class.

It's a tool that says: "Let's use AI not to do the work for us, but to help us understand the work we are already doing."

Technical Summary

Here is a detailed technical summary of the paper "Stan: An LLM-based thermodynamics course assistant."

1. Problem Statement

The prevailing discourse on AI in education focuses almost exclusively on student-facing tools (chatbots, tutors, problem generators). However, the potential for AI infrastructure to support instructors in managing, analyzing, and reflecting on their own teaching remains largely unexplored. Furthermore, existing solutions often rely on proprietary cloud APIs, raising concerns regarding:

• Data Privacy & Security: Risks of prompt injection and data leakage.
• Cost Predictability: Variable costs based on usage.
• Hallucinations: In precision-focused fields like thermodynamics, confident but incorrect outputs are dangerous.
• Reproducibility: Dependence on third-party services that may change or disappear.

There is a need for a system that serves both students and instructors using a shared data foundation, operates locally to ensure privacy and cost control, and minimizes hallucinations through grounded retrieval.

2. Methodology & System Architecture

The authors developed Stan, a suite of tools built on a Retrieval-Augmented Generation (RAG) pipeline. The system is designed to run entirely on locally controlled hardware using open-weight models, eliminating dependence on cloud APIs.

A. Hardware & Deployment

• Dual-Tier Architecture:
  • GPU Workstation (Batch Processing): A Linux workstation with an NVIDIA RTX 4090 (24 GB VRAM) handles heavy lifting: audio transcription (Whisper large-v3) and initial index building.
  • Consumer Laptop (Interactive Query): Standard laptops run the RAG retrieval, vector search, and interactive query interface using Ollama for local LLM inference (7–8B parameter models like Llama 3.1 8B).
• Privacy & Cost: No internet connection is required for inference; costs are limited to the one-time hardware investment.

B. Data Sources & Processing Pipeline

1. Lecture Transcription:

   • Tool: faster-whisper (CTranslate2 optimized) using the large-v3 model.
   • Domain Adaptation: To handle technical terminology (e.g., "fugacity," "Peng-Robinson"), the authors implemented vocabulary prompting, feeding key thermodynamic terms into the decoder prompt.
   • Hallucination Mitigation: A three-layer strategy was used to prevent repetition loops:
     1. Decoder-level suppression (repetition penalty and 3-gram blocking).
     2. Context isolation (condition on previous text=False).
     3. Post-processing to detect and remove identical consecutive segments.
   • Result: Hallucination rates dropped from 6.4% (baseline) to 0.02% across 39 lectures.

2. Textbook Representation:

   • Instead of vector embeddings of raw text, the system uses a structured JSON index extracted from the textbook's back-of-book index and Table of Contents.
   • Rationale: This leverages the author's expert curation, avoiding the complexity and potential errors of chunking and embedding raw text.

C. Student-Facing Tool: Textbook Query

• Dual-Path Term Extraction:
  1. Regex Extractor: Fast, deterministic extraction of predefined technical phrases.
  2. LLM Extractor: Uses Llama 3.1 8B to expand queries (e.g., "fugacity" $\to$ "fugacity coefficient," "activity coefficient").
• Max-Score Merge: Results from both extractors are merged, taking the maximum score for each index entry to ensure no valid matches are lost.
• Grounded Synthesis: The top 5 matches (with page/chapter context) are passed to the LLM with strict system prompts forbidding the invention of facts or page numbers. The output is a concise response with specific citations.

D. Instructor-Facing Tool: Lecture Analysis

The same transcript corpus is processed through four analysis pipelines to generate structured JSON:

1. Lecture Summaries: Titles, topics, key concepts, equations, and classification (new material vs. review).
2. Question Identification: Detects student/instructor questions, classifies type (conceptual, procedural), and rates pedagogical significance.
3. Confusion Detection: Identifies moments of student confusion or instructor re-explanation with severity ratings.
4. Anecdotes & Analogies: Catalogs teaching devices (jokes, real-world examples, historical notes).

Failure Mode Mitigations:

• Context Truncation: Explicitly set context window to 16,384 tokens (vs. default 2,048) to fit full lectures.
• Bimodal Output: For question extraction, a two-pass architecture was implemented. Pass 1 extracts raw candidates; Pass 2 filters/ranks them. This prevented the model from either generating fixed, formulaic outputs or exhaustive, unfiltered lists.
• Schema Drift: Defensive parsing handles variations in JSON field names.

3. Key Contributions

• Dual-Perspective Design: A unified infrastructure that simultaneously acts as a student resource pointer and an instructor analytics engine.
• Local-First Implementation: A fully functional, reproducible system running on consumer hardware with open-weight models (Whisper, Llama 3.1), ensuring data sovereignty and predictable costs.
• Domain-Specific Optimization:
  • Vocabulary Prompting: Significantly improved ASR accuracy for specialized engineering terms.
  • Structured Indexing: Utilizing textbook indices rather than raw embeddings for more reliable retrieval.
  • Two-Pass Analysis: Solving the bimodal output problem in long-context structured extraction.
• Open Source Framework: The system is implemented as a modular Python library, encouraging modification and extension by the academic community.

4. Results

• Transcription Quality: The domain-prompted Whisper system achieved a 49× real-time throughput on 35.7 hours of audio. It significantly outperformed institutional ASR (Kaltura) in recognizing technical terms (e.g., correctly identifying "fugacity" vs. "GASB") and preserving semantic meaning in complex sentences.
• Hallucination Reduction: The three-layer mitigation strategy reduced hallucinated repetition loops from 6.4% to 0.02%.
• Instructor Analytics: The system successfully processed 35 lectures in ~15 minutes, generating structured summaries, identifying 11 questions per lecture (on average), and cataloging anecdotes.
• Student Interaction: The RAG pipeline successfully answered natural language queries (e.g., "Explain fugacity") with responses grounded in specific textbook pages, avoiding hallucinations by restricting the LLM's context to retrieved evidence only.

5. Significance

• Pedagogical Shift: Stan moves beyond "automating" instruction (grading, content generation) to extending instructional capacity. It provides instructors with a searchable, semester-scale record of their teaching, enabling reflection on concept evolution, recurring student misconceptions, and the effectiveness of analogies.
• Scalability & Equity: By running on local hardware, the system avoids subscription fees and cloud dependency, making advanced AI tools accessible to institutions with limited budgets while maintaining data privacy.
• Reliability in STEM: The paper demonstrates that with careful engineering (structured retrieval, domain prompting, local inference), LLMs can be safely deployed in high-stakes, precision-focused disciplines like thermodynamics without the risk of dangerous hallucinations.
• Future Directions: The authors envision expanding the system to include interactive computational tools (e.g., Python-based solvers for phase equilibria) and scaling the hierarchical indexing framework across entire chemical engineering curricula.

In conclusion, Stan represents a practical, robust, and ethically sound approach to integrating Generative AI into higher education, prioritizing instructor agency, data privacy, and pedagogical utility over the "black box" nature of commercial cloud services.