ACE-TA: An Agentic Teaching Assistant for Grounded Q&A, Quiz Generation, and Code Tutoring

The paper introduces ACE-TA, an agentic framework that autonomously routes programming course queries to specialized modules for grounded conceptual Q&A, adaptive quiz generation, and interactive, sandboxed code tutoring using pre-trained Large Language Models.

The Gist

Imagine you are trying to learn how to cook a complex new dish, like a soufflé. You have a recipe book (your textbook), but you're stuck. You don't know why the eggs need to be beaten a certain way, you keep burning the bottom of the pan, and you have no one to ask because your chef instructor is asleep.

This is exactly the problem college students face in programming classes. They have the "recipe" (the textbook), but they struggle to actually "cook" the code, and they often can't get help when they need it most (like at 2 AM).

Enter ACE-TA. Think of ACE-TA not just as a chatbot, but as a super-smart, tireless teaching assistant that never sleeps, never gets tired, and has the entire library of course materials memorized.

Here is how ACE-TA works, broken down into three simple "superpowers":

1. The "Fact-Checker" (Conceptual Q&A)

The Problem: You ask, "What is a variable?" and a normal AI might give you a vague, made-up definition that sounds cool but isn't quite right for your specific class.
The ACE-TA Solution: ACE-TA acts like a librarian with a magnifying glass. Before it answers your question, it doesn't just guess; it runs to the specific pages of your textbook, finds the exact paragraph that explains it, and then rewrites that explanation in simple, easy-to-understand language.

• Analogy: It's like having a tutor who refuses to answer a question unless they can point to the exact page in the book where the answer lives. This ensures you aren't learning "fake news" from the AI.

2. The "Quiz Master" (Adaptive Quizzes)

The Problem: Taking a test is scary. If the questions are too hard, you give up. If they are too easy, you don't learn anything.
The ACE-TA Solution: ACE-TA is like a personalized video game coach. It creates a quiz for you on the spot.

• If you get a question right, it says, "Great job! Here's a slightly harder one to stretch your brain."
• If you get it wrong, it says, "No worries, let's try a simpler version to make sure you get the basics first."
• Analogy: Imagine a gym trainer who watches your workout. If you lift a heavy weight easily, they immediately hand you a heavier one. If you struggle, they swap it for a lighter one so you can build strength without getting hurt. ACE-TA does this with knowledge.

3. The "Step-by-Step Code Coach" (Code Tutoring)

The Problem: This is the hardest part. You have to write code. If you make a mistake, the computer crashes, and you have no idea why. A normal AI might just spit out the whole answer, which doesn't help you learn.
The ACE-TA Solution: ACE-TA acts like a patient co-pilot. It breaks a big, scary programming problem into tiny, manageable steps.

• Step 1: It asks you to write just one small line of code.
• Step 2: It runs that line in a "safe sandbox" (a virtual test kitchen) to see if it works.
• Step 3: If it fails, it doesn't just say "Error." It says, "Hey, you forgot a colon at the end of that line. Let's fix just that one thing."
• Analogy: Imagine building a Lego castle. Instead of handing you the finished castle, ACE-TA hands you one brick at a time. It checks if the brick fits. If it doesn't, it tells you exactly which brick is wrong before you try to add the next one. This way, you build the whole castle yourself, but with a safety net.

Why is this a big deal?

In the past, AI tutors were like fortune tellers: they guessed answers based on patterns. ACE-TA is like a detective: it gathers evidence (from your textbook), plans a strategy (breaking down the problem), and uses tools (running code in a safe box) to solve the mystery.

Because it is "agentic," it can think ahead. It knows that if you are stuck on a coding error, you probably need a concept explained first. It can switch between being a librarian, a quiz master, and a coding coach instantly, all without you having to leave the chat window.

In short: ACE-TA is the ultimate study buddy that ensures you aren't just memorizing answers, but actually understanding the "why" and "how" of programming, with a safety net that catches you every time you stumble.

Technical Summary

1. Problem Statement

Higher education faces a critical challenge in scaling support for large enrollment STEM courses, particularly in programming. Traditional Teaching Assistants (TAs) are often overwhelmed by workloads and limited availability, leaving students without timely help during homework or late-night study sessions. While Large Language Models (LLMs) offer potential, existing educational AI systems suffer from three main limitations:

• Narrow Scope: Most focus solely on natural language Q&A or administrative tasks, lacking support for complex computational reasoning.
• Lack of Grounding: Generic LLMs often hallucinate or provide answers not aligned with specific course materials.
• Insufficient Scaffolding: Existing tools often provide complete solutions rather than guiding students through step-by-step reasoning, debugging, and formative assessment.

There is a distinct need for a unified, Agentic Artificial Intelligence (AAI) framework that can autonomously route queries, ground responses in specific textbooks, generate adaptive assessments, and guide code construction through iterative feedback.

2. Methodology: The ACE-TA Architecture

ACE-TA is a local, multi-module framework that integrates Retrieval-Augmented Generation (RAG), adaptive assessment, and stepwise code coaching. It operates under a shared orchestration layer using two local LLMs loaded via llama-cpp-python:

1. Router/Validator: Phi-3-Mini-128K-Instruct (compact, deterministic routing).
2. Generator: GPT-OSS-20B (larger model for complex generation).

The system routes user queries to one of three specialized modules:

A. Orchestration Layer (Query Router)

A lightweight router classifies incoming queries into four categories: Conceptual Q&A, Quiz Generator, Code Tutor, or Unknown. It uses a constrained schema and deterministic temperature settings (0) to ensure consistent routing before dispatching the query to the specific module while reusing preloaded model objects to minimize latency.

B. Conceptual Q&A Module

This module provides textbook-aligned explanations using a Hybrid Retrieval pipeline:

• Retrieval: Combines FAISS-based dense retrieval and BM25 lexical retrieval over a "Hybrid Vector DB."
• Reranking: A cross-encoder (MS-MARCO-MiniLM-L6-v2) reranks the top 20 candidates from both channels, selecting the top 5 for context.
• Generation: The generator is prompted with a structured "Harmony style" prompt, conditioning its response strictly on the retrieved evidence. It is instructed to explicitly acknowledge when context is insufficient.

C. Quiz Generator Module

This module creates adaptive, higher-order Multiple Choice Questions (MCQs):

• Topic Decomposition: The validator breaks broad topics into five subtopics for learner selection to reduce topic drift.
• Diverse Retrieval: Uses Maximum Marginal Relevance (MMR) on a "Quiz Vector DB" to select diverse passages, ensuring coverage of different sub-concepts.
• Synthesis: The generator constructs MCQs with Bloom's taxonomy tags (Apply, Analyze, Evaluate, Create), distractor rationales, and correct answers.
• Adaptivity: Difficulty is adjusted dynamically based on student performance (promoting higher-order questions for correct answers, reinforcing basics for errors).

D. Code Tutor Module

This is an interactive, stepwise coding assistant:

• Planning: The generator creates a numbered sequence of logic-level steps (no code yet) to decompose the problem.
• Iterative Execution: For each step, the system prompts the LLM for the smallest Python snippet implementing only that step.
• Sandboxed Verification:
  1. AST Check: Code is parsed via Abstract Syntax Tree to catch syntax/indentation errors before execution.
  2. Sandbox Execution: Code runs in a sandbox.
  3. Feedback Loop: If execution fails, the system provides specific feedback to modify the step. If successful, an LLM comparator checks semantic equivalence against a reference snippet, accepting alternate logic.
• Interaction: Students type only the new lines needed for the current step, preventing premature inclusion of future logic.

3. Key Contributions

1. Unified Agentic Framework: Designed ACE-TA to seamlessly integrate grounded conceptual Q&A, adaptive quiz generation, and real-time stepwise coding guidance within a single pipeline.
2. Novel Technical Integrations:
   • Hybrid retrieval with late-chunking and semantic chunking for Q&A fidelity.
   • MMR-based diverse retrieval for multi-concept quiz generation.
   • Incremental code tutoring with AST validation and sandboxed execution for safe, iterative learning.
3. Comprehensive Evaluation: Demonstrated effectiveness using a mix of quantitative metrics (RAGAS, BERTScore, CodeBLEU, DWPM) and qualitative Subject Matter Expert (SME) evaluations.

4. Results and Evaluation

The system was evaluated on a dataset derived from Learning Python (5th Edition), comprising 100 Q&A pairs, 108 MCQs, and 150 coding problems.

• Conceptual Q&A Performance:

  • Retrieval: Achieved near-perfect context precision and high recall.
  • Answer Fidelity: Achieved an Answer Relevancy score of 0.94 and a BERTScore F1 of 0.93, indicating minimal semantic drift.
  • Expert Rating: SMEs rated ACE-TA explanations significantly higher in depth (3.8/5) compared to Gemini 2.5 Pro (1.3/5), noting ACE-TA's answers were more focused and less cluttered.

• Quiz Generator Performance:

  • Coverage: Achieved a mean Breadth score of 1.73 (on a scale where >2 is very broad) and a Balance score of 0.97, indicating quizzes cover topics broadly without skewing toward specific subtopics.
  • Adequacy: SMEs rated explanation adequacy at 3.57/5, noting generally high quality with minor issues regarding token limits in single-pass generation.

• Code Tutor Robustness:

  • Model Independence: Tested across 7 LLMs (ranging from 7B to 27B parameters). Larger models (e.g., Gemma-3-27B) achieved near-ceiling Difficulty Weighted Performance Measure (DWPM) of 1.0.
  • Correlation: A strong positive Spearman correlation ($\rho = 0.875$) was found between "Effective Capacity" (parameters + precision) and robustness.
  • Step Clarity: SMEs rated step clarity at 3.84/5 overall, with only a modest decline in clarity as problem difficulty increased (4.00 for Easy vs. 3.73 for Hard).
  • Solution Quality: High CodeBERT similarity (>0.99) indicated semantic correctness, while moderate CodeBLEU scores reflected the system's acceptance of diverse, semantically equivalent coding styles.

5. Significance

ACE-TA represents a significant advancement in educational AI by moving beyond simple chatbots to agentic, task-specific tutoring.

• Pedagogical Impact: It addresses the "last mile" problem in programming education by guiding students through the process of coding rather than just providing answers, fostering deeper conceptual understanding.
• Scalability: By automating the roles of Q&A, assessment, and coding guidance, it offers a scalable solution for large STEM classes, reducing TA workload while maintaining high-quality, personalized support.
• Reliability: The use of retrieval grounding and sandboxed execution ensures that the AI operates within the bounds of course materials and produces executable, verified code, mitigating the risks of hallucination common in generative AI.

Future work aims to generalize the framework to other courses beyond Python and conduct classroom deployment studies to measure direct learning outcomes.