SortingHat: Redefining Operating Systems Education with a Tailored Digital Teaching Assistant

Technical Summary: SortingHat

Problem Statement
Operating Systems (OS) education faces significant challenges due to the inherent complexity of the subject matter, which spans process management, memory allocation, and kernel development across diverse hardware architectures. Traditional teaching methods struggle to accommodate the varying educational backgrounds, learning speeds, and practical needs of students. Current resources, such as static lecture materials and limited office hours, are insufficient for providing the immediate, context-sensitive guidance required for hands-on experimentation. Furthermore, existing AI-based educational tools often lack emotional engagement, rely on single-modal text interactions, and fail to maintain up-to-date domain-specific knowledge without impractical retraining efforts.

Methodology
SortingHat is a personalized digital teaching assistant designed to address these gaps through three integrated technical components:

1. Customized Digital Human Interface:

   • Architecture: The system utilizes a 3D digital human powered by Large Language Models (LLMs) to provide multimodal (visual, auditory, emotional) interactions.
   • Knowledge Base: To ensure domain accuracy, the system employs a Retrieval-Augmented Generation (RAG) framework based on GraphRAG. This leverages a corpus of over 60,000 OS-related documents, including source code and technical documentation on emerging architectures, alongside 10,000 generated instructions.
   • Personalization: An agent extracts student profiles (learning progress, knowledge gaps) from a database to dynamically adjust the tone, depth, and emotional expression of the assistant's responses. A Text-to-Speech (TTS) server converts text into context-appropriate speech.

2. Adaptive Exercise Generation:

   • Workflow: A multi-agent framework generates tailored exercises based on individual student histories and performance.
   • Process: The workflow involves selecting materials relevant to the student's level, followed by a multi-agent discussion where generation agents create exercises and evaluation agents critically review them for accuracy, clarity, and relevance. This iterative collaboration ensures the exercises are coherent and educationally effective.

3. MARL-Based Evaluation Pipeline:

   • Challenge: Traditional grading is biased and inefficient, while standard LLMs suffer from context-length constraints, attention decay, and hallucinations.
   • Solution: SortingHat implements a Multi-Agent Reinforcement Learning (MARL) system enhanced with RAG.
   • Process: Long student submissions are segmented into manageable chunks. Information Extraction (IE) agents process these sections, storing context in memory. These agents then engage in collaborative discussions with evaluator agents following an MARL framework. The evaluation is broken down into 20 specific goals. Finally, a feedback generator synthesizes these insights with the student's academic level and history to produce actionable, personalized feedback.
   • Dynamic Updates: The system utilizes the Arxiv API to weekly crawl, evaluate, and recommend new papers on "Operating Systems," ensuring the knowledge base remains current.

Key Contributions

• Lifelike 3D Digital Assistant: The introduction of a multimodal, emotionally engaging digital human that provides expert-level, context-aware guidance, moving beyond the limitations of text-only interfaces.
• LLM-Driven Personalized Exercises: A system that generates customized learning materials targeting specific weak areas and advanced concepts based on real-time student data.
• MARL Evaluation System: A novel approach to grading that ensures consistency and fairness by using collaborative multi-agent discussions to mitigate hallucinations and instability, while providing detailed, personalized feedback.

Results and Demonstration
The paper demonstrates the system's capabilities through several visualizations:

• Interface: The digital human provides guidance tailored to a student's academic level (Figure 5).
• Exercise Generation: The system successfully generates exercises that reflect a student's specific academic standing (Figure 6).
• Evaluation Consistency: To demonstrate stability, the framework evaluated three papers of varying quality 100 times each. The results (Figure 9) showed consistent evaluation scores, with variations accurately reflecting the inherent quality differences between submissions rather than system instability.
• Feedback: The system provides comprehensive scoring and reasoning tailored to the student's submission history (Figure 8).

Significance
SortingHat represents a step forward in personalized education by offering a scalable solution for OS curricula. By integrating a customizable digital human, LLM-based reasoning agents, and a student database, the system addresses critical limitations in traditional education, including outdated exercises, a lack of individual support, and inconsistent evaluation. The authors posit that this framework lays a foundation for future advancements in adaptive learning technologies, transforming OS education into an engaging, immersive, and scalable experience.