🤖 AI

Learning to Code with Context: A Study-Based Approach

This paper presents a user study within a university game development course that investigates how students utilize generative AI tools throughout the software lifecycle and evaluates the effectiveness of a repository-aware, locally deployed LLM assistant using Retrieval-Augmented Generation to provide contextually grounded support.

Original authors: Uwe M. Borghoff, Mark Minas, Jannis Schopp

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Uwe M. Borghoff, Mark Minas, Jannis Schopp

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching Coders in the Age of AI

Imagine a university class where students are like a team of architects and builders tasked with constructing a complex video game from scratch. This is a "programming project" course. Usually, these students learn to build by doing it themselves, but now, they have a new, powerful tool in their pocket: Generative AI (like ChatGPT or GitHub Copilot).

The researchers at the University of the Bundeswehr Munich wanted to answer two big questions:

How are students actually using these AI tools right now?
Can we build a better AI tool that actually understands the specific game the students are building, rather than just guessing?

Part 1: The "Wild West" Study (What Students Did Naturally)

First, the researchers watched what happened when students were left to their own devices with free, public AI tools. They asked 38 students to fill out a survey after their project was done.

The Findings:

The "Text" vs. "Code" Divide: Students loved using AI for writing things down. It was like having a super-fast secretary who could draft requirements, write user manuals, or explain what a piece of code might do.
The "Hallucination" Problem: When it came to actually writing code for their specific game, the AI often got lost. Imagine asking a travel agent to book a flight to a city that doesn't exist yet. The AI would confidently invent details that didn't match the students' actual project.
The "Context" Gap: The biggest issue was that the public AI tools didn't know about the specific "blueprints" (source code) the students were working with. They were like a mechanic trying to fix a Ferrari while looking at a manual for a Ford. The AI would suggest parts that didn't fit or didn't exist in the students' code.

The Verdict: Students used AI a lot (89% of them!), mostly to save time on writing and explaining. But when they tried to use it to build the actual game, they often had to fix the AI's mistakes because the AI didn't "know" their specific project.

Part 2: The "Local Librarian" Experiment (The New Solution)

Realizing that the public AI tools were like "general knowledge" encyclopedias that lacked specific details, the researchers built their own custom AI assistant.

How it Works (The Analogy):
Think of the public AI as a genius but amnesiac writer. They know how to write code, but they don't know what your specific house looks like.

The researchers built a Local Librarian.

The Library: They put the students' entire game code and documentation into a private, local library (a "repository").
The Librarian: This AI assistant lives on a computer right on campus (for privacy). Before it answers a question, it doesn't just guess; it goes to the library, finds the exact blueprints and code files relevant to the question, and reads them.
The Result: Now, when a student asks, "How do I add a sound effect to the ship?", the Librarian looks at the actual ship code in the library and gives an answer that fits perfectly, rather than making things up.

The Experiment:
They tested this "Local Librarian" with two specific tasks:

Task A: Changing how ships look in the game (from boxes to 3D models).
Task B: Adding background music that can be turned on/off.

They tested different "personalities" of the AI (different models) and different settings (how creative vs. how strict the AI should be).

The Results:

Context is King: The Local Librarian was much better at giving answers that actually fit the project. It didn't invent fake code as often because it had the real blueprints to look at.
The "Goldilocks" Setting: They found that if the AI was too "creative" (high randomness), it started hallucinating again. If it was too "strict," it was boring. A middle-ground setting worked best.
Size Doesn't Always Matter: Interestingly, the biggest, most powerful AI models weren't always the best. A medium-sized model actually did a great job, while the smallest model struggled to understand the complex instructions.
One Stubborn Flaw: Even with the Librarian, the AI sometimes forgot to update a specific "resource list" (like forgetting to register a new sound file in the game's menu). This happened across almost all models, suggesting it's a tricky habit for AI to break, not just a lack of knowledge.

The Bottom Line

The paper concludes that while AI is a fantastic tool for helping students write documentation and understand general concepts, it struggles when it tries to build specific software without "reading the room" (the project context).

By building an AI that has direct access to the project's code and documents (the "Local Librarian"), the researchers showed that AI can become a much more reliable partner. It stops guessing and starts helping with accurate, context-aware solutions.

What's Next?
The researchers plan to turn this "Local Librarian" into a true Tutor. Instead of just giving the answer, they want the AI to guide students through the problem, helping them learn how to solve it, rather than just handing them a ready-made solution. This ensures students still learn the skills they need, even with AI by their side.

Technical Summary: Learning to Code with Context: A Study-Based Approach

Problem Statement

The rapid emergence of generative AI tools is transforming software development, necessitating an adaptation in software engineering education. While students frequently utilize general-purpose AI tools (e.g., ChatGPT, GitHub Copilot), they often encounter significant limitations when applying these tools to structured, team-based software projects. Specifically, general-purpose models lack awareness of specific project contexts, including proprietary source code, project documentation, and domain-specific frameworks (e.g., jMonkeyEngine). This leads to "hallucinations," inaccurate code generation, and integration failures, particularly when students attempt to use AI for code comprehension and implementation rather than just text generation. Furthermore, there is a concern regarding "skill degradation," where over-reliance on AI might hinder the development of independent problem-solving skills. The core challenge is to develop an AI support system that provides meaningful, context-aware assistance without simply providing ready-made solutions, while operating within data protection constraints that require local deployment.

Methodology

The authors employed a two-stage empirical approach within a university programming project course where student teams develop 3D multiplayer computer games using Java and a waterfall process model.

Stage 1: User Study (Quantitative & Qualitative)

Context: Conducted during the October–December 2024 term with 38 participants (approx. 78% of the cohort).
Data Collection: A structured questionnaire based on the Technology Acceptance Model was administered during the final project presentations. It assessed AI usage across four phases: Preparation, Analysis, Design, and Coding & Testing.
Focus: The study investigated which tasks students performed with AI, the perceived helpfulness, time savings, and specific challenges encountered (e.g., lack of context, inaccurate results).

Stage 2: Case Study (Repository-Aware LLM Assistant)

System Design: To address the context limitations identified in Stage 1, the authors developed a locally deployed, repository-aware Large Language Model (LLM) assistant.
Architecture:
- Retrieval-Augmented Generation (RAG): The system employs a dual-path grounding mechanism.
  1. Document-level RAG: Retrieves relevant project documentation and auxiliary materials via semantic embedding.
  2. Code-level Lookup: Uses read-only GitLab API tools (file_path_finder, get_content_from_file, get_methods) to retrieve specific source code files and method signatures based on user queries.
- Deployment: The system runs on a local NVIDIA DGX-H100 server using Docker containers. It utilizes vLLM for model serving (OpenAI-compatible API) and Open WebUI for the chat interface.
- Data Privacy: Local deployment ensures data control and reproducibility, addressing institutional data protection requirements.
Evaluation:
- Tasks: Two representative tasks from the "Preparation" phase of the Battleship project were used: (1) Adding 3D models for ships based on length, and (2) Implementing background music controls.
- Variables: The study conducted a "sampling sweep" (varying temp, top_p, min_p) and a "model sweep" (testing six models ranging from 7B to 236B parameters, including DeepSeek-Coder, Mistral, and Qwen variants).
- Analysis: Generated outputs were analyzed against a "Defect Catalog" containing 12 categories of errors (e.g., Hallucination, Tool misuse, Insufficient robustness, Missing resource entry).

Key Contributions

Empirical Evidence of Context Limitations: The user study provides quantitative data demonstrating that while students heavily adopt AI for documentation and requirements (text-heavy tasks), they struggle significantly with code comprehension and implementation due to the lack of project context in general-purpose tools.
Repository-Aware Assistant Architecture: The paper presents a concrete implementation of a locally deployed, RAG-based assistant that integrates document retrieval with code-level repository lookups. This system is designed to ground responses in actual project artifacts.
Defect Catalog for AI in Education: The authors define a specific catalog of 12 recurrent defect types observed when LLMs interact with software repositories, providing a framework for qualitative analysis of AI behavior in educational settings.
Parameter and Model Sensitivity Analysis: The study offers empirical findings on how sampling parameters (specifically temperature and min_p) and model size affect the frequency of specific defects (e.g., hallucinations) in a repository-aware setting.

Results

User Study Findings

Adoption: 89.4% of participants used AI tools. Adoption was highest in the Coding & Testing phase (33/38 users) and lowest in the Design phase (10/38 users).
Helpfulness: AI was rated as "helpful" or "very helpful" for Preparation and Coding & Testing phases, particularly for documentation and code generation. However, it was rated as least helpful (often "neutral" or "not helpful") during the Design phase, largely due to the inability of AI to generate accurate diagrams (class, sequence, state).
Challenges: The primary issues reported were "inaccurate results," "missing context" (AI not knowing the specific project code or frameworks like jMonkeyEngine), and "hallucinations" (generating non-existent methods).
Perception: Despite challenges, 24 of 34 users felt AI saved them significant time, and the majority expressed intent to use AI in future projects.

Case Study Findings

Effectiveness of Context: The repository-aware assistant consistently produced outputs that were more aligned with the project structure compared to the general tools observed in the user study. Grounding responses in actual repository artifacts reduced integration issues.
Defect Patterns:
- Hallucination: Remained the most frequent defect (occurring in 14/15 sampling configurations), particularly at higher temperatures ( $T=1.0$ ). However, the system's ability to retrieve files often allowed for self-correction in subsequent turns.
- Missing Resource Entry: A recurring defect across almost all models and configurations where the model failed to register new resources (e.g., localization keys) despite successfully retrieving the relevant documentation. This suggests a "retrieval-to-action gap."
- Sampling Parameters: The configuration $T \approx 0.5$ , $top\_p = 0$ (or low) proved most stable, yielding the fewest defects. High temperatures combined with filtering often increased error rates.
Model Performance:
- Model size did not linearly correlate with success. The 24B Devstral-Small-2505 produced highly usable outputs, while the 7B Mistral-7B failed to complete tasks without extensive follow-up.
- Very large models (e.g., 235B) provided detailed reasoning but did not consistently outperform mid-sized models in integration fidelity.
- A recurring issue across models was the failure to utilize retrieved context effectively for specific actions (e.g., writing to resource files), indicating that tool integration is a bottleneck independent of model size.

Significance and Claims

The paper claims that integrating context-aware retrieval (specifically repository grounding) is a critical step toward making AI useful in software engineering education. The authors argue that while general-purpose AI tools are valuable for text-based tasks, they are insufficient for code implementation without access to project-specific artifacts.

The study demonstrates that a locally deployed, repository-aware assistant can mitigate the "integration weakness" of general AI, producing more coherent and reproducible outputs. However, the authors remain modest about the current state of the technology, acknowledging that:

Hallucinations and tool misuse persist even with RAG.
Diagram generation and visual design remain areas where AI support is weak.
Skill degradation is a risk; therefore, the proposed future direction is not an AI that solves problems for students, but a "tutor" that provides hints and guides independent problem-solving.

The significance of this work lies in its shift from evaluating generic AI tools to engineering and evaluating context-specific AI assistants that respect data privacy and educational goals, providing a blueprint for how such systems can be integrated into software engineering curricula.