🤖 AI

From Domain Understanding to Design Readiness: a playbook for GenAI-supported learning in Software Engineering

This paper presents a two-week master's course experiment demonstrating that a curated GenAI tutor effectively supports rapid upskilling in cryptocurrency-finance and Domain-Driven Design with high accuracy and pedagogical value, leading to significant student self-efficacy gains while offering seventeen concrete practices to optimize future GenAI-supported learning workflows.

Original authors: Rafal Wlodarski

Published 2026-04-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rafal Wlodarski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a complex, high-speed trading robot for a class project. But there's a catch: you and your teammates are experts in coding, but you know almost nothing about cryptocurrency or Domain-Driven Design (a fancy way of saying "how to organize software to match real-world business rules").

Normally, you'd have to spend weeks reading dry textbooks and asking a busy professor for help. But in this study, the professor gave you a super-smart, 24/7 digital tutor (a customized version of ChatGPT) that had been fed all the class notes, project rules, and examples.

Here is the story of what happened when 29 students used this AI tutor, explained simply.

🍕 The "Pizza Delivery" Analogy

Think of the students as a pizza delivery team. They know how to drive the car (coding), but they don't know the map of the city (the crypto domain) or the rules of the neighborhood (DDD).

The professor couldn't drive every car for them. So, they gave every driver a GPS (the AI tutor) that was pre-loaded with the exact map and traffic rules for this specific city.

🧪 What Did the Study Find?

The researchers watched how the students used this GPS and graded the answers the AI gave. Here is the breakdown:

1. The GPS Was Shockingly Accurate (The "Truth" Score)

The Result: The AI got the facts right 99% of the time.
The Metaphor: Imagine asking your GPS, "How do I get to the pizza place?" and it giving you the exact route without ever sending you down a dead end. In fact, out of 60 questions checked, the AI only made two tiny, harmless mistakes. It didn't "hallucinate" (make things up), which is a common fear with AI.
Why? Because the professor didn't let the AI wander off. They locked it inside a "knowledge box" containing only the course materials.

2. The Answers Were Helpful, But a Bit Robotic (The "Teacher" Score)

The Result: The AI explained things well and didn't use confusing jargon. However, it was very dry.
The Metaphor: The AI was like a very efficient librarian. If you asked for a book, it handed it to you perfectly. But if you looked sad, it didn't say, "You got this! Keep going!" It just said, "Here is the book."
The Problem: Students felt the AI lacked "warmth." It didn't cheer them on or encourage them when they were stuck. It was a great information machine, but a poor coach.

3. The Students Felt Much More Confident

The Result: Before using the AI, students felt lost. After using it, they felt like they could actually build the project.
The Metaphor: It's like going from trying to assemble IKEA furniture with a manual written in a foreign language, to having a friend stand next to you pointing at the right screw. The students didn't just learn the facts; they felt capable.

🛠️ The "Playbook": How to Make AI Work for Class

The author realized that just giving students an AI isn't enough. You have to set the rules. Here are the "Golden Rules" they discovered, translated into everyday advice:

Don't let the AI talk too much: AI loves to write long intros and summaries. The professor told it, "Stop the fluff, just give me the answer." (Like telling a chatty friend, "Get to the point!")
Give it a "Persona": The AI was too robotic. The fix? Tell the AI to act like a supportive coach who says, "Great job asking that!" instead of just a database.
Lock the Knowledge: Don't let the AI use its general internet knowledge (which can be wrong). Force it to only use the specific class notes and project files.
Make it Count: The professor made the AI work mandatory and gave a tiny bit of credit for it. This stopped students from ignoring it.

🏁 The Bottom Line

This study proves that AI can be a fantastic sidekick for learning, especially when you need to learn a new, difficult topic quickly.

The Good: It's accurate, fast, and helps students feel confident.
The Bad: It can be boring and lacks emotional support.
The Fix: If you give the AI a strict set of rules (a "playbook") and tell it to be a bit more encouraging, it becomes a powerful tool that helps students learn without needing a human teacher hovering over their shoulder every second.

In short: AI is the ultimate study buddy, but you have to teach it how to be a good friend first.

1. Problem Statement

Software Engineering (SE) education requires students to rapidly acquire knowledge in two distinct areas: application domain understanding (e.g., finance, crypto) and modeling methods (e.g., Domain-Driven Design or DDD). Traditional lecture-based methods struggle to provide the necessary scale of interaction, frequent feedback, and individualized support required for rapid upskilling in these areas. While Large Language Models (LLMs) offer scalable tutoring, their instructional quality, reliability (hallucinations), and pedagogical fit in complex SE contexts remain under-evaluated. There is a lack of empirical data on how to effectively integrate GenAI tutors to support domain learning and modeling tasks without compromising educational quality.

2. Methodology

The study was conducted in a 12-unit Master's course on Functional Programming at Carnegie Mellon University Silicon Valley during Fall 2024.

Participants: 29 Master's students with diverse backgrounds (mostly CS/Engineering, some with finance experience).
Context: A two-week milestone focused on building a cryptocurrency arbitrage system ("ArbitrageGainer") using Domain-Driven Design (DDD).
Intervention:
- A customized ChatGPT (GPT-3.5 Turbo) tutor was deployed.
- Grounding: The model was grounded in a curated knowledge base containing course slides, project specifications, and exemplar solutions.
- Workflow: Students completed three mandatory pre-class "upskilling" sessions. Each session involved executing two prescribed prompts followed by a knowledge check and a reflection report.
- Data Collection: All interactions were logged. The study analyzed a stratified random sample of 60 prompt–answer pairs (34.5% of the total 174 pairs) across the three sessions.
Evaluation Framework:
- Rubric: A five-dimensional rubric was used to score responses: Accuracy, Relevance, Pedagogical Value, Cognitive Load, and Supportiveness.
- Self-Efficacy: Pre- and post-intervention surveys measured student confidence in using GenAI for domain learning and DDD application.
- Raters: The instructor and two trained Teaching Assistants (TAs) scored the data, achieving high inter-rater reliability ( $\kappa \approx 0.76-0.78$ ).

3. Key Contributions

The paper provides a practical "playbook" for educators, offering:

Empirical Evaluation: A rigorous assessment of GenAI tutoring quality in a real-world SE course setting, moving beyond theoretical speculation.
Actionable Teaching Practices: A set of 17 concrete practices categorized into Prompt/Configuration, Course Design, and Workflow/Assessment to optimize GenAI integration.
Replicable Workflow: A documented methodology for collecting, sampling, and analyzing GenAI interactions in SE education.
Theoretical Application: The application of Cognitive Load Theory (managing verbosity/jargon) and ICAP framework (structuring follow-ups) to GenAI prompt engineering.

4. Key Results

A. Answer Quality (RQ1)

Accuracy: Extremely high (98.9%). No factual hallucinations were found; only 2/60 responses had minor inaccuracies that did not mislead students.
Relevance: Very high (92.2%). Responses were consistently on-topic, though some suffered from "boilerplate" (unnecessary intros/summaries).
Self-Efficacy: Students reported statistically significant gains in confidence:
- Using GenAI for domain knowledge ( $p < .001$ , large effect).
- Applying DDD with GenAI ( $p < .001$ , large effect).

B. Pedagogical Quality (RQ2)

Pedagogical Value: High (89.4%). Responses provided structured reasoning and relevant examples.
Cognitive Load: Generally appropriate (82.78%). However, 5% of responses were flagged as overwhelming due to verbosity or unexplained jargon.
Supportiveness: Low (37.78%). The default model persona was neutral and informational, lacking praise, validation, or motivational language.

C. Effective Practices (RQ3)

The study identified specific levers that enhanced effectiveness:

Prompt Engineering: Setting expected granularity, constraining verbosity, capping the number of examples, and blocking unfamiliar jargon.
Configuration: Curating a specific knowledge base (slides/specs) and embedding "guardrail" examples.
Course Design: Making GenAI use a mandatory Learning Objective (LO), assigning course credit (1–5%), and requiring brief interaction logs.

5. Significance and Implications

Feasibility of GenAI in SE: The study demonstrates that with a curated knowledge base and strict prompt guardrails, GenAI can deliver highly accurate and relevant support for complex domain and modeling tasks, effectively bridging the gap between lecture and practice.
The "Playbook" Value: The paper shifts the conversation from "Can we use GenAI?" to "How do we configure it?" It provides a low-cost, scalable framework for instructors to integrate AI tutors without requiring massive TA overhead.
Critical Limitations & Future Work:
- Tone: The primary weakness is the lack of emotional supportiveness; future iterations must explicitly prompt for a "coaching persona."
- Cognitive Load: Instructors must actively manage verbosity and example counts to prevent overwhelming students.
- Generalizability: While results are promising, they are specific to a Master's cohort and a specific domain (Finance/DDD). Future work should explore replication in other SE domains and larger class sizes.

Conclusion: GenAI-supported learning is a viable, scalable complement to traditional instruction in Software Engineering, provided it is anchored in course-specific materials and governed by a structured "playbook" of prompt engineering and workflow design.