Integrating AI into Requirements Quality Learning in Software Engineering Education: A TPACK-Guided Empirical Study
This empirical study demonstrates that a TPACK-guided integration of a multi-agent AI tool into a master-level requirements engineering course effectively scaffolds students' selective use of AI for analyzing and evaluating requirement quality, thereby enhancing their understanding of specific quality criteria while fostering responsible engagement with generative AI.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are building a massive, intricate Lego castle. Before you snap the first brick, you have to write down exactly what the castle should look like, how it should function, and what it's for. In the world of software, this writing phase is called Requirements Engineering. It's the blueprint stage where engineers figure out what a program needs to do. If the blueprint is vague or wrong, the whole castle collapses later.
Now, imagine a super-smart robot assistant (Generative AI) that can instantly write these blueprints for you. It's fast and creative, but here's the catch: if you just let the robot do all the work, you might never learn how to build the castle yourself, or you might not notice if the robot's blueprint has a hidden flaw. This is the big question facing teachers today: How do we use these powerful robots to help students learn, without turning them into lazy copy-pasters who don't understand the rules?
To answer this, researchers use a teaching map called TPACK. Think of TPACK as a recipe for a perfect lesson that mixes three ingredients: the Content (the actual rules of building), the Pedagogy (the teaching tricks to make it stick), and the Technology (the robot tools). The goal is to mix them so the robot helps you think, rather than just doing the thinking for you.
The Experiment: Teaching Students to Be the Boss of the Robot
In a master's level software engineering class at Tampere University, researchers tried out a new way to use AI. They didn't just say, "Here's a robot, write some requirements." Instead, they designed a specific "dance" for the students and the AI to follow, based on the TPACK recipe.
The assignment was about User Stories—short sentences that describe what a user wants a software feature to do. To make sure these stories were high quality, the class used a checklist called INVEST (which stands for Independent, Negotiable, Valuable, Estimable, Small, and Testable).
Here is how the "dance" worked:
- Human First: Students had to manually rewrite some old, messy requirements into good User Stories using the INVEST checklist. No robots allowed yet.
- Robot Second: Then, they used a multi-agent AI tool (a robot with different "personalities" representing different people, like a customer or a developer) to generate its own version of the stories.
- The Showdown: Students had to compare their human-written stories with the robot's stories. They had to decide which ones were better, edit the robot's work, and explain why.
- The Peer Review: Finally, students swapped their work with classmates to grade each other using the same checklist.
What They Found: The Robot is a Helper, Not a Replacement
The study looked at 72 students who participated. Here is what happened when they tried to partner with the AI:
1. Students Didn't Just Hit "Accept All"
If the robot was just a magic button, students would have accepted everything it wrote. But they didn't. The students were picky. On average, the robot generated about 16.5 user stories per student, but the students only approved about 9.5 of them. That's an approval rate of 56%.
- The Metaphor: It's like ordering a pizza from a robot chef. The robot makes 16 pizzas, but you only keep 9 because the others have too much cheese or the wrong toppings. You are the taste-tester, not the robot.
- The Behavior: About 24% of students explicitly edited the robot's output, and most students went through at least one round of refining the stories. They treated the AI as a draft generator, not a final solution.
2. The Robot Was Great at Some Things, Tricky at Others
When the researchers checked how well the students understood the quality rules (the INVEST checklist) before and after using the AI, they saw a split result.
- The Wins: The students got much better at spotting stories that were Testable (could be checked with a test) and Valuable (actually useful). The robot helped make these parts clearer.
- The Mixed Bag: For a quality called Negotiable (meaning the requirements can be changed through discussion), the results were weird. Students felt like the robot helped them understand this better, but when their answers were checked against the teacher's "correct" answers, their scores actually went down slightly.
- The Lesson: The robot is great at making things look structured and clear (like fixing grammar), but it struggles with the fuzzy, human parts of software design where you have to guess what people might want.
3. Trust, But Verify
When asked how they felt about the tool, students were cautiously optimistic.
- Usefulness: Most agreed the robot helped them generate ideas and add details. One student said the robot gave them "structured language" and "clearer acceptance criteria."
- Trust: They didn't blindly trust it. About 14 students said they actively checked the robot's work against the rules, and 10 explicitly rejected or challenged the robot's ideas.
- The Catch: Some students found the robot's interface a bit clunky, and sometimes the robot gave answers that were too vague or didn't fit the specific project (like suggesting a drone delivery feature when the project was about something else).
The Big Takeaway
The study suggests that the secret to using AI in education isn't just having a cool tool; it's about how you use it.
If you let students use the AI first, they might just copy the answers. But if you force them to do the hard thinking first, and then use the AI as a second opinion to compare and refine, the AI becomes a powerful learning partner. The students in this study didn't become lazy; they became editors. They learned to spot the robot's mistakes and use its strengths to improve their own work.
However, the researchers warn that this isn't a magic fix. The robot is still a bit confused by the "fuzzy" parts of software requirements. So, teachers need to be careful. They should use AI to help with the clear, structural parts of learning, but keep the human teacher in the loop to guide the tricky, interpretive parts.
In short: The robot is a fantastic co-pilot, but the student must keep their hands firmly on the steering wheel.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.