LLM-Generated Design Problems for Assessing Higher-Order Thinking in Project-Based Learning

Imagine you're in a cooking class where the whole semester is spent mastering the art of making the perfect pizza. You learn to knead the dough, choose the sauce, and bake it until it's golden. At the end of the term, the teacher usually grades you on the pizza you actually made. But here's the problem: just because you can make a great pizza doesn't mean you truly understand why it works, or that you could use those same skills to bake a cake or fix a broken oven. You might have just memorized the steps for the pizza, or maybe you got help from a robot chef (like an AI) that did the heavy lifting while you just watched.

This is exactly the puzzle computer science teachers face with Project-Based Learning (PjBL). Students build cool software projects, but traditional tests often only check if they can copy-paste code or write a report about the specific thing they built. They miss the "higher-order thinking" (HOT)—the ability to take what they learned and apply it to a totally new, tricky situation.

The New Tool: "Design Problems"

To fix this, the authors of this paper introduced something called Design Problems (DPs). Think of these as a "kitchen challenge" that happens right after the pizza class. Instead of asking, "Show me your pizza," the teacher says:

"Okay, imagine a hospital needs a system to track how elderly patients move their fingers to detect health issues. Using the same logic you used for your pizza app, design a high-level plan for this new system. You have 30 minutes."

This forces students to stop just reciting facts and start transferring their knowledge. They have to analyze, evaluate, and create something new, proving they really get the concepts, not just the specific code they wrote for their final project.

The Robot Helper: Can AI Write the Questions?

Creating these tricky, new-scenario questions is hard work for teachers. It takes forever to invent a fresh situation that fits every single student project. So, the researchers asked: Can a Large Language Model (LLM)—a super-smart AI chatbot—write these questions for us?

They tested this by asking two different AI models (a standard one and a "reasoning" one that thinks step-by-step) to generate 80 of these Design Problems based on four different student projects (like a security tool, a distributed game, and a mobile app).

What they found:

• The AI is pretty good: The "reasoning" AI was especially impressive. It created scenarios that were realistic and perfectly matched the learning goals. In fact, 50% of the AI-generated problems got a perfect score from human experts.
• The AI isn't perfect: Sometimes the AI got lazy. It would create a scenario that was too similar to the original project (like asking for a pizza when you already made a pizza), or it would make the question too vague.
• The Verdict: The paper suggests that AI can be a huge help, acting like a tireless assistant that drafts the questions, but a human teacher still needs to review them to make sure they aren't too easy or confusing.

The Classroom Test: Did Students Get It?

The researchers then dropped these AI-written Design Problems into three real college classes. They gave students 30 minutes to solve one of these new scenarios individually.

Here's the twist:
When they compared the students' grades on these new Design Problems to their grades on the original semester-long projects, there was almost no connection.

• Some students were "Masters": They aced the project and the new challenge.
• Some were "Implementers": They built a great project (maybe with help or by following a recipe) but failed the new challenge because they couldn't apply the logic elsewhere.
• Some were "Conceptualizers": They struggled with the project but crushed the new challenge because they understood the deep concepts.

This proves that traditional project grades don't tell the whole story. A student can build a great app but still not know how to think like an engineer in a new situation. The Design Problems caught this difference.

How Students Actually Thought

The researchers also looked at how students typed their answers (using keystroke data). They found that students didn't just type furiously.

• They paused for about 2 minutes at the start, likely thinking and planning.
• They deleted and rewrote parts of their answers (a "revision ratio" of about 12–16 characters deleted for every 100 typed).
• They took frequent breaks (pauses longer than 10 seconds) while writing.

This behavior suggests that students were actually doing the hard mental work of analyzing and synthesizing ideas, rather than just copying and pasting.

What the Paper Rules Out

The paper is careful to say what this method is not:

• It is not a magic bullet that replaces all other testing.
• It is not something that works perfectly if students can use AI during the test (that's why the teachers made them do it in class, without phones).
• It is not a way to grade students based on how long they take; the data showed that spending more time didn't necessarily lead to better grades.
• It is not a perfect system yet; the paper explicitly notes that grading these open-ended answers is still subjective and hard to scale without help.

The Bottom Line

The study suggests that using AI to generate these "Design Problems" is a promising way to check if students truly understand computer science concepts, rather than just memorizing how to build one specific thing. It helps teachers see the difference between a student who can follow a recipe and a student who can cook up a whole new meal. While the AI isn't perfect and needs a human supervisor, it could be the key to making sure students are ready for the real world, where problems never look exactly like the ones they practiced in class.