Is Solving Better Than Evaluating GenAI Solutions?

Problem Statement
The rapid advancement of Generative AI (GenAI) and Large Language Models (LLMs) has fundamentally altered the landscape of computing education, particularly in upper-division, theory-heavy courses like algorithms. While LLMs can generate syntactically correct code and solve many introductory problems, there is a growing pedagogical debate regarding how to adapt curricula. The core question addressed by this study is whether shifting the student role from solution generation to solution evaluation—specifically, having students grade and critique often-flawed GenAI-generated solutions—can improve conceptual understanding and learning outcomes compared to traditional problem solving. Existing literature largely focuses on the capabilities of LLMs or student perceptions, with a noted scarcity of controlled empirical evidence regarding learning outcomes in post-introductory courses where abstraction and proof-like reasoning are central.

Methodology
The authors conducted a randomized A/B crossover study (N=220) in a junior-level "Introduction to the Analysis of Algorithms" course at Purdue University during the Spring 2025 semester.

• Experimental Design: Students were organized into working groups and randomly assigned to one of two conditions: "Monte Carlo" (Group B) or "Las Vegas" (Group A). The study utilized a crossover design where the groups switched conditions midway through the semester.
  • Condition A (GenAI Evaluation): Students were provided with a standardized prompt to generate a solution using ChatGPT-4o. Their task was to evaluate the generated solution, identifying and explaining errors (distinguishing between minor bugs and major logical flaws) and justifying their grading. They were not required to provide a corrected solution.
  • Condition B (Traditional Solving): Students solved the same challenging algorithmic problems directly, requiring them to design an algorithm, prove correctness, and analyze runtime.
• Intervention Scope: The intervention applied to the fourth, most difficult exercise of six homework assignments. The first three exercises in each assignment were identical traditional problems for all students.
• Data Collection: The study measured:
  1. Summative Performance: Midterm and final examination scores, overall course grades, and performance on specific exam questions structurally aligned with the homework interventions.
  2. Homework Performance: Scores on the specific GenAI-evaluation vs. direct-solution exercises.
  3. Self-Reported Data: Pre-midterm surveys on comfort/confidence and post-final surveys on perceived helpfulness and changes in study habits.
• Analysis: The authors employed rigorous statistical analysis, including assumption screening (normality, homogeneity of variance), parametric (t-tests) and non-parametric (Mann-Whitney U, Wilcoxon signed-rank) tests, and effect size calculations (Cohen's d, rank-biserial r).

Key Results
The study yielded several critical findings regarding the efficacy of GenAI evaluation as a pedagogical tool:

1. No Difference in Summative Learning: There were no statistically significant differences between the two groups on midterm scores, final exam scores, overall course grades, or the normalized change in exam performance.
2. No Difference in Transfer: Performance on exam questions specifically designed to test the conceptual insights emphasized in the GenAI-based assignments showed no significant difference between students who had previously evaluated AI solutions versus those who had solved the problems directly.
3. Homework Score Disparity: Students received significantly higher scores on the homework items during the periods they were evaluating GenAI solutions compared to when they were solving problems directly. However, the authors note this localized advantage did not translate into downstream summative gains.
4. Study Habits and Perceptions:
   • Most students (77%) reported that the modified assignments did not change their study habits.
   • Among the minority who did report adapting their study strategies, those students rated the GenAI-evaluation assignments as significantly more helpful than those who did not change their habits.
   • There was no significant difference between groups in perceived helpfulness or self-reported confidence prior to the midterm.

Key Contributions

• Empirical Evidence in Upper-Division Courses: The study provides rare controlled empirical evidence regarding GenAI's impact on learning in a theory-heavy, upper-division algorithms course, moving beyond the focus on introductory programming or LLM capability benchmarks.
• Redistribution of Effort: The findings suggest that GenAI evaluation tasks redistribute student effort from open-ended solution construction toward verification, diagnosis, and judgment.
• Feasibility without Loss: The study demonstrates that incorporating GenAI-evaluation activities into algorithms coursework does not result in broad performance losses, suggesting it is a viable alternative to traditional problem solving in terms of maintaining baseline competency.
• Limitations of Simple Evaluation: The research indicates that simply asking students to "grade" an AI solution is insufficient to automatically produce stronger conceptual transfer or learning gains.

Significance and Claims
The authors conclude that while GenAI-evaluation activities can be incorporated into algorithms coursework without harming overall performance, they do not automatically yield superior learning outcomes compared to traditional problem solving. The paper argues that the pedagogical value of such activities likely depends on deliberate scaffolding that pushes students beyond simple error diagnosis.

The authors posit that for meaningful learning gains to occur, future iterations of these assignments may need to require students to:

• Identify the first invalid step in an argument.
• Construct counterexamples where proposed algorithms fail.
• Propose corrected algorithms or invariants.
• Explain why the corrected solution works.

The study emphasizes that the mere presence of AI-generated content is not the driver of learning; rather, the specific design of the evaluation task determines its effectiveness. The authors call for future research to investigate which forms of scaffolding make these activities most effective and to develop validated assessment instruments for competencies like verification, critique, and calibrated reliance on AI.