Exploring the Value of Diverse LLM Explanations in Introductory Programming

Technical Summary: Exploring the Value of Diverse LLM Explanations in Introductory Programming

Problem Statement
While Large Language Models (LLMs) demonstrate the capacity to generate code explanations that often surpass peer-generated quality, a critical gap remains in understanding how presenting multiple explanations with intentionally distinct semantic emphases affects novice programmers. Existing research has established that LLMs can generate diverse formats, but it is unclear whether varying the conceptual focus of these explanations (e.g., function, concept, goal) enhances student understanding compared to generic, all-purpose explanations. Drawing on Variation Theory and insights from computational creativity, the authors hypothesize that exposing learners to systematic variations in explanation emphasis may help them discern critical features of programming concepts, thereby deepening comprehension beyond what a single high-quality or generic explanation might achieve.

Methodology
The authors conducted a large-scale, between-subjects study involving 971 first-year computing students enrolled in a C programming course at a large public research university.

• Experimental Design: Participants were randomly assigned to receive either Generic explanations (three identical prompts asking for a plain-text summary) or Diverse explanations (three distinct prompts targeting specific dimensions: function, concept, and goal) for two programming exercises.
• Materials: Three code snippets were prepared (sumArray, randomizeString, countChar), though each student viewed only two to limit cognitive load. Explanations were generated using GPT-4o.
• Procedure: After viewing the explanations for each snippet, students answered:
  1. A Multiple-Choice Question (MCQ) testing core concepts.
  2. An Open-Ended (OE) question requiring conceptual comparison or explanation.
  3. Likert-scale items regarding perceived helpfulness and information load.
  4. Open-ended reflections on the utility and usage of the explanations.
• Analysis:
  • Performance: MCQ accuracy was analyzed quantitatively. OE responses were qualitatively coded for correctness using a rubric, with inter-rater reliability established at Fleiss' Kappa ($\kappa = 0.75$).
  • Perception: Likert data was analyzed using Kruskal-Wallis H tests.
  • Thematic Analysis: Open-ended reflections were coded using Braun and Clarke's guidelines, grouped into compound themes (e.g., "Engaged Understanding," "Active Monitoring," "Overload/Frustration").

Key Results

• Performance (RQ1):
  • MCQs: Accuracy was consistently high (>87%) across all conditions. Chi-square tests revealed no statistically significant differences between diverse and generic explanation groups.
  • Open-Ended Questions: Students receiving diverse explanations showed a consistent performance improvement of approximately 7.7% across all three code snippets compared to the generic condition. However, when modeled via logistic regression with Bonferroni adjustment, the condition effect did not reach statistical significance ($z = 2.15, p = 0.095$).
• Perceived Cognitive Load and Helpfulness (RQ2):
  • There were no statistically significant differences in perceived helpfulness or perceived information load between the diverse and generic groups.
  • Students generally rated the amount of information as appropriate (mean ratings ~3.3–3.4 on a 5-point scale), indicating that the diverse condition did not induce information overload.
• Student Reflections:
  • Thematic analysis of open-ended responses indicated that students in the "Diverse-Diverse" (DD) condition were more likely to report "Engaged Understanding" and less likely to report "Overload/Frustration" compared to other groups, though these patterns did not reach statistical significance in chi-square tests.
  • "Strategic Use" (using explanations to aid code reading) was the most common theme across all conditions.

Key Contributions

1. Empirical Investigation of Semantic Diversity: The study provides one of the first large-scale empirical tests of whether varying the semantic emphasis (function vs. concept vs. goal) of LLM-generated explanations improves learning outcomes compared to generic explanations.
2. Scalable Data on Student Engagement: With 971 participants, the study offers robust data on how introductory students interact with multiple AI-generated explanations, specifically addressing the trade-off between explanation quantity/diversity and cognitive load.
3. Nuanced Performance Metrics: The paper highlights a discrepancy between MCQ performance (where no difference was found) and OE performance (where a consistent ~7.7% gain was observed for diverse explanations), suggesting that diverse explanations may support deeper conceptual articulation even if they do not immediately shift multiple-choice accuracy.

Significance and Claims
The authors modestly claim that while diverse explanations did not produce statistically significant gains in comprehension in this specific context, they were associated with consistently higher performance (approx. 7.7% gain in OE accuracy) and more engaged student responses across multiple measures.

The paper argues that the lack of statistical significance may be due to a "ceiling effect" (high baseline performance) or that the specific dimensions of variation used were not perfectly aligned with the critical features students needed to discern. However, the findings suggest that providing multiple, concise perspectives is a promising strategy that does not increase cognitive load. The authors conclude that rather than repeating generic explanations, educators and automated tutoring systems could leverage LLMs to generate varied perspectives, allowing students to select the angle most helpful to their learning, thereby supporting differentiated instruction at scale. Future work is suggested to explore long-term retention and the impact of varying explanation types based on student expertise.