Patterns of Learner-AI Interaction and Academic Performance in an Object-Oriented Programming Course

Technical Summary: Patterns of Learner–AI Interaction and Academic Performance in an Object-Oriented Programming Course

Problem Statement
The rapid integration of Generative Artificial Intelligence (GenAI) tools into computer science education has created a complex landscape where empirical evidence regarding their educational impact remains mixed. While some studies suggest AI assistants improve performance and self-efficacy, others indicate they may hinder problem-solving development or lead to lower outcomes, particularly when students rely on AI for direct solutions rather than conceptual understanding. A specific gap exists in the literature regarding Object-Oriented Programming (OOP), a threshold concept in computer science education where students frequently struggle. Despite the prevalence of OOP courses, few studies have examined how students specifically integrate GenAI tools in this context or how distinct patterns of interaction correlate with learning outcomes. Current research often relies on binary comparisons (AI use vs. no use) rather than analyzing fine-grained behavioral patterns.

Methodology
This study employed a learner-centered, observational approach within a first-year Object-Oriented Programming course (Java) at the University of Tartu. The course utilized a flipped-classroom model where GenAI use was permitted for homework and group assignments but strictly prohibited during assessments.

• Sample: Data were collected from 210 undergraduate students (66.5% of the cohort) during the spring semester of the 2024/2025 academic year.
• Data Collection: A survey administered in week 11 captured self-reported frequency of GenAI use across three dimensions: code generation, debugging (bug identification), and explanation seeking. The survey also included Likert-scale items regarding perceived difficulty, self-assessed understanding, time investment, and attitudes toward AI (e.g., trust, ethical norms). Performance data (scores from two programming tests, a final exam, and total course points) were linked to survey responses.
• Analysis:
  • Descriptive & Correlational: Non-parametric tests (Friedman, Wilcoxon) analyzed usage frequencies. Spearman's rank-order correlation assessed relationships between usage frequency and academic performance.
  • Cluster Analysis: K-means clustering was applied to standardized usage indicators to identify distinct learner–AI interaction profiles. The optimal number of clusters was determined using silhouette scores and inertia.
  • Comparative Analysis: Kruskal–Wallis and Mann–Whitney U tests compared clusters across demographic variables, perceptions, attitudes, and assessment scores.

Key Contributions

1. Identification of Interaction Profiles: The study moves beyond aggregate usage metrics to define five distinct learner–AI interaction profiles: Zero usage, Low usage, Moderate usage, "Smart" high usage, and Very high usage.
2. Differentiation of Use Cases: It highlights that in an OOP context, students predominantly use GenAI for explanation seeking and debugging rather than code generation, challenging the assumption that students primarily rely on AI for solution generation.
3. Empirical Link to Performance: The research provides evidence on the relationship between specific interaction patterns and academic performance in AI-free assessments, addressing the lack of OOP-specific studies in the current literature.

Results

• Usage Patterns: 87.1% of respondents used GenAI tools at least once. Usage was significantly higher for explanation seeking and debugging compared to code generation.
• Cluster Characteristics:
  • The "Smart" high usage cluster (n=32) was characterized by low reliance on code generation but high usage for debugging and explanations.
  • The Very high usage cluster (n=37) showed consistently high engagement across all three dimensions.
  • Gender differences were observed only in the Zero usage cluster, which had a significantly higher proportion of male participants.
• Perceptions and Attitudes:
  • Students in the Very high usage cluster reported higher perceived difficulty of assignments and lower self-assessed understanding compared to the Zero usage cluster.
  • The "Smart" high usage cluster reported the highest trust in AI-generated code.
  • Norm-related attitudes (willingness to use AI when prohibited) varied by cluster, with high users showing greater willingness to bypass restrictions if peers were doing so.
• Academic Performance: Crucially, no significant differences were found in academic performance (Test 1, Test 2, Final Exam, or Total Points) across the five clusters. While usage patterns correlated with perceived difficulty and trust, they did not predict measurable learning gains or losses in AI-free assessments.
• Correlations: Weak negative correlations were found between the frequency of AI use and assessment scores, but these were not strong or consistent enough to suggest a definitive causal link.

Significance and Claims
The paper claims that self-directed GenAI use alone does not lead to measurable learning gains or hinder assessment results in an OOP course. The findings suggest that the relationship between AI use and learning outcomes is complex and context-dependent.

• Pedagogical Implications: The study underscores that simply permitting AI access is insufficient for improving learning. The "Smart" high usage pattern, which aligns with constructive engagement (seeking explanations and debugging), did not result in higher grades than other patterns. This suggests that without pedagogically guided, process-aware support, even strategic AI use may not translate into improved independent problem-solving skills.
• Future Direction: The authors argue for a shift from binary comparisons to pattern-level analyses. They emphasize the need for instructional frameworks that promote critical evaluation of AI outputs, reflection on problem-solving processes, and scaffolding to manage cognitive load, rather than relying on unguided, task-oriented interactions.
• Limitations: The authors modestly note that the reliance on self-reported data limits the ability to assess the depth of engagement and that the observational nature of the study precludes causal claims. The findings are specific to the course context and may not generalize to all educational settings without further research.