Harnessing Hype to Teach Empirical Thinking: An Experience With AI Coding Assistants

1. Problem Statement

Software engineering (SE) education often struggles to engage students with empirical methods and hypothesis-driven inquiry. These concepts are frequently perceived as abstract, dry, or disconnected from practical application when taught in isolation. Conversely, the software industry is currently driven by "hype cycles" surrounding emerging technologies, such as AI coding assistants (e.g., GitHub Copilot). While students are highly motivated to learn these trending tools, they often lack the critical, evidence-based thinking required to evaluate the validity of claims made about them.

The core challenge addressed by this paper is: How can educators leverage student excitement around hype-driven technologies to teach rigorous empirical research methods without sacrificing methodological rigor?

2. Methodology

The authors conducted a one-semester seminar at Saarland University designed to test the hypothesis that grounding empirical learning in a popular, hype-driven topic increases engagement and learning outcomes.

Course Structure and Design

• Participants: 18 students were selected from 168 applicants (a record number for the department), resulting in 13 completers. The class was supported by 9 instructors (doctoral and postdoctoral researchers), creating a high staff-to-student ratio (approx. 1:1 to 1:2) for intensive supervision.
• Duration: 10 weeks of in-class sessions (90 minutes each) followed by 9.5 weeks of independent writing.
• Pedagogical Framework: The course utilized Constructive Alignment, ensuring learning objectives, activities, and assessments were mutually supportive.
• Key Activities:
  1. Hands-on Sessions: Three sessions focused on Code Generation, Code Explanation/Refactoring, and Debugging. Students performed tasks with and without AI assistants, collecting comparative data to understand the tool's limitations.
  2. Instructional Modules: Brief lectures on empirical study design, literature search, and scientific presentation.
  3. Student-Led Empirical Studies: Each student designed a small-scale empirical study (25–30 minutes) on a specific aspect of AI coding assistants.
  4. Iterative Piloting: Students conducted two pilot studies with peers, refining their study designs based on feedback and preliminary results.
  5. Deliverables: A final seminar paper (15 pages) and a 12-minute presentation.

Evaluation Methodology

To assess the seminar's effectiveness, the authors employed a mixed-methods approach:

• Quantitative Survey: A voluntary post-seminar survey administered to 10 of the 13 participants (8 completers responded). It utilized Likert-scale questions (1–5) to measure changes in perception of AI tools, confidence in empirical thinking, and the link between hype and motivation.
• Qualitative Analysis: Open-ended survey responses were analyzed using thematic coding (following Cruzes and Dyba's guidelines) to identify recurring themes regarding student experiences and learning gains.
• Observational Data: Instructors tracked application numbers, dropout rates, and the evolution of student study designs from draft to final version.

3. Key Contributions

The paper makes three primary contributions to computing education research:

1. Pedagogical Strategy: It demonstrates that hype-driven topics can serve as an effective "hook" to lower the barrier to entry for abstract concepts like empirical research. The excitement surrounding AI coding assistants successfully attracted students who might otherwise avoid methodological courses.
2. Curriculum Artifact: The authors provide a replicable framework for a "dual-purpose" seminar that simultaneously teaches technical skills (using AI tools) and research skills (designing studies, hypothesis testing).
3. Empirical Evidence: The study provides data showing that students can effectively transition from "users" of hype technologies to "investigators" who critically evaluate those technologies through self-designed empirical studies.

4. Results

Engagement and Interest

• Application Volume: The seminar received 168 applications for 18 spots, significantly exceeding the departmental median of 25 applications per seminar. This indicates that framing empirical research around a trending topic drastically increases student interest.
• Motivation: Survey data showed that 80% of respondents agreed that working on a hyped topic motivated them more than a conventional topic.

Learning Outcomes

• Empirical Thinking: Students reported significant gains in their ability to distinguish between anecdotal and empirical evidence (Mean = 4.0/5) and in their confidence to assess claims about emerging technologies (Mean = 4.2/5).
• Critical Reflection: Through the iterative pilot process, students learned to refine study designs, reduce scope, and address validity threats. For example, many students realized that small sample sizes (e.g., two participants) preclude strong quantitative claims but are valuable for identifying design flaws.
• Perception of AI Tools: Students shifted from viewing AI assistants as "accurate and reliable" to understanding their limitations and risks. They developed a more nuanced view, recognizing that effectiveness depends heavily on prompt engineering and specific use cases.

Challenges Identified

• Expectation Mismatch: Some students dropped out because they expected a purely technical "how-to" course on AI, rather than a research-focused seminar. This highlights the risk of hype attracting students with misaligned expectations.
• Methodological Uniformity: Due to the course structure (short pilots, peer pairing), most students defaulted to controlled experiments rather than exploring other methods (e.g., qualitative interviews), limiting methodological diversity.

5. Significance and Implications

• Reframing "Hype": The paper argues that hype should not be dismissed as a distraction but leveraged as a pedagogical catalyst. By anchoring the "sugar" (hype) to the "broccoli" (empirical methods), educators can make rigorous research skills accessible and engaging.
• Ownership of Inquiry: The study confirms that when students design their own investigations (even small-scale ones), they develop a deeper sense of ownership and critical engagement than when replicating established studies.
• Scalability and Adaptation: While the high instructor-to-student ratio was crucial for success, the authors suggest that the core model (hands-on + pilot studies + reflection) can be adapted for larger classes by shifting toward group feedback mechanisms.
• Future of SE Education: As AI and other emerging technologies continue to dominate the industry, this approach offers a blueprint for teaching students not just how to use new tools, but how to scientifically evaluate their impact, productivity claims, and risks.

In conclusion, the authors successfully demonstrated that a seminar centered on AI coding assistants can effectively teach empirical thinking, provided the course is carefully structured to balance technical exploration with rigorous methodological training.