Investigating Conversational Agents to Support Secondary School Students Learning CSP

1. Problem Statement

Secondary school students enrolled in the AP Computer Science Principles (CSP) course frequently rely on web resources (tutorials, Q&A sites) to understand complex concepts. However, effective utilization of these resources is hindered by information literacy challenges, specifically the difficulty of finding content appropriate for their learning level and background.

• The Core Issue: Traditional web search often fails to support "exploratory search" (learning and investigative intents) because it requires students to filter vast amounts of irrelevant or overly complex information.
• The Proposed Solution: The authors investigate the potential of Conversational Agents (CAs) as an alternative interface. They compare two distinct types of CAs:
  1. Fixed-Response Agents: Rule-based systems with pre-defined, curriculum-aligned responses.
  2. Generative Agents: Large Language Model (LLM) based systems (e.g., ChatGPT) that generate dynamic responses.

2. Methodology

The study employed a within-subject experimental design involving 45 high school students (grades 9–11, ages 14–17) across six AP CSP sections.

A. Experimental Conditions

Students interacted with three different tools to complete two exploratory search tasks:

1. Aida (Fixed-Response): A custom agent built on Google DialogFlow ES.
   • Design: Follows the Think-Pair-Share pedagogical pattern (Warm-up, Activity, Wrap-up) and the Socratic method.
   • Mechanism: Uses rule-based grammar and ML intent matching to guide students through predefined intents. It provides real-world analogies, asks clarifying questions, and strictly adheres to the AP CSP curriculum scope (excluding advanced topics like Big-O notation).
2. ChatGPT (Generative): Used the GPT-3.5-turbo model via a HuggingFace interface.
   • Design: General-purpose, no fine-tuning or custom system prompts were applied to simulate a standard student experience.
   • Mechanism: Generates responses based on training data without explicit curriculum constraints.
3. Google Search (Baseline): Conventional web search engine used as a control group.

B. Tasks

Students completed two tasks aligned with the CSP curriculum:

1. Binary/Linear Search Task (Comparison): Compare and contrast the algorithms, efficiency, and use cases.
2. API/Library Task (Knowledge Acquisition): Explain the components of an API/library and how to document them.

C. Data Collection & Metrics

Data was collected via dialogue logs, browser history, and post-study surveys. The analysis focused on:

• Effectiveness:
  • Completeness: Proportion of required criteria met in the student's final answer.
  • Accuracy: Factual correctness of the information provided.
  • Extraneousness: Amount of irrelevant information in the answer.
  • Scope Adherence: Frequency of "out-of-scope" content (e.g., content excluded from the AP exam, such as Big-O analysis).
• Engagement:
  • Exploratory Actions: Number of messages sent (for CAs) or websites visited (for Google).
  • Task Duration: Time spent on the task.
• Perception: Likert-scale surveys and open-ended feedback regarding usefulness, helpfulness, and ease of use.

3. Key Contributions

1. Pedagogical Agent Design: Development of Aida, a fixed-response agent specifically engineered for CSP using the Think-Pair-Share framework and Socratic questioning.
2. Comparative Evaluation: A rigorous empirical study comparing fixed-response agents, generative LLMs, and traditional search engines in a secondary school setting.
3. New Metrics: Introduction of specific metrics for evaluating CA effectiveness in education, including "out-of-scope" detection and "extraneousness" scoring.
4. Insights on Generative AI in Education: Evidence regarding the trade-offs between the high engagement/perceived utility of LLMs and the pedagogical safety of fixed-response systems.

4. Key Results

A. Effectiveness

• Completeness: Students using ChatGPT produced the most complete answers (Mean = 0.48), significantly higher than Aida (0.33) and Google (0.38).
• Accuracy: Aida produced the most accurate answers (Median = 1.0), followed by ChatGPT (0.86) and Google (0.83). ChatGPT and Google occasionally included factual errors or hallucinations.
• Scope Adherence (Critical Finding):
  • Aida had 0 "out-of-scope" instances.
  • ChatGPT frequently provided content outside the AP CSP curriculum (e.g., Big-O notation, recursive implementations), with a mean of 1.59 out-of-scope instances per session.
  • Google had the highest variability and highest mean (2.39) of out-of-scope content, often leading students to complex coding implementations not required for the course.
• Extraneousness: Aida produced the least irrelevant information. Google results often required students to copy/paste large blocks of text, increasing extraneousness.

B. Engagement

• Interaction Depth: Aida generated the highest number of exploratory actions (Median = 10 messages) and the longest task duration (Median = 12 mins), indicating deep engagement.
• Efficiency vs. Depth: ChatGPT required the fewest actions (Median = 4) and shortest duration (Median = 5.8 mins). Google was the fastest (Median = 5.5 mins) but involved the most "copy-paste" behavior.
• Student Preference: Despite Aida's pedagogical depth, students strongly preferred ChatGPT.
  • 90% agreed ChatGPT helped them understand the user well, compared to lower scores for Aida.
  • Students found Aida's rigid questioning "linear" and "roadblocking," whereas they appreciated ChatGPT's flexibility and speed.

C. Qualitative Insights

• Trust: Students trusted ChatGPT and Aida more than Google search results, often viewing web sources as difficult to verify or too technical.
• Pedagogy vs. Convenience: While Aida's Socratic method was designed to foster critical thinking, many students found it frustrating and preferred the "instant answer" nature of ChatGPT.
• Risk of "Lazy Learners": The ease of ChatGPT raised concerns that students might accept answers without deep processing, though the study noted students still engaged in some synthesis.

5. Significance and Implications

• The "Accuracy vs. Engagement" Trade-off: The study highlights a critical tension in AI education tools. Generative agents (ChatGPT) offer superior user experience, engagement, and answer completeness but risk providing inaccurate or curriculum-inappropriate content (hallucinations, advanced topics). Fixed-response agents (Aida) ensure pedagogical safety and accuracy but suffer from lower user engagement and perceived rigidity.
• Curriculum Alignment: Generative models, by default, do not respect specific educational boundaries (e.g., AP exclusion statements). Without customization, they can overwhelm students with information beyond their current learning level.
• Future Directions: The authors propose a hybrid approach that combines the pedagogical structure and safety of fixed-response agents with the flexibility and natural language capabilities of generative models. This would aim to provide the "best of both worlds": high engagement with curriculum-aligned, accurate content.
• Educational Policy: The findings suggest a need for explicit instruction on prompt engineering and critical evaluation of AI outputs, as students currently lack the literacy to distinguish between helpful scaffolding and misleading information in generative models.

In conclusion, while generative AI is currently more popular and perceived as more effective by students, fixed-response agents remain superior for ensuring curriculum fidelity and factual accuracy. The future of AI in CS education likely lies in hybrid systems that leverage LLMs for generation while constraining them within pedagogical guardrails.