Chatbot-Based Assessment of Code Understanding in Automated Programming Assessment Systems

1. Problem Statement

The rapid advancement of Large Language Models (LLMs) has fundamentally disrupted automated programming assessment. While traditional Automated Programming Assessment Systems (APASs) rely on unit tests and static analysis to verify functional correctness, these methods are no longer sufficient. Students can now generate functionally correct code using LLMs without possessing the underlying conceptual understanding.

This creates a critical gap: current systems cannot distinguish between a student who understands the logic and one who has simply copied or generated a solution. There is a lack of widely adopted, scalable design patterns for integrating conversational agents into APASs to verify code comprehension (the ability to explain logic, execution traces, and design decisions) rather than just output correctness.

2. Methodology

The paper employs a two-part research design:

A. Systematic Scoping Review

• Approach: A saturation-based scoping review following a PRISMA-style workflow.
• Search Strategy: Five search strings based on the PICOC framework (Population, Intervention, Comparison, Outcome, Context) were executed across major databases (ACM, IEEE, arXiv, etc.) between October and November 2025.
• Selection Criteria: Peer-reviewed papers and gray literature from 2018–2025 focusing on conversational agents/chatbots used specifically for assessing code understanding in programming education.
• Process:
  • Initial search yielded 123 papers; 24 were eligible for full review.
  • Snowballing (backward and forward) was applied to eligible papers, resulting in a final corpus of 31 primary studies.
  • Studies were categorized by technical architecture, pedagogical strategy, and assessment methods.

B. Framework Synthesis

• Based on the review findings, the authors derived a design proposal (the Hybrid Socratic Framework) intended to bridge the gap between deterministic code analysis and generative dialogue.

3. Key Contributions

A. Taxonomy of Conversational Assessment Approaches

The review identified three dominant architectural families in the literature:

1. Rule-Based/Template-Driven Systems:
   • Mechanism: Use deterministic algorithms (AST analysis, dynamic execution traces) and predefined templates to generate questions.
   • Pros: High consistency, auditability, low computational cost.
   • Cons: Rigid, repetitive, high maintenance cost for expanding templates, poor handling of novel code patterns.
2. LLM-Based Systems:
   • Mechanism: Use generative models (GPT, Llama, etc.) to create natural, multi-turn dialogues that adapt to student input.
   • Pros: High flexibility, natural language fluency, ability to handle diverse code patterns.
   • Cons: Prone to hallucinations, inconsistent grading, risk of providing direct solutions instead of scaffolding, high computational cost.
3. Hybrid Systems:
   • Mechanism: Combine deterministic code analysis (for fact extraction) with ML components (for dialogue management and evaluation).
   • Pros: Balances scalability with control; reduces the fragility of rule-based systems and the unpredictability of pure LLMs.
   • Cons: Higher architectural complexity and integration effort.
   • Finding: Hybrid approaches were the most frequent (5/12) in the reviewed corpus, suggesting they are the current state-of-the-art direction.

B. The Hybrid Socratic Framework

The paper proposes a novel framework for integrating conversational verification into APASs. Key technical components include:

• Dual-Agent Architecture:
  • Instructor Agent: Generates Socratic questions based on deterministic facts extracted from the student's code (e.g., specific variable states, loop boundaries). It is strictly constrained to ask questions, not provide solutions.
  • Verifier Agent: Evaluates student responses against a reference reason grounded in the code analysis output, checking for conceptual alignment rather than keyword matching.
• State Space & Knowledge Tracker: Maintains a persistent model of the student's understanding (knowledge components and misconceptions) to drive adaptive questioning.
• Code Analysis Engine: Performs static (AST) and dynamic (execution trace) analysis to generate "ground truth" facts that constrain the LLM's prompts.
• Socratic Guardrails: System prompts and constraints that prevent the AI from acting as a solution-giving assistant, forcing it to redirect students toward reasoning about execution states.

C. Integrity Safeguards

To counter students using LLMs to generate their own explanations, the framework introduces specific technical safeguards:

• Randomized Trace Questions: Questions are tied to specific, randomized runtime states of the student's code, making pre-generated or generic LLM answers useless.
• Stepwise Reasoning: Students must predict intermediate states (e.g., "What is the value of i before the final iteration?") rather than providing a single polished summary.
• Proctored Deployment: High-stakes assessments require supervised or remote-proctored sessions to ensure the student is the one reasoning.

4. Results and Findings

• Pedagogical Gap: The review confirmed a significant disconnect between "unproductive success" (passing tests via trial/error or copying) and actual understanding. Students often pass static structure questions (>80% success) but fail dynamic execution questions (<50% success).
• Benefits: Conversational agents offer scalable, personalized feedback and can reduce instructor burden while increasing student engagement.
• Limitations: Technical risks include LLM hallucinations and inconsistent outputs. Behavioral risks include student over-reliance on AI for debugging and "gaming" the system.
• Design Requirements: Effective systems must ground questions in deterministic facts, support multi-turn probing, discourage answer copying via runtime-specific prompts, and separate formative support from summative grading.

5. Significance and Future Directions

• Paradigm Shift: The paper argues that APASs must evolve from purely functional testing to conversational verification of reasoning. The goal is not to ban AI but to use constrained conversational agents to verify that students can reason about the code they submit.
• Practical Implementation: The authors developed a Proof of Concept (PoC) prototype using a Python/Streamlit stack with a dual-agent architecture. It supports model-agnostic backends (including local models for privacy) and demonstrates the end-to-end flow of scenario-based questioning and semantic grading.
• Limitations of the Study: The proposed framework is a design synthesis, not a fully validated product. It lacks large-scale classroom deployment data, inter-rater reliability studies against human examiners, and systematic evaluation of latency/cost.
• Future Work: Priorities include empirical validation in classrooms, improving knowledge tracing across sessions, exploring multi-modal explanations (diagrams/traces), and investigating the ethical implications of automated grading.

Conclusion: The paper establishes that while LLMs challenge traditional assessment, a Hybrid Socratic Framework—anchoring generative dialogue in deterministic code analysis—offers a viable path to scalable, integrity-preserving assessment of deep code understanding.