Towards Personalizing Secure Programming Education with LLM-Injected Vulnerabilities

1. Problem Statement

Secure programming education faces a persistent challenge: generic, textbook-style examples of software vulnerabilities often lack contextual relevance. Students struggle to connect abstract security concepts (Common Weakness Enumerations, or CWEs) to their own coding practices, leading to low engagement and poor internalization of security principles. Traditional instructional materials fail to leverage constructivist learning theory, which posits that students learn more effectively when new concepts are anchored in their prior knowledge and personal experience. Furthermore, manually personalizing educational materials for every student is prohibitively labor-intensive for instructors.

2. Methodology

The authors propose and evaluate InjectEd, a modular Agentic AI system designed to automate the injection of security vulnerabilities into student-authored code.

System Architecture

InjectEd utilizes a pipeline of four autonomous agents orchestrated via CrewAI, supported by Langfuse for prompt management and observability. The system uses GPT-4o-mini as the underlying LLM.

1. Injector Agent: Parses student code using an Abstract Syntax Tree (AST) to identify viable modification points (e.g., unchecked inputs). It injects specific CWEs while preserving syntactic and semantic integrity. It employs syntax validation tools to prevent hallucinations and lightweight test suites to ensure core functionality remains intact aside from the vulnerability.
2. Evaluator Agent: Assesses the pedagogical quality of injected variants using structural (Ratcliff/Obershelp algorithm) and semantic (Gensim embeddings) similarity scores. It rates the injection on dimensions like relevance, naturalness, and cognitive load.
3. Ranker Agent: Selects the single most pedagogically valuable injection from the candidates based on the Evaluator's scores and qualitative rationales.
4. Learning Outcome Generator: Creates formative assessments (MCQs, reflection prompts) based on the selected injection.

Experimental Design

• Participants: $N=71$ students from two upper-level undergraduate courses (Software Engineering and Database Systems).
• Setup: A controlled study comparing two groups:
  • Treatment Group: Received a version of their own project code with an LLM-injected CWE.
  • Control Group: Received a matched, generic textbook-style example illustrating the same CWE.
• Hybrid Execution: While the full system includes four agents, this specific study focused on the Injector Agent. The roles of the Evaluator, Ranker, and Learning Outcome Generator were performed manually by the course instructor to ensure high instructional integrity and consistency.
• Data Collection: A post-project survey using 12 Likert-scale items (measuring learning impact and engagement) and open-ended questions.
• Analysis: Non-parametric Mann–Whitney U tests were used for quantitative data; thematic analysis was used for qualitative responses.

3. Key Contributions

1. Novel Injection Method: A technique for using LLMs to embed specific, pedagogically aligned CWEs directly into student-authored codebases while preserving local semantics.
2. Agentic AI Framework: The design of InjectEd, a scalable system using autonomous agents and tools (AST parsing, semantic scoring) to orchestrate vulnerability generation, evaluation, and ranking.
3. Empirical Evaluation: A comparative study demonstrating that while quantitative learning gains were not statistically significant, personalized injections significantly reduced student confusion and increased perceived relevance compared to generic examples.

4. Results

The study yielded mixed quantitative results but strong qualitative insights:

• Quantitative Findings:

  • No statistically significant differences ($p > .05$) were found in primary learning outcomes such as "Understanding," "Confidence," or "Fixing Capability" between the InjectEd and Baseline groups.
  • Significant Finding: There was a statistically significant reduction in Confusion for the InjectEd group ($p < .05$, $r=2.53$). This suggests that seeing flaws in their own code made the concepts clearer and less ambiguous than textbook examples.
  • Trends favored the InjectEd group in "Real-world Connection" and "Fixing Capability," though not statistically significant.

• Qualitative Findings:

  • Students described the personalized injections as "eye-opening" and "interesting."
  • Participants reported that seeing vulnerabilities in their own logic prompted deeper reflection and a realization that their code was less secure than they thought.
  • Some students noted the injected flaws were "subtle" and "realistic," enhancing the authenticity of the learning experience.
  • A minority of students experienced confusion due to code complexity or a perceived mismatch between the CWE and their specific assignment logic.

5. Significance and Future Work

Significance:
The paper demonstrates that personalization via LLMs is a viable and effective strategy for secure programming education. While it did not immediately prove a massive boost in test scores, it successfully lowered cognitive friction (confusion) and increased student engagement by grounding abstract concepts in familiar contexts. This validates the constructivist approach in the domain of software security.

Future Work:

• Refinement of Selection: Integrating the full Ranker Agent to better match CWEs to specific student code semantics, reducing instances where the injection feels forced.
• Longitudinal Studies: Moving beyond single-time interventions to evaluate the cumulative effect of repeated injections across multiple projects.
• Automated Validation: Developing robust automated mechanisms to standardize injection quality and reduce reliance on manual instructor review.
• Expanded Scope: Testing the system across more diverse courses, institutions, and a wider variety of vulnerability types to improve generalizability.