💻 computer science

Learning Profiling with Automated Feedback for Supporting the Assessment of Cybersecurity Education

This paper presents an AI-driven profiling model within a cyber range environment that continuously evaluates learner performance to generate automated, personalized feedback and adaptive learning paths, thereby addressing key challenges in scalability and personalization for cybersecurity education.

Original authors: Willi Lazarov, Pavel Seda, Ondrej Pospisil, Milos Seda, Zdenek Martinasek

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Willi Lazarov, Pavel Seda, Ondrej Pospisil, Milos Seda, Zdenek Martinasek

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling city where everyone is connected. Like any city, it has its share of troublemakers—hackers, scammers, and digital vandals—who try to break into homes, steal mail, or shut down power plants. To keep this city safe, we don't just need better locks; we need to train the people who build and guard the locks. This is the world of cybersecurity education. Traditionally, learning this trade has been a bit like watching a cooking show: you sit and listen to a teacher explain the theory, or maybe you watch a video. But to actually learn how to cook (or hack), you need to get your hands dirty in the kitchen. In the digital world, this "kitchen" is called a cyber range: a safe, isolated sandbox where students can practice breaking and fixing things without accidentally burning down the real city.

However, there's a problem with these digital kitchens. When a student is practicing, a teacher can't be hovering over their shoulder for every single move, especially if there are thirty students cooking at once. If a student makes a mistake, they might just get a score, like "You got 8 out of 10," but they don't know why they failed or what to do next. It's like getting a test back with a red "F" but no comments on how to study better. This paper asks: Can we build a smart system that acts like a super-attentive coach, watching every move a student makes, understanding their unique strengths and weaknesses, and then giving them a personalized report card and a custom training plan?

The authors of this paper, a team from Brno University of Technology, say yes. They have built a system that combines a cyber range with artificial intelligence to create a "learning profile" for every student. Think of this profile not as a static report card, but as a living, breathing map of a student's skills. As a student tackles cybersecurity challenges (often in a game-like format called Capture the Flag, where you solve puzzles to find hidden "flags"), the system tracks everything: how long they took, how many times they guessed wrong, whether they asked for hints, and how hard the puzzle was.

Instead of just tallying up points, the system uses a mathematical formula to weigh these factors. It asks: Did the student solve the puzzle quickly but with many wrong guesses? Did they breeze through easy tasks but get stuck on the hard ones? Did they rely too much on hints? By crunching these numbers, the system assigns a "profiling score" that tells the story of how the student learned, not just what they got right.

Here is where the magic happens: this score is fed into a Large Language Model (LLM), which is essentially a very advanced AI that can read and write human language. The AI takes the cold, hard numbers and turns them into a warm, helpful conversation. For the student, it generates a personalized feedback report that says things like, "You're a wizard at finding hidden network vulnerabilities, but you tend to rush through the setup phase. Next time, try slowing down on the first step." It even suggests the perfect next challenge: if you aced the easy stuff, it pushes you to a harder level; if you struggled, it offers a slightly easier path to build your confidence.

For the teachers, the system acts like a crystal ball. It aggregates all the student data to show the big picture. It can tell the instructor, "Your whole class is great at hacking, but everyone is struggling with the same specific type of firewall defense," or "This particular task was way too hard for everyone; maybe we need to tweak it."

To prove this works, the team tested their system with 35 real university students. They set up a scenario where the students had to act like digital detectives and hackers, trying to flood a fake website or break into a server. The results were promising. The system successfully tracked the students' progress, calculated their unique profiles, and generated detailed, readable feedback that felt like it came from a human mentor. The study suggests that this approach is technically feasible and offers a way to make cybersecurity training more personal and effective, moving away from one-size-fits-all grading toward a system that truly understands how each learner thinks and struggles. While the paper doesn't claim this solves every problem in education, it shows a clear path toward a future where AI helps teachers give every student the specific guidance they need to become a cybersecurity expert.

Technical Summary: Learning Profiling with Automated Feedback for Supporting the Assessment of Cybersecurity Education

Problem Statement
Cybersecurity education faces significant challenges in scaling effective assessment and personalization. While traditional methods range from passive learning to hands-on exercises in cyber ranges and gamified Capture the Flag (CTF) scenarios, existing approaches struggle with objective learner assessment, feedback scalability, and personalization. Current scoring systems often rely on simple point accumulation, which fails to capture the nuance of skill development, such as the time taken, the number of attempts, or the reliance on hints. Furthermore, manual supervision is insufficient for large groups, and existing adaptive frameworks often require complex, error-prone manual configuration of fine-grained performance weights for every task phase. There is a critical gap in providing continuous, individualized learner profiling and automated, descriptive feedback that can guide adaptive learning paths without overburdening instructors.

Methodology
The authors propose an innovative profiling model integrated within a cyber range environment (specifically the Brno University of Technology Cyber Arena, or BUTCA) to address these gaps. The methodology consists of three primary components:

Quantitative Profiling Model:
The system evaluates learner performance across a series of scenarios ( $S_i$ ) containing multiple tasks ( $T_{ij}$ ). The model calculates a weighted profiling score ( $\tilde{P}_w(S_i)$ ) by normalizing and aggregating three key dimensions:
- Performance Score ( $\tilde{p}$ ): Derived from base points adjusted for task complexity ( $C_{ij}$ ) and penalized for hint usage ( $r_{ij}$ ) and incorrect final test answers ( $r_{ijq}$ ).
- Time Efficiency ( $\tilde{\Delta t}$ ): The difference between actual solution time and expected time, normalized to penalize excessive duration while capping rewards for extremely rapid completion.
- Accuracy ( $\tilde{E}$ ): The relative number of incorrect attempts, normalized by task complexity.
These dimensions are combined using global weights ( $w_1, w_2, w_3$ ) to produce a scalar score $\pi(S_i) \in [0, 1]$ . This score determines the learner's transition to the next scenario ( $S_y$ ) via a deterministic function, effectively creating adaptive learning paths that align difficulty with the learner's evolving needs.
Automated Feedback Generation System:
To transform numerical profiling data into pedagogical insights, the system employs Large Language Models (LLMs). To address context window limitations and ensure data accuracy, the authors utilize a deterministic preprocessing pipeline in Python. This pipeline filters and aggregates raw data (task scores, time, attempts, hints) into structured JSON payloads before they are passed to the LLM.
- Student Feedback: Generated via a specific prompt schema to provide personalized summaries, identify strengths and weaknesses, explain test errors, and recommend next steps.
- Instructor Feedback: Generated via a separate schema to provide aggregated group statistics, compare group performance against global benchmarks, and highlight difficult or easy tasks based on collective data.
- LLM Options: The system supports both local LLMs (e.g., Gemma-7b-it) for data privacy and cloud-based APIs (e.g., Llama-3-70B, GPT-4.1-mini) for scalability, with the preprocessing step ensuring that raw user data is not directly exposed to the model in a way that violates context limits.
Integration Environment:
The model operates within a CTF-based workflow where students interact with isolated sandbox environments (virtual machines). Data collection occurs at the operational layer, analytics at the management layer, and feedback delivery at the learning layer.

Key Contributions
The paper outlines three primary contributions:

An Innovative Profiling Model: A formalized method for assessing learner performance during hands-on training that moves beyond simple point-scoring to include time, attempts, and hint usage, normalized and weighted to produce a holistic profile.
Automated Feedback System: A dual-output system that generates individualized, descriptive feedback for students (including strength/weakness analysis and error explanations) and aggregated performance summaries for instructors, all driven by LLMs and a deterministic data pipeline.
Empirical Validation: A case study demonstrating the technical feasibility of the approach within a real cyber range environment, integrating the profiling model with automated feedback generation.

Results
The proposed approach was validated through a case study involving 35 third-year Information Security students at Brno University of Technology. The students completed a single training scenario focused on penetration testing and resilience against flooding/logical attacks, consisting of eight tasks of varying difficulty.

Technical Feasibility: The study confirmed that the profiling model could successfully calculate scores and determine scenario transitions.
Feedback Generation: The system successfully generated structured feedback using both local (Gemma-7b-it) and cloud-based (Llama-3-70B, GPT-4.1-mini) LLMs. The deterministic preprocessing pipeline effectively managed the large volume of data (approx. 97k tokens) by reducing it to context-window-compliant payloads.
Output Quality: The generated feedback included personalized scenario summaries, topic comprehension analysis, strength/weakness identification, and detailed explanations for incorrect test answers, as well as aggregated statistical insights for the instructor.

Significance and Claims
The authors position this work as a data-driven method to bridge the gap in cybersecurity education assessment. They claim that by integrating profiling with automated feedback, the system:

Objectifies Assessment: Moves beyond simple scoring to provide a nuanced view of learner competence based on multiple performance indicators.
Enables Personalization: Facilitates continuous development by recommending adaptive learning paths that match the learner's current proficiency level.
Supports Instructors: Reduces the cognitive load on educators by providing aggregated insights and eliminating the need for manual, fine-grained weight calibration for every task phase, a limitation identified in prior adaptive frameworks.
Scalability: Demonstrates that automated profiling and feedback can be deployed in controlled cyber range environments to support structured, individualized assessments for learners and aggregated summaries for instructors.

The paper concludes modestly, stating that the case study demonstrates the technical feasibility of the approach, contributing a method for objective assessment and personalized training, while acknowledging that future research directions remain open.

More like this