From Objective AI Literacy Assessment to Training Needs Mapping in Greek Secondary Education

Technical Summary: From Objective AI Literacy Assessment to Training Needs Mapping in Greek Secondary Education

Problem Statement
While Artificial Intelligence (AI) is increasingly embedded in educational infrastructure, empirical evidence regarding teachers' AI literacy remains limited. Existing research predominantly relies on self-reported readiness, confidence, or attitudes, which are susceptible to social desirability bias, inaccurate self-calibration, and contextual inflation. This reliance on subjective measures obscures specific conceptual misconceptions and uneven knowledge coverage. In the context of Greek secondary education, where teachers operate across diverse disciplinary specialties and uneven professional-learning pathways, the lack of a localized baseline of objective AI knowledge risks professional development (PD) becoming either too generic to be useful or too technical to align with actual needs. The study addresses the gap between "AI awareness" (familiarity with tools) and "AI literacy" (stable conceptual resources for interpreting system behavior), arguing that teachers require functional conceptual knowledge to evaluate AI outputs, understand limitations, and make responsible professional judgments.

Methodology
The study employed a cross-sectional online survey conducted in June 2025 with a sample of 271 Greek secondary education teachers (mainstream and special education). Recruitment utilized a convenience sampling strategy through professional and personal networks.

• Instrument: The study utilized a Greek adaptation of a 26-item objective AI literacy knowledge test originally developed by Ding et al. [13], grounded in the competency-oriented framework of Long and Magerko [8]. The instrument was translated and back-translated to ensure conceptual equivalence.
• Scoring: Responses were scored dichotomously (1 for correct, 0 for incorrect or "I don't know"). The inclusion of the "I don't know" option allowed for a conservative estimate of functional knowledge, distinguishing between confident answers and uncertainty.
• Analysis:
  • Descriptive Statistics: Total scores and item-level proportions correct ($p_{correct}$) were calculated.
  • Group Differences: Due to non-normal distribution of total scores (Shapiro-Wilk $p < .001$), non-parametric Kruskal-Wallis tests with Dwass-Steel-Critchlow-Fligner post-hoc comparisons were used to examine differences by teaching specialty, ICT training, and demographics.
  • Predictors: Spearman correlations and multiple linear regression (using bias-corrected bootstrap confidence intervals) were used to identify associations between teacher characteristics, prior training, and AI literacy scores.
  • Needs Mapping: A "NeedIndex" was calculated for each item as $1 - p_{correct}$. This metric transformed item-level deficits into a prioritization tool for professional development, where higher values indicate greater training needs.

Key Results

• Overall Performance: The sample demonstrated a moderate baseline objective AI literacy ($M = 15.82/26$, $SD = 3.53$; overall $p_{correct} = 60.85\%$). However, performance varied significantly across items ($p_{correct}$ range: 0.09–0.90).
• Conceptual Gaps (Lowest Performing Items): The most significant knowledge deficits clustered around foundational technical concepts:
  • Distinguishing automation from AI (e.g., automatic washing machines).
  • Differentiating narrow AI from general AI (e.g., voice assistants like Siri/Alexa).
  • Understanding the machine-learning workflow (data collection $\to$ training $\to$ deployment).
  • Matching AI techniques to applications.
  • Recognizing limits in AI perception and generalization (performance in unfamiliar contexts).
• Strongest Performing Items: Teachers performed relatively well on items concerning data interpretation, the misconception that AI is always smarter than humans, and the use of AI in dangerous work.
• Predictors of Literacy:
  • Significant Factors: Teaching specialty (ICT teachers scored higher than non-ICT groups), level of ICT training, self-reported PC/Internet familiarity, and exposure to AI-specific training were significantly associated with higher scores.
  • Non-Significant Factors: Demographic variables such as age, gender, highest degree, years of service, and work area (urban/rural) showed no statistically significant association with objective performance.
  • Training Impact: Regression analysis indicated that AI-specific training exposure was the dominant predictor. Teachers with semester/annual programs, certified knowledge, or undergraduate AI coursework scored significantly higher than those with only short-term seminars or no training.

Key Contributions

1. Objective Baseline: The study provides the first localized, objective baseline of AI literacy for Greek secondary teachers, moving beyond self-reported metrics to identify actual conceptual understanding.
2. Diagnostic Granularity: By utilizing item-level analysis, the study reveals that a "moderate" average score conceals substantial conceptual unevenness. Teachers may possess strong ethical or data-related knowledge while lacking foundational technical understanding.
3. Training Needs Mapping (NeedIndex): The study introduces a practical mechanism ($NeedIndex = 1 - p_{correct}$) to translate assessment data directly into professional development priorities. This allows PD designers to sequence content based on observed conceptual deficits rather than assumed relevance.
4. Differentiated PD Model: The findings support a tiered professional development model:
   • Tier 1: Foundational technical concepts (AI vs. automation, ML workflows).
   • Tier 2: Capabilities, limitations, and uncertainty.
   • Tier 3: Classroom-relevant pedagogical application.
   • Tier 4: Ethics, governance, and accountability.

Significance and Claims
The authors position this work as an applied baseline and training-needs mapping study rather than a comprehensive validation of the instrument or a predictor of classroom practice. The primary significance lies in shifting the focus from "AI awareness" to "conceptual accuracy." The study argues that without a stable understanding of how AI systems function (e.g., data dependence, probabilistic outputs, generalization limits), teachers cannot effectively evaluate AI tools, explain system failures to students, or distinguish realistic educational affordances from inflated promises.

The findings challenge the assumption that AI literacy is a generational issue or a byproduct of general teaching experience. Instead, the study claims that AI literacy is a trainable professional knowledge domain strongly linked to disciplinary background and structured learning opportunities. Consequently, the authors advocate for evidence-informed capacity building that prioritizes foundational technical concepts before moving to ethical or pedagogical applications, ensuring that teachers possess the conceptual resources necessary for responsible human oversight in AI-enabled schools.

Limitations
The authors explicitly note several limitations: the use of a convenience sample limits generalizability to the entire Greek teacher population; the cross-sectional design precludes causal inference regarding the impact of training; the instrument measures conceptual knowledge rather than pedagogical enactment; and the 26-item test represents a breadth-depth trade-off that cannot cover the full complexity of AI literacy (e.g., generative AI nuances). Future research is recommended to combine objective testing with scenario-based and practice-based measures.