Teaching testing seriously in academia

Technical Summary: Teaching Testing Seriously in Academia

Problem Statement
The paper identifies a critical misalignment between software testing education in academia and professional practice, particularly as systems grow more complex and incorporate AI. Current curricula predominantly operate under a rationalist paradigm, framing testing as a prescriptive, algorithmic process of confirming expected outcomes against clear specifications. This approach, reflected in standard textbooks and methods like Equivalence Class Partitioning (ECP) and Boundary Value Analysis (BVA) taught as rigid rules, fails to cultivate critical thinking. Consequently, students develop narrow views of testing, lack motivation, and struggle to reason under uncertainty. The authors argue that this paradigm is fundamentally flawed for modern contexts, especially with generative AI, where "clear requirements" often do not exist, and systems exhibit emergent, unpredictable behaviors. The core issue is that testing is taught as a solution-finding exercise rather than an empirical, inquiry-driven investigation.

Methodology and Instructional Design
To address these structural problems, the authors propose a shift from rationalism to empiricism, treating testing as a complex, whole-task professional skill. The methodology is grounded in the Four-Component Instructional Design (4C/ID) model, which supports the learning of complex skills through integrated whole tasks rather than isolated techniques.

The paper introduces P4TEST, a pedagogical framework designed to operationalize this vision. P4TEST structures testing education around:

1. Core Competencies: Modelling (reasoning about the system and context) and Empirical Investigation (gathering evidence through observation).
2. Epistemic Moves: Sensemaking (framing and revising explanations) and Reflection-in-action (re-framing actions in real-time).
3. Habits of Mind: Disciplined scepticism, epistemic openness, tolerance for the unknown, and deliberate curiosity.

The curriculum is organized into four Task Classes (TC) that increase in complexity and autonomy while systematically fading instructional scaffolding:

• TC1 (Analysis): Students analyze worked-out examples (e.g., testing a pen or an algorithm) to recognize P4TEST elements and alternative test ideas.
• TC2 (Guided Execution): Students test simple, constrained Systems Under Test (SUTs) using structured support to integrate modelling and exploration.
• TC3 (Autonomous Execution): Students test more complex, less constrained systems, managing trade-offs and engaging in reflection-in-action with minimal prompts.
• TC4 (Authentic Practice): Students perform realistic testing assignments with high autonomy, justifying evidence-based evaluations of quality and risk.

To support the development of specific "habits of mind" often neglected in traditional education, the authors propose puzzle-based learning as a supportive didactic strategy. Unlike game-based bug-finding competitions, these puzzles present situations with multiple plausible solutions, forcing learners to question assumptions and explore alternative explanations.

Key Contributions

• P4TEST Framework: A pedagogical framework that explicitly maps the core competencies, epistemic moves, and habits of mind required for professional testing, distinguishing between the "product space" (reasoning about the whole system) and "assessment space" (focused investigation).
• 4C/ID Operationalization: A concrete curriculum design that sequences learning tasks from high-scaffolding analysis to autonomous authentic practice, avoiding the decomposition of testing into linear procedures.
• Assessment Innovation: A proposal to move away from traditional written exams and retrospective reports, which obscure reasoning processes. Instead, the paper advocates for video-based assessment (screen recording with think-aloud verbalization) to evaluate observable epistemic moves, decision-making under uncertainty, and reflection-in-action, similar to clinical reasoning assessments in medicine.
• Reframing Scaffolding: A theoretical resolution to the tension between the need for educational scaffolding and the non-linear nature of expert testing. The authors argue that scaffolding (e.g., teaching ECP/BVA) should be viewed as a tool for learning the "epistemic game" of testing, not as a prescriptive doctrine for professional practice.

Results and Claims
As a position paper and design instrument, the work does not present empirical results from a deployed curriculum. Instead, it offers a theoretical and structural argument supported by existing literature on learning complex skills (van Merriënboer et al.) and the context-driven testing school (Kaner, Bach, Bolton).

The paper claims that:

1. Current rationalist approaches fail to prepare students for the uncertainty inherent in AI-driven and complex systems.
2. The proposed 4C/ID-based design, guided by P4TEST, provides a viable pathway to teach testing as an empirical inquiry rather than a verification procedure.
3. Video-based assessment is a necessary evolution to capture the "whole-task" performance and cognitive processes that written exams miss.

Significance
The paper positions itself as a foundational design instrument for redesigning testing curricula at bachelor, master, and potentially vocational levels. Its significance lies in providing a structured alternative to the prevailing "checklist" approach to testing education. By explicitly linking the how of teaching (4C/ID) with the what of testing (P4TEST), it offers educators a way to cultivate the critical, inquiry-driven mindset required for modern software quality assurance. The authors emphasize that P4TEST is a scaffold for thinking about testing, intended to be used and eventually outgrown by students, rather than a rigid set of rules. Future work involves exploring behavioral logging as a complementary evidence source and conducting experiments to evaluate the framework's utility in actual academic programs.