Integrated Validation Framework for Educational Module Development (IVF-EMD): A Systematic Evidence Synthesis of Validation Practices and Methodological Gaps
This study synthesizes evidence from 51 educational development studies to reveal significant methodological fragmentation in current validation practices, leading to the proposal of the Integrated Validation Framework for Educational Module Development (IVF-EMD) as a comprehensive, four-domain architecture that reimagines validation as a progressive and iterative evidence-building process.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Educational modules are the structured packages of learning materials that teachers and institutions use to guide students through a subject. Whether it is a digital app for learning a new language, a printed guide for understanding local history, or a set of activities for a science class, these tools are designed to solve specific educational problems. Before a module is released for widespread use, developers must prove that it works. This process, known as validation, involves checking that the content is accurate, that it makes sense to the people who will use it, and that it actually helps students learn. For decades, researchers have relied on established blueprints, such as the ADDIE model, to organize the creation of these tools. These blueprints tell developers the order in which to build a product: first analyze the need, then design the solution, then create it, and finally test it. However, a gap has emerged between the plan for building a product and the proof that the product is good. While the blueprints tell us how to construct a module, they do not always ensure that every step of the proof is connected, leaving developers with pieces of evidence that do not form a complete picture.
A team of researchers from the University of Malaya set out to understand exactly how this proof is currently being gathered. They conducted a systematic review, which is a method of carefully collecting and analyzing many different studies to find the big picture. They examined 51 separate studies that described the creation and testing of educational modules. Their goal was not to invent new ways to test these tools, but to see how the existing methods were being used and where the gaps lay. They looked for nine specific types of evidence that researchers typically gather: starting with a needs analysis to see if the module is necessary, getting expert opinions, checking the content for accuracy, asking users if it looks clear, running small trial tests, checking if the tests are consistent, seeing if it is easy to use, watching it work in a real classroom, and finally measuring if it actually improves learning outcomes.
The researchers found that the field is full of good intentions but suffers from a lack of connection. Almost every study they reviewed, 98 percent, began by identifying a need for the module. A large majority, about 78 percent, also brought in experts to review the design, and 72 percent checked that the content was accurate. These early steps were very common. However, as the process moved forward, the evidence began to fall apart. Fewer studies checked if the module was easy for students to use, and even fewer ran the small trial tests or checked if the measurements were consistent. The most significant drop occurred when researchers tried to prove that the module worked in a real classroom and actually improved learning. While many studies claimed to have done these final steps, they often did so without having completed the earlier checks. When the researchers looked for studies that had done all nine steps in a connected way, they found only four. This means that less than 8 percent of the studies provided a complete chain of evidence from the initial idea to the final result.
The core problem identified is not that researchers are failing to test their modules, but that they are testing them in fragments. A developer might have a perfect expert review and a clear content check, but then skip the step of seeing if a student can actually use the tool. Another might jump straight to seeing if a module works in a classroom without ever checking if the content was accurate or if the tool was easy to use. This creates a situation where a module might be declared "validated" based on a single piece of evidence, even though the full picture is missing. The researchers argue that validation should not be seen as a checklist where you simply tick off boxes. Instead, it should be viewed as a process of building a case, where each piece of evidence supports a specific claim. If a developer claims their module is ready for the classroom, they need evidence of usability. If they claim it improves learning, they need evidence of effectiveness. The current practice often skips these links, making it difficult to know exactly what a "validated" module actually means.
To fix this, the researchers proposed a new way of organizing these checks, called the Integrated Validation Framework for Educational Module Development. This framework does not replace the existing blueprints for building modules. Instead, it sits alongside them, acting as a map for the evidence. It groups the nine types of checks into four connected domains. The first is Foundation Validation, which ensures there is a real need for the product. The second is Design Validation, which confirms that experts and content checks support the idea. The third is Product Validation, which focuses on whether the finished tool is usable, consistent, and ready for a trial. The final domain is Field Validation, which looks at how the tool performs in a real classroom and whether it achieves the desired learning outcomes.
This new structure emphasizes that these domains are connected and that evidence from later stages can inform earlier ones. If a module fails in a real classroom, for example, it might mean the design was flawed or the content was unclear, requiring the developers to go back and re-check those earlier steps. The framework suggests that the amount of proof a developer needs should match the strength of the claim they are making. A simple prototype might only need to prove it is a good idea and looks right, while a tool intended for national use needs proof that it works in real classrooms. By organizing the evidence this way, the framework helps researchers and developers be more honest and precise about what their modules can and cannot do. The study concludes that while the tools to test educational modules exist, the field needs a better system to connect them, ensuring that the final product is supported by a complete and coherent story of evidence.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.