Revisiting validation in language testing: an updated critical review of argument-based approaches in the era of justice-oriented and AI-mediated assessment
This paper updates a 2019 critical review of argument-based validation in language testing by examining developments through 2026, highlighting the field's expansion into diverse contexts and adjacent disciplines while addressing emerging challenges posed by justice-oriented critiques and AI-mediated scoring, ultimately arguing for the integration of fairness and algorithmic transparency as core warrants within validity arguments.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Every day, millions of people take standardized tests to prove they can speak a language well enough for a job, to enter a university, or to move to a new country. These scores act as gatekeepers, deciding who gets an opportunity and who does not. For decades, experts have asked a single, vital question: how do we know these tests actually measure what they claim to measure? The answer has evolved from simple checklists to a more complex method called the "argument-based approach." Imagine a lawyer building a case for a jury; instead of just listing facts, they must construct a logical chain of reasoning that connects a student's performance on a test to a real-world decision, like granting a visa. Each link in this chain—how the test is scored, how that score represents a student's true ability, and how that ability predicts future success—must be supported by evidence. If one link is weak, the entire case falls apart.
Recently, the landscape of language testing has shifted dramatically. Two powerful forces are reshaping how these tests are built and judged. First, there is a growing demand for fairness, with researchers arguing that tests must actively work against racial and social inequalities, not just avoid obvious errors. Second, artificial intelligence is now grading many of these tests, replacing human readers with computer algorithms that can process language at incredible speeds. A new review of the field, published in 2026 by researcher Thi Tuyet Hanh Nguyen, examines how the "lawyer's case" for test validity is holding up against these new challenges. By looking at studies published between 2019 and 2026, Nguyen traces how the field is trying to balance rigorous proof with the urgent need for justice and the mysterious nature of machine scoring.
The review begins by looking at how researchers have been applying the argument-based approach in recent years. Nguyen found that while the method has spread to new types of tests, from immigration exams to assessments for young children, the evidence supporting these tests remains uneven. Most studies focus heavily on the early links in the chain: proving that the test is scored consistently and that the scores are reliable. However, the later, more difficult links are often ignored. Researchers rarely gather strong evidence to show that a test score actually predicts how well a person will perform in a real-world job or that the decisions made based on those scores are truly fair. This pattern has persisted for years; even as the field has grown, the "case" for a test often stops short of proving its ultimate value in the real world.
A significant portion of the review addresses a new theoretical movement known as justice-oriented validity. This approach challenges the idea that a test can be considered valid simply because it is technically accurate. Instead, it asks whose values are being used to define what is "correct." If a test is built by a specific group of people, it might unintentionally favor their cultural background while penalizing others, even if the scoring is mathematically perfect. Nguyen highlights a debate within the field: some researchers worry that focusing too much on social justice might weaken the scientific rigor of testing, while others argue that without addressing these deep-seated inequalities, the tests are fundamentally flawed. The review suggests that fairness should not be an afterthought or a separate check; it must be woven into the very core of the argument for why a test is valid.
The second major development is the rise of artificial intelligence in scoring. Large language models, the same technology behind advanced chatbots, are now grading essays and speaking tests. This creates a unique problem for validation. When a human teacher grades a paper, they can explain why they gave a certain score. When an AI does it, the process is often a "black box," where the reasoning is hidden inside complex computer code. Nguyen points out that current testing frameworks were not designed to handle this opacity. Studies show that while AI can sometimes agree with human graders, it often fails to do so consistently across different languages or for students from different backgrounds. The review warns that relying on AI without demanding transparency and proof of fairness could introduce new, invisible biases into the system.
The paper concludes by proposing seven key considerations for the future of language testing. The first five, identified in an earlier review, remain relevant: researchers must analyze the real-world tasks the test mimics, involve a wide range of people in the process, and use mixed methods to gather evidence. However, Nguyen argues that two new considerations are now essential. First, fairness and justice must be treated as explicit parts of the validity argument, not optional extras. This means testing must prove that scores are fair for every subgroup of people, not just on average. Second, when AI is used to score tests, the evidence must include proof that the algorithm is transparent and free from bias. The review suggests that test developers should build these ethical checks into the design of the test from the very beginning, rather than trying to fix problems after the test is already in use.
Ultimately, the paper does not call for abandoning the argument-based approach. Instead, it argues for expanding the definition of what counts as strong evidence. The "lawyer's case" for a test must now include proof that the system is just and that the machine grading it is trustworthy. As the field moves forward, the goal is to make these considerations as routine and expected as checking for scoring consistency. By doing so, the language testing community hopes to ensure that the gates these tests open are fair for everyone, regardless of their background or the technology used to measure them.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.