📊 statistics

When Better Writing Changes the Order: A Monte Carlo Study of Rank Reversals and Selection Instability in AI-Assisted L2 Writing Assessment

This Monte Carlo study demonstrates that while AI-assisted writing improvements do not automatically preserve the relative proficiency rankings of learners, the resulting rank reversals and selection instability vary significantly depending on the assistance mechanism, with weakness-targeted gains causing substantial ordering errors and selection accuracy declines that cannot be fully corrected by product-only score calibration.

Original authors: Tan Nguyen Huynh Si

Published 2026-10-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tan Nguyen Huynh Si

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of language learning, writing is often judged by the final product: a polished essay that demonstrates a student's command of grammar, vocabulary, and organization. For decades, educators have used these scores to rank students, deciding who gets into a prestigious university or who receives a scholarship. The assumption has been that if a student's writing improves, their rank should reflect that improvement. However, a new layer of complexity has arrived with artificial intelligence. Today, students can use AI tools to revise their drafts, fixing errors and refining sentences. The critical question for educators and policymakers is not just whether the final essay looks better, but whether the AI has changed the relative standing of the students in a fair way. If a student with weaker skills uses a powerful AI tool to jump ahead of a naturally talented peer, the ranking system may no longer be measuring the students' own abilities, but rather their access to or skill with the technology. This creates a tension between the quality of the text and the validity of the score used to make high-stakes decisions.

To untangle this problem, a researcher named Nguyen Huynh Si Tan conducted a massive computer simulation to see how different types of AI assistance affect student rankings. Instead of testing real students, the study generated one thousand groups of one thousand virtual learners. Each virtual student was assigned a hidden, true level of writing ability across four areas: grammar, vocabulary, organization, and mechanics. The researcher then simulated two scenarios for each student: a first draft written without help, and a second draft revised with the help of an AI tool. The AI tools in the simulation were programmed to behave in specific, controlled ways. Some gave every student the same amount of help, while others targeted specific groups, such as giving the most help to the weakest writers or the strongest writers. The goal was to see if these different ways of helping would scramble the order of the students, causing a less capable writer to appear more capable than a truly skilled one.

The study found that the answer depends entirely on how the assistance is delivered. When the AI gave a uniform boost to everyone—improving every essay by the same amount or compressing the scores without changing the order—the rankings remained perfectly stable. The students who were best at writing without help were still the best after the help. However, when the AI targeted the weaker writers with extra help, the situation changed dramatically. In this scenario, the AI helped the struggling students improve so much that they overtook their peers. While the essays themselves were better, the ranking system began to fail its primary job. The simulation showed that in this "weakness-targeted" condition, about 16.6% of the student pairs were now ordered incorrectly compared to their true ability. This means that nearly one in six comparisons between two students resulted in the wrong person being ranked higher. In contrast, when the AI targeted the stronger writers, the rankings stayed surprisingly accurate, even though many individual pairs swapped places. The net effect was that the system remained fair because the errors canceled each other out. Furthermore, under conditions of high measurement noise, the study found that preferential gains for stronger writers could actually improve comparative accuracy, challenging the idea that heterogeneous gains always worsen rankings.

The consequences of these ranking shifts became clear when the researcher looked at how students would be selected for top spots, such as the top 10% of a class. In the scenario where the AI helped the weakest writers the most, the accuracy of selecting the truly best writers dropped sharply. Before the AI help, the system correctly identified about 74.5% of the top students. After the targeted help, that accuracy fell to just 50.5%. This means that in a real-world setting, half of the students selected for a top prize would not have been the top performers if they had written without help. The study also explored whether simple mathematical adjustments could fix this problem. The results showed that you cannot simply re-scale the scores to make them fair again; if the order is wrong, a straight line adjustment cannot fix it. Only a model that knew exactly how much help each student received could recover the true ranking, but such information is rarely available in a real classroom.

Ultimately, the research demonstrates that an improvement in writing quality does not automatically guarantee a fair ranking. The study indicates that the method of assistance matters more than the improvement itself. If an AI tool helps weaker students disproportionately, it can distort the competitive landscape, making it difficult to distinguish between those who have truly mastered the skill and those who have simply received the most help. The findings serve as a warning that as schools adopt AI tools, they must carefully consider how these tools distribute their benefits. Without understanding the specific mechanics of the assistance, educators risk using scores that no longer reflect the independent abilities of their students, potentially making unfair decisions about who gets the best opportunities. The simulation provides a clear, mathematical proof that the path to a better essay is not always the same as the path to a fair ranking, though these conclusions concern conditional mathematical and simulated assessment properties rather than estimates of real learners or AI products.

Technical Summary: When Better Writing Changes the Order

Problem Statement
The paper addresses a critical gap in the validity of AI-assisted L2 writing assessment: the assumption that improvements in the quality of a final written product necessarily preserve the relative ordering of writers based on their independent proficiency. While existing research focuses on revision practices and score gains, this study isolates the specific measurement question of whether AI assistance maintains, distorts, or improves the comparative ranking of learners. The core problem is distinguishing between observable rank reversals (changes in the order of scores) and newly introduced ranking errors (changes that move a writer away from their true latent proficiency order). The study challenges the notion that score improvement is synonymous with valid comparative judgment, particularly in high-stakes selection contexts where errors near a cutoff have disproportionate consequences.

Methodology
The study employs an ADEMP-aligned (Aims, Data generation, Estimands, Methods, Performance) synthetic Monte Carlo design. It does not use real learner data but generates a known "oracle" of latent independent proficiency to serve as a ground truth.

Data Generation: 1,000 independent synthetic cohorts were generated, each containing 1,000 learners. Each learner possesses four correlated latent proficiency domains (grammar, lexical resource, organization, mechanics). A global latent target ( $\theta$ ) is defined as the unweighted mean of these domains.
Scoring Model:
- Unaided Score ( $U$ ): Derived from latent proficiency plus independent Gaussian noise ( $SD=3.5$ ).
- Assisted Score ( $A$ ): Calculated as $U + G + \xi$ , where $G$ is a condition-specific gain and $\xi$ is additional noise.
Experimental Conditions: Ten primary scenarios were tested, including:
- Controls: Zero gain, uniform additive gain, and monotone score compression (to test if variance reduction implies rank loss).
- Heterogeneous Gains: Weakness-targeted (deterministic and stochastic), strength-targeted, random positive gains, and nonlinear targeting.
- Negative Controls: Added measurement noise and equal-variance permutation (randomizing scores while preserving distribution).
Estimands:
- Pairwise Analysis: 25,000 unordered writer pairs per cohort were sampled to calculate "newly incorrect" pairs (where the assisted order contradicts the latent order but the unaided order did not) and "resolved" pairs (where assistance corrected a prior unaided error).
- Selection Analysis: Top-k selection (5%, 10%, 20%, 30%) was evaluated against the latent oracle to measure selection accuracy (overlap) and membership churn (newly excluded vs. rescued candidates).
- Recovery: Various calibration models (linear, isotonic, metadata-aware regression) were tested to see if they could recover the latent order from assisted scores.
Validation: The study includes a 500-cohort independent confirmation, 160-cohort recovery benchmarks, 120-cohort robustness checks (varying noise, correlation, and score bounds), and an exact enumeration audit of the pair-sampling estimator to ensure computational accuracy.

Key Contributions

Conceptual Distinction: The paper establishes a framework separating rank reversals (observed order changes) from ranking validity (agreement with latent proficiency). It demonstrates that a high rate of reversals does not automatically equate to a loss of validity if those reversals correct prior errors.
Mechanism-Specific Effects: It proves that the impact of AI assistance on ranking is contingent on the mechanism of gain distribution, not just the magnitude of improvement.
- Uniform gains and monotone compression preserve all strict rankings.
- Heterogeneous weakness-targeted gains (giving larger help to lower-proficiency writers) introduce significant net ranking errors.
- Strength-targeted gains can produce frequent reversals but result in near-zero net error, as they often correct prior noise-induced errors.
Selection vs. Correlation: The study shows that rank correlation metrics (e.g., Kendall's $\tau$ ) can mask critical selection errors. A mechanism might preserve overall correlation while drastically altering who is selected at a specific cutoff (e.g., top 10%).
Limits of Calibration: It mathematically demonstrates that strictly monotonic score transformations (linear or isotonic calibration) cannot reverse strict rank inversions. Recovery of the latent order requires additional information (metadata) beyond the score itself.

Results

Ranking Instability: Under heterogeneous weakness-targeted conditions, the fraction of newly incorrect pairs averaged 16.6%, while only 2.6% of errors were resolved, leading to a net deterioration in ranking accuracy. Conversely, strength-targeted gains produced frequent reversals but a net error near zero.
Selection Accuracy: In the top-decile selection scenario, heterogeneous weakness targeting reduced oracle overlap from a baseline of 74.5% to 50.5%. In contrast, strength-targeted gains maintained baseline accuracy (~74.2%) despite frequent pairwise reversals.
Robustness: The negative impact of weakness-targeted gains persisted across various score distributions, noise levels, and correlation structures, though the magnitude varied. Notably, under high-noise conditions, strength-targeted gains actually improved ranking accuracy (negative net error).
Recovery: Score-only calibration (linear or isotonic) failed to recover the latent order. However, a metadata-aware regression model (using synthetic gain drivers) partially recovered the order, demonstrating that the loss of ranking information is recoverable if the mechanism of assistance is known and observable.
Tie Sensitivity: In bounded/rounded score scenarios, the sign of net error for near-zero effects (like strength targeting) was sensitive to tie-handling conventions, whereas the deterioration from weakness targeting remained robust.

Significance and Claims
The paper claims to provide a "reproducible demonstration" that score improvement and relative proficiency ranking are distinct assessment properties. Its primary significance is methodological:

It argues that validity arguments for AI-assisted writing must explicitly define the target construct (e.g., independent proficiency vs. collaborative product quality).
It asserts that "better writing" (higher scores) does not guarantee "better ranking" (accurate selection).
It warns against assuming that AI assistance is inherently neutral or inherently damaging; the outcome depends entirely on the structure of the gain (e.g., whether it targets specific proficiency levels) and the noise in the baseline measurement.
The study concludes that its findings are conditional mathematical properties of the simulated system. It does not claim to estimate the prevalence of these effects in real classrooms but provides a framework for evaluating them. The author emphasizes that real-world application requires independent proficiency anchors, paired unaided/assisted tasks, and explicit validity arguments regarding the intended use of the scores.

Technical Summary: When Better Writing Changes the Order

More like this