💬 NLP

Counterargument for Critical Thinking as Judged by AI and Humans

This intervention study demonstrates that students effectively employ critical thinking by writing counterarguments to AI-generated content and confirms that frontier large language models, when guided by clear rubrics, can reliably assess such written work at scale with moderate alignment to human evaluations.

Original authors: Tosin Adewumi, Marcus Liwicki, Foteini Simistira Liwicki, Lama Alkhaled, Hamam Mokayed, Esra Sümer-Arpak

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tosin Adewumi, Marcus Liwicki, Foteini Simistira Liwicki, Lama Alkhaled, Hamam Mokayed, Esra Sümer-Arpak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a classroom where the teacher hands out a challenge: "Here is a strong opinion written by a super-smart robot. Now, you must write a counter-argument to it." This is exactly what researchers at Luleå University of Technology did with 36 master's students. They wanted to see two things:

Does fighting back against an AI's argument actually make students think harder and better?
Can AI itself be a good referee to grade these student essays, or do we still need human teachers?

Here is the breakdown of their study, explained simply.

The Setup: The "Robot vs. Human" Debate

The researchers picked four hot-button topics (like "Do babies know how to speak before they are born?" or "Is statistics just about numbers?").

Step 1: Students asked an AI to write an argument for one of these topics.
Step 2: The students had to put down their phones and write a counter-argument (a rebuttal) against the AI's text. They had to do this without help, using their own brains, logic, and references.
Step 3: The essays were graded by three different "judges":
- The Human Teacher: An experienced expert.
- The Student Peers: Two other students in the class.
- The AI Judges: Six different, cutting-edge AI models (like ChatGPT and Gemini) were programmed to act as strict teachers.

The Big Question: Did the Students Think?

The Claim: Yes.
The researchers found that when students wrote these counter-arguments, they didn't just copy-paste or ramble. They actually used logic.

The Metaphor: Think of critical thinking as a muscle. If you just let the AI do the work (cognitive offloading), your muscle gets weak. But when you have to punch back at the AI's argument, you have to flex that muscle. The study found that the students' writing showed they were actively analyzing, comparing, and using logic—the exact ingredients of critical thinking.

The Big Question: Can AI Grade Like a Human?

The Claim: Yes, surprisingly well.
Usually, we worry that AI might be too lenient or too harsh. But in this study, the AI judges and the human judges were on the same page.

The Metaphor: Imagine a panel of judges at a talent show. You have the "Expert Judge" (the teacher), the "Audience Judges" (the students), and the "Robot Judges" (the AI). The researchers found that the Robot Judges were giving scores very similar to the Expert and the Audience.
The Score: They used a statistical tool (Gwet's AC2) to measure how much the judges agreed. Most of the AI models agreed with the humans about 33% of the time (which is considered a decent baseline for this type of complex grading), and in some areas, the agreement was even stronger.
The Catch: The AI needed very clear instructions (a "rubric"). If you tell the AI exactly what "Logic" or "Style" means, it can do the job. If you just say "Grade this," it might get confused.

What They Found in the Details

Logic is King: The most important part of the essay was "Logic." The AI and humans agreed that the students did a good job here.
The "Hallucination" Check: The researchers were worried that students might cheat and let the AI write the counter-argument for them. They used "AI detectors" (like a metal detector for essays), but found those detectors were unreliable (they were like a metal detector that beeps at a belt buckle and a gold ring equally). Instead, the human teacher used their gut feeling and experience to spot the one student who cheated.
The Rubric Matters: The study showed that if you give the AI a clear checklist (Focus, Logic, Content, Style, Correctness, References), it can score essays almost as well as a human.

The Bottom Line

This study is like a test drive for the future of education.

For Students: Writing a counter-argument against AI is a great workout for your brain. It forces you to engage deeply rather than just passively reading.
For Teachers: You don't necessarily need to throw away your grading pen. You can use AI as a "co-pilot" to grade essays, provided you give it very clear rules. The AI and the human teacher will likely give you the same result.

What the paper does NOT say:

It does not claim AI is perfect or that it can replace teachers entirely.
It does not say this works for every subject or every type of essay (only counter-arguments based on specific rubrics).
It does not suggest using this for high-stakes medical or legal decisions.

In short: AI can be a good grader if you give it a clear rulebook, and fighting back against AI arguments is a great way to keep your critical thinking skills sharp.

Technical Summary: Counterargument for Critical Thinking as Judged by AI and Humans

Problem Statement
The rapid integration of Generative AI (GenAI) in education presents a dual challenge: while it offers benefits for engagement and planning, it risks "cognitive offloading," where students shift cognitive tasks to AI, potentially leading to cognitive atrophy and reduced critical thinking skills. Existing literature suggests that reliance on tools like ChatGPT for essay writing can diminish brain activity in areas linked to cognitive processing. Furthermore, objectively assessing critical thinking in open-ended formats, such as written counterarguments, is difficult due to subjective interpretations of critical thinking components. There is a specific gap in understanding how student-written counterarguments to AI-generated content promote critical thinking and whether GenAI systems can reliably assess such work compared to human raters.

Methodology
The study employed a mixed-method intervention design involving 36 Master's students in a Text Mining course at Luleå University of Technology (one submission was disqualified for irregularity, leaving $n=35$ ).

Task: Students selected one of four thesis statements from popular debates (spanning statistics, linguistics, genetics, and pedagogy). They first prompted an LLM to generate an argument for the thesis. Subsequently, without using GenAI, they wrote a minimum 300-word counterargument to the AI-generated content.
Assessment Protocol: Each counterargument was evaluated by three human raters (two student peer-reviews and one experienced teacher) and six frontier Large Language Models (LLMs).
Rubrics: Assessments utilized six established rubrics: Focus, Logic, Content, Style, Correctness, and References. Ratings were given on a 5-point Likert scale.
AI Models: The study evaluated six models: ChatGPT 5.2 (Flagship and Instant), ChatGPT 5.1, Gemini 3 Thinking, DeepSeek V3.1, and LLaMA 4-Maverick 17B. The models were prompted with a specific persona (expert educator) and clear instructions to score the counterarguments based on the rubrics.
Analysis: The study used Spearman's correlation analysis to examine relationships between rubrics and Gwet's AC2 to measure inter-rater reliability between AI models and human raters. AI-text detection tools (Grammarly and ZeroGPT) were initially used but found unreliable; expert judgment was used to identify and exclude the one AI-generated submission.

Key Contributions

Critical Thinking Evidence: The study demonstrates that students' self-written counterarguments to AI-generated content contain logic, a fundamental component of critical thinking, alongside other key elements.
AI as an Assessor: The research shows that GenAI can be successfully used at scale to assess student written work (specifically counterarguments) when guided by clear rubrics. These AI assessments generally align with human assessments, including expert teacher reviews and student peer reviews.

Results

Assessment Alignment: Visual analysis (diverging stacked bar charts, box plots) indicated that most counterarguments received high scores (Agree or Strongly Agree) across all assessment types. The "Logic" rubric showed the expert teacher was slightly stricter than both average student and average AI raters.
Reliability: Gwet's AC2 inter-rater reliability values for the AI models were generally 0.33, except for LLaMA (0.17). These values were lower than the agreement between students and the expert but indicated a baseline level of consistency.
Correlations: Spearman's correlation analysis revealed very strong positive monotonic correlations between "Logic" and "Focus" (0.849 for humans, 0.882 for LLMs) and between "Logic" and "Correctness" (0.919 for humans, 0.893 for LLMs).
Model Performance: ChatGPT 5.1 deviated most from other models on key rubrics, often scoring lower (average of 3). The ChatGPT 5.2 models showed the largest average difference between the "Logic" and "Correctness" rubrics.
Qualitative Feedback: AI models provided reasoning for their scores, often noting word count for "Content" and checking for publication years or links for "References," while acknowledging that deeper qualitative judgment often requires human reading.

Significance and Claims
The paper claims that argument-based learning involving counterarguments to AI content is a viable pedagogical strategy to encourage critical thinking, as evidenced by the logical construction of student responses. It posits that GenAI can serve as a scalable, cost-efficient alternative for assessing such work, provided that clear rubrics are used. The authors emphasize that while AI can approximate human judgment in structured evaluation tasks, the validity of such assessments depends on the clarity of the criteria and the specific design of the rubric to distinguish between general writing proficiency and critical thinking constructs. The study concludes that GenAI should be used to support, rather than replace, core thinking processes, and calls for continued research into the interaction between GenAI and cognitive skill development across diverse learning paradigms.