Toward LLM-Supported Automated Assessment of Critical Thinking Subskills

1. Problem Statement

Critical thinking (CT) is a defining 21st-century skill, yet it remains under-researched in Educational Data Mining (EDM). Existing challenges include:

• Operationalization: CT is often defined vaguely, making it difficult to measure quantitatively.
• Assessment Limitations: Current assessments (e.g., multiple-choice) lack ecological validity, while open-ended assessments (e.g., essays) are time-consuming to grade manually.
• Scalability: There is a lack of automated tools capable of providing timely, fine-grained feedback on CT subskills within authentic learning contexts.

The paper addresses the feasibility of using Large Language Models (LLMs) to automatically assess specific critical thinking subskills (e.g., evaluating evidence, using counterarguments) from student argumentative essays, moving beyond holistic scoring to granular, rubric-based evaluation.

2. Methodology

2.1 Dataset and Rubric Development

• Data Source: The study utilized a subset of the PERSUADE 2.0 corpus, consisting of 500 argumentative essays written by US students (grades 6–12).
• Rubric Construction: Based on the Critical Thinking Progression for High School Students (ETS/Carnegie), the authors developed a coding rubric for three core skills and six subskills:
  1. Information Analysis: Synthesizing Multiple Sources, Evaluating Evidence Strength.
  2. Argument Generation: Using Counterarguments, Using Facts and Opinions.
  3. Logical Reasoning: Drawing Conclusions, Using Logical Fallacies.
• Proficiency Levels: The rubric uses a 5-point ordinal scale: Not Applicable, Below Emerging, Emerging, Expanding, Exemplifying.
• Human Coding: Two groups of graduate student coders (total 7 coders) scored the 500 essays. Inter-rater reliability was established using Krippendorff's Alpha (threshold $\ge$ 0.60) and Weighted Kappa.

2.2 Experimental Design: LLM Approaches

The authors evaluated three distinct automated scoring approaches across three models:

1. Proprietary Model (Prompting): GPT-5 tested in Zero-shot and Few-shot (3 in-context examples) modes.
2. Open-Source Models (Fine-tuning):
   • Llama 3.1 8B Instruct: A decoder-only model. Fine-tuned using PEFT (LoRA).
   • ModernBERT-base: An encoder-only transformer (149M parameters). Fine-tuned with a classification head.

Fine-tuning Variants:

• Input: Subskill name, definition, rubric descriptors, and essay text.
• Output: Proficiency label (text or ID).
• Justification: Some variants included a Chain-of-Thought (CoT) step where the model generated a justification before the label (synthesized using GPT-4).
• Rubric Presence: Experiments compared inputs with full rubric descriptors vs. without (name/definition only).

2.3 Evaluation Metrics

Performance was measured against human annotations using:

• Accuracy: Exact match rate.
• RMSE: Root Mean Squared Error (penalizing distance from true ordinal level).
• F1 Scores: Macro (unweighted) and Weighted (accounting for class imbalance).
• Krippendorff's $\alpha$: Agreement metric treating the LLM as a second human annotator (crucial for ordinal data).

2.4 Data Splits

• Essay-based Split: 70/10/20 train/val/test split (standard evaluation).
• Subskill-based Split: One subskill held out entirely for testing to evaluate generalization to unseen subskills.

3. Key Results

3.1 Overall Performance

• Fine-tuning > Prompting: Fine-tuned Llama 3.1 8B achieved the best results, outperforming both GPT-5 (few-shot) and ModernBERT.
• Human-Level Agreement: The best Llama variant (With Rubric, Without Justification) achieved a Krippendorff's $\alpha$ of 0.606, crossing the threshold set for sufficient human inter-rater reliability.
• Model Comparison:
  • Llama 8B: $\alpha \approx 0.606$ (Best).
  • ModernBERT: $\alpha \approx 0.541$ (Underperformed Llama by ~6.5%).
  • GPT-5 (Few-shot): $\alpha \approx 0.542$.
  • GPT-5 (Zero-shot): $\alpha \approx 0.487$.

3.2 Subskill-Specific Performance

Performance varied significantly based on label distribution and task complexity:

• High Performance: Synthesizing Multiple Sources ($\alpha = 0.74$). The model excelled here because the data was highly skewed (mostly "Not Applicable"), making the majority class easy to predict.
• Low Performance: Evaluating Evidence Strength ($\alpha = 0.23$) and Using Counterarguments ($\alpha = 0.31$). These tasks suffered from class imbalance (dominated by "Emerging") and subtle distinctions between proficiency levels, causing the model to default to the majority class.
• Logical Fallacies: Accuracy was low (~50%) due to the difficulty of distinguishing subtle fallacies from valid evidence.

3.3 Generalization (Unseen Subskills)

• When tested on a subskill not seen during training (Subskill-based split), performance dropped significantly ($\alpha \approx 0.42$).
• Role of Rubrics: Rubric descriptors were essential for generalization. Models without rubric descriptors in the prompt achieved near-zero agreement ($\alpha \approx 0$).
• Role of Justifications: Unlike the essay-based split, adding generated justifications improved generalization performance by 6% in $\alpha$, suggesting CoT helps the model map unseen tasks to the rubric logic.

3.4 Error Analysis

• Most errors occurred between adjacent proficiency levels (e.g., predicting "Emerging" instead of "Expanding").
• The models struggled to detect subtle differences in the balance between evidence and logical fallacies, often misinterpreting tone or volume of citations as evidence of reasoning quality.

4. Key Contributions

1. Operationalization of CT: Successfully defined and coded a complex, multi-dimensional construct (Critical Thinking) into six measurable subskills for an EDM context.
2. Benchmarking LLMs: Provided a comprehensive comparison of prompting vs. fine-tuning, and proprietary vs. open-source models, for fine-grained educational assessment.
3. Methodological Insight: Demonstrated that fine-tuning open-source models on task-specific data is superior to prompting for achieving human-level reliability, while highlighting the critical need for rubric descriptors when generalizing to new skills.
4. Data & Code: Established a human-coded dataset of 500 essays with multi-dimensional CT labels, serving as a resource for future research.

5. Significance and Future Directions

• Scalability: The study proves that automated, fine-grained assessment of critical thinking is feasible, offering a path toward scalable formative feedback in authentic learning environments.
• Cost and Privacy: Fine-tuning open-source models offers a cost-effective and privacy-preserving alternative to expensive API calls (e.g., GPT-5) for large-scale educational deployment.
• Limitations & Future Work:
  • Class Imbalance: Future work must address data imbalance (e.g., via synthetic oversampling) to improve performance on rare proficiency levels.
  • Generalization: While promising, models still struggle with unseen subskills without specific labeled data. Future research should explore meta-prompting or transfer learning to reduce the need for subskill-specific human labeling.
  • Hybrid Approaches: Given the difficulty of detecting subtle fallacies, a hybrid human-AI workflow is recommended for high-stakes assessment.

In conclusion, this paper lays the groundwork for integrating AI into the assessment of higher-order thinking skills, demonstrating that with careful rubric design and fine-tuning, LLMs can effectively support the measurement of critical thinking subskills in student writing.