💬 NLP

From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment

This paper proposes a framework combining SHAP attributions and LLM-generated rationales to evaluate rubric-based teaching quality scoring, finding that while fine-tuned PLMs achieve higher accuracy, SHAP provides more faithful, coherent, and transferable explanations than LLM rationales.

Original authors: Ivo Bueno, Babette Bühler, Philipp Stark, Tim Fütterer, Ulrich Trautwein, Dorottya Demszky, Heather Hill, Enkelejda Kasneci

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ivo Bueno, Babette Bühler, Philipp Stark, Tim Fütterer, Ulrich Trautwein, Dorottya Demszky, Heather Hill, Enkelejda Kasneci

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a stack of student essays. You have a strict checklist (a "rubric") to decide if the feedback in the essay is good, bad, or somewhere in between. Now, imagine you hire two different types of "AI assistants" to do this grading for you.

This paper is a report card on how well these two AI assistants work, and more importantly, how well they can explain why they gave a certain grade.

Here is the breakdown of the study using simple analogies:

The Two Contenders

The researchers tested two types of AI models to grade classroom conversations (transcripts) based on a specific rubric called "Quality of Feedback."

The "Specialized Intern" (Fine-tuned PLMs): These are standard AI models that have been specifically trained on thousands of examples of this exact grading task. They are like a student who has memorized the textbook and practiced the test a hundred times.
The "Smart Generalist" (Prompted LLMs): These are massive, powerful AI models (like the ones you might chat with online) that were not trained specifically on this task. Instead, the researchers just gave them a set of instructions (a prompt) saying, "Here is the rubric, grade this." They are like a brilliant expert who has never seen this specific test before but is asked to take it on the spot.

The First Test: Who Gets the Grade Right? (Scoring)

The researchers asked: Who gives the most accurate score?

The Result: The Specialized Intern was much more accurate. It consistently gave scores very close to what human experts would give.
The Catch: The Intern had a weird habit of being "safe." It rarely gave the highest (7) or lowest (1) scores. It tended to cluster all its grades in the middle (3, 4, or 5). It was accurate but a bit boring and risk-averse.
The Generalist: The Smart Generalist was less accurate overall. However, it was more "brave." It gave a wider variety of scores, including the very high and very low ones, even if it wasn't always right.

The Takeaway: If you want the most precise number, use the trained model. If you want a model that isn't afraid to give extreme scores (even if it makes more mistakes), use the big chatbot.

The Second Test: Who Tells the Truth? (Explanations)

This is the most important part of the paper. In education, you don't just want a score; you want to know why.

The Specialized Intern used a mathematical tool called SHAP. Think of SHAP as a forensic accountant. It looks at the transcript, sentence by sentence, and calculates exactly how much each sentence changed the final score. It's like a calculator that says, "This sentence added 0.5 points; that sentence subtracted 0.2."
The Smart Generalist just talked. It generated a list of sentences it thought were important, writing them out in natural language. It's like a student raising their hand and saying, "I think this part is important because..."

The researchers tested who was telling the truth by playing a game of "Delete and See."

They took the sentences the AI said were important.
They deleted those sentences from the transcript.
They asked the AI to grade the empty transcript again.

The Logic: If the AI truly understood why it gave a score, deleting the "important" sentences should cause the score to crash or change dramatically. If the AI was just making up a story, deleting those sentences shouldn't change the score much.

The Result: The Specialized Intern (SHAP) was honest. When they deleted the sentences SHAP flagged, the score changed a lot. The AI was clearly relying on those specific sentences to make its decision.
The Result: The Smart Generalist (LLM) was often lying. When they deleted the sentences the LLM said were important, the score barely changed. The LLM was generating a "plausible-sounding" explanation, but those sentences weren't actually the ones driving its decision. It was like a student giving a convincing speech about why they deserve an A, but when you remove the evidence they cited, their grade doesn't actually change.

The Third Test: Can They Swap Notes? (Cross-Model)

The researchers tried to see if the explanations were universal.

They took the "important sentences" identified by the Specialized Intern and deleted them from the Smart Generalist's input. Result: The Generalist's score changed significantly. The Intern's math worked on the Generalist, too.
They took the "important sentences" identified by the Smart Generalist and deleted them from the Specialized Intern's input. Result: The Intern's score barely moved. The Generalist's "opinion" didn't matter to the Intern.

The Takeaway: The mathematical explanation (SHAP) is a universal truth that works across different AI brains. The chatbot's explanation is specific to that one chatbot and doesn't hold up when tested against others.

The Final Verdict

The paper concludes that while big, powerful chatbots (LLMs) are great at generating text that sounds like a good explanation, they are unreliable for explaining why they made a decision in high-stakes situations like grading teachers.

SHAP (The Math): Is like a transparent window. You can see exactly which bricks are holding up the wall. It is trustworthy and consistent.
LLM Rationales (The Chat): Are like a magician's sleight of hand. They look impressive and convincing, but if you look closely (by deleting the "important" parts), you realize they aren't actually holding the weight of the decision.

In short: If you need a trustworthy reason for a grade in a school setting, trust the math (SHAP), not the chatbot's story. The chatbot is great for writing, but it's currently a bad witness for its own decisions.

Technical Summary: From Scoring to Explanations

Problem Statement

Automated scoring models are increasingly deployed to assign rubric-based quality ratings to complex language performances, such as classroom transcripts. However, these systems typically function as "black boxes," providing scalar scores without insight into the reasoning behind them. This lack of transparency is particularly problematic in high-stakes educational contexts where stakeholders must understand, trust, and potentially contest automated judgments. While Large Language Models (LLMs) can generate free-form rationales, recent literature suggests these explanations may be persuasive yet unfaithful to the model's actual decision-making process. The central challenge addressed is determining which parts of a text truly drive a model's rubric-based score and how to evaluate whether an explanation faithfully captures these drivers.

Methodology

The authors propose a unified framework for sentence-level interpretability of rubric-based scoring, instantiated on the Quality of Feedback (QoF) dimension of the Classroom Assessment Scoring System (CLASS) using the NCTE corpus (6,005 annotated elementary mathematics classroom transcript segments).

Experimental Setup

The study compares two families of models:

Fine-tuned Pretrained Language Models (PLMs): Including BERT, ALBERT, RoBERTa, and DeBERTa (base and large variants). These models are fine-tuned for regression to predict a 1–7 QoF score.
Instruction-tuned Large Language Models (LLMs): Including Llama 3.1, Mixtral, Qwen 3, and Mistral variants. These models perform scoring via few-shot prompting and sentence ranking via zero-shot prompting.

Interpretability and Evaluation Protocols

The framework evaluates explanations using three primary mechanisms:

Explanation Generation:
- SHAP (SHapley Additive exPlanations): Applied at the sentence embedding level within a hierarchical PLM architecture to generate model-agnostic feature attributions.
- LLM Rationales: LLMs are prompted to rank the top 10 sentences most influential for a given score.
Faithfulness Analysis (Deletion-based): The system progressively removes the top- $k$ (where $k=10$ ) sentences identified by an explanation method and re-runs the scoring model. The magnitude of the prediction change ( $\Delta$ ) serves as a proxy for faithfulness; larger shifts indicate the explanation successfully identified influential text.
Cross-Model Evaluation: Explanations generated by one model family (e.g., SHAP from a PLM) are used to perturb the input of the other family (e.g., an LLM), and vice versa. This tests whether explanations transfer robustly across architectures or remain model-specific.
Alignment Metrics: Jaccard similarity and Spearman rank correlation are computed to measure the overlap in sentence selection between SHAP and LLMs.

Key Contributions

General Framework: A principled framework for sentence-level interpretability that combines model-agnostic Shapley value attributions with LLM-generated rationales, specifically tailored for rubric-based scoring.
Comparative Analysis: A systematic comparison of specialized fine-tuned PLMs versus prompted LLMs on both scoring performance and explanation faithfulness.
Faithfulness Evaluation: An empirical evaluation of explanation methods using deletion-based tests, revealing that SHAP identifies sentences that reliably drive predictions, whereas LLM rationales often induce smaller, less coherent shifts.
Cross-Model Consistency: A novel protocol demonstrating that SHAP attributions transfer robustly across architectures, while LLM rationales exert limited and inconsistent influence on alternative models.
Educational Implications: A discussion on the design of transparent feedback tools, cautioning against the use of unvalidated LLM rationales as authoritative justifications in high-stakes settings.

Results

Scoring Performance (RQ1)

PLMs: Fine-tuned PLMs significantly outperform LLMs in prediction accuracy. The best PLM (DeBERTaV3 large) achieves a Mean Absolute Error (MAE) of 0.96, comparable to a constant baseline. However, fine-tuned PLMs exhibit label compression, rarely predicting scores at the extremes (1 or 7) and collapsing toward the mid-range (3–5).
LLMs: Prompted LLMs show higher error rates (best MAE: 1.02 for Mistral Small) but produce predictions spanning the full 1–7 scale, preserving score variability better than PLMs.

Explanation Faithfulness (RQ2)

SHAP vs. LLMs: Deletion-based tests show that sentences identified by SHAP produce larger and more coherent prediction shifts than those selected by LLMs.
Alignment: The overlap between SHAP and LLM sentence selections is low (mean Jaccard similarity: 0.085; mean Spearman correlation: 0.062).
Stability: LLMs exhibit limited controllability, often failing to return the exact number of requested sentences, and their rationales induce smaller, often unstable prediction changes even when applied to the models that generated them.

Cross-Model Transfer (RQ3)

PLM to LLM: Removing sentences identified by SHAP (from a PLM) causes significant, immediate shifts in LLM predictions, suggesting SHAP identifies features relevant across architectures.
LLM to PLM: Removing sentences identified by LLMs produces substantially smaller prediction shifts in PLMs compared to SHAP-based removal. LLM rationales often result in non-monotonic trajectories, indicating they do not reliably capture the features driving PLM predictions.

Significance and Claims

The paper concludes that SHAP provides more faithful and transferable explanations for rubric-based scoring compared to LLM-generated rationales. While LLMs offer out-of-the-box usability and broader score distributions, their explanations are often unfaithful to the underlying computation and do not generalize well across model architectures.

The authors assert that the proposed framework offers a principled basis for evaluating scoring models and their explanations in high-stakes educational settings. They emphasize that in contexts like teacher feedback, where trust and contestability are vital, reliance on unvalidated LLM rationales is risky. Instead, model-agnostic attribution methods like SHAP provide a more stable foundation for transparent and actionable automated assessment. The work does not claim LLMs are useless for scoring but highlights that their current rationales should not be treated as authoritative justifications for automated judgments without rigorous faithfulness evaluation.