Technical Summary: REC-CBM

Problem Statement

Open-ended grading is essential for equitable and personalized education but remains labor-intensive and costly, necessitating automated solutions. While recent neural and Large Language Model (LLM) based systems have achieved high predictive accuracy, they function as "black boxes." Their scoring processes and rationales are opaque, making it difficult for educators to verify alignment with rubrics, audit specific decisions, or intervene when judgments are questionable. This lack of transparency undermines trust in high-stakes educational contexts.

Concept Bottleneck Models (CBMs) offer a potential middle ground by routing predictions through human-interpretable concepts, providing mechanistic transparency. However, standard CBMs are ill-suited for open-ended grading due to three specific limitations:

1. Shared Latent Space: They assume all concepts reside in a shared space, failing to capture fine-grained rubric dimensions that attend to different aspects of a response.
2. Ordinal Semantics: They treat concept scores as unordered categories, discarding the natural ranking information inherent in rubric annotations.
3. Annotation Noise: They assume concept labels are reliable, ignoring the inherent subjectivity, annotator disagreement, and ambiguity found in real-world human grading.

Methodology: REC-CBM

The authors propose REC-CBM (Rubric-Aware Error-Correction Concept Bottleneck Model), a framework designed to address these limitations through three core components:

1. Rubric-Aware Concept Encoder

Instead of a shared representation, REC-CBM employs a concept query bank ($Q = \{q_k\}$) where each learnable vector $q_k$ acts as a dimension-specific retrieval query.

• Mechanism: Using soft attention, the model aggregates token representations from the input text specifically relevant to each rubric dimension $k$.
• Output: This produces distinct concept-specific representations ($h_k$) that are then passed through linear classifiers to predict ordinal distributions ($\hat{p}_k$) for each concept.
• Loss: Standard cross-entropy loss ($\mathcal{L}_{con}$) is used to supervise concept prediction.

2. Ordinal Pairwise Concept Calibration

To preserve the ranking structure of rubric levels, the model introduces an ordinal pairwise calibration objective.

• Mechanism: Inspired by the Bradley-Terry model, the system computes expected ordinal scores ($\hat{c}_k$) from the predicted distributions. It then constructs within-batch pairs of responses where one is annotated with a higher concept score than the other.
• Loss: A logistic ranking loss ($\mathcal{L}_{rank}$) is minimized to ensure the predicted scores respect the observed ordering ($\hat{c}_i > \hat{c}_j$). This captures the semantics of the scoring scale better than treating levels as unordered categories.

3. Latent Concept Error Correction

Recognizing that concept annotations are noisy, REC-CBM incorporates a latent error-correction module before the final grade prediction.

• Model: The calibrated concept scores are treated as noisy observations ($\tilde{s}$) of an underlying latent concept vector ($z$). A multivariate Gaussian prior is placed on $z$ with a learnable precision matrix ($\Omega$).
• Correction: The model computes a closed-form posterior mean ($\mu_{post}$) which serves as the denoised concept vector. This process shrinks concepts with high estimated noise and leverages correlations between rubric dimensions (via off-diagonal entries in $\Omega$) to share statistical strength.
• Training: This stage minimizes a task loss ($\mathcal{L}_{task}$), a denoising alignment loss ($\mathcal{L}_{den}$) to keep corrected concepts close to normalized annotations, and a sparsity penalty ($\mathcal{L}_{spa}$) to prevent overfitting dependencies.

Training Paradigm

REC-CBM utilizes a two-stage training process to prevent the latent head from distorting the interpretable bottleneck:

• Stage I: Trains the text encoder, concept query bank, and concept classifiers using $\mathcal{L}_{con}$ and $\mathcal{L}_{rank}$. These components are then frozen.
• Stage II: Trains the latent correction parameters ($\Omega$, noise variances) and the task classifier ($W$) using $\mathcal{L}_{task}$, $\mathcal{L}_{den}$, and $\mathcal{L}_{spa}$ on top of the fixed concept outputs.

Key Contributions

1. Problem Formulation: The paper frames trustworthy open-ended grading through transparent decision-making, grounding the process in human-interpretable concepts to enable educator inspection and intervention.
2. Data Resources: The authors curated and annotated three public open-ended grading benchmarks (Mohler, ASAP 2.0, MOCHA) with rubric-aligned concept labels using a human-in-the-loop pipeline, creating a resource for interpretable educational assessment.
3. Methodological Innovation: REC-CBM integrates rubric-aware encoding, ordinal pairwise calibration, and latent error correction to address the specific shortcomings of standard CBMs in grading contexts.
4. Empirical Validation: Extensive experiments demonstrate that REC-CBM outperforms both state-of-the-art black-box models and transparent baselines in grading accuracy while providing more faithful concept-level reasoning.

Experimental Results

The authors evaluated REC-CBM on three datasets (Mohler, ASAP 2.0, MOCHA) using five backbone architectures (BERT, BART, GPT-2, RoBERTa, T5-Base).

• Performance: REC-CBM consistently achieved the highest Task Accuracy and Task F1 scores across all backbones and datasets. It matched or exceeded the performance of strong black-box baselines (e.g., fine-tuned RoBERTa) while maintaining interpretability.
• Concept Reliability: REC-CBM improved concept-level F1 scores significantly compared to Vanilla CBMs and other transparent baselines (C3M, CT-CBM, CB-LLM).
• Ablation Studies: Removing any single component (Rubric-Aware Encoder, Ordinal Calibration, or Latent Error Correction) degraded performance, confirming that the modules provide complementary rather than redundant signals. The Rubric-Aware Encoder was identified as the most critical component.
• Human Intervention: Simulated interventions showed that modifying concept predictions (e.g., correcting a "wrong" label to an "oracle" label) predictably shifted the final grade, validating the model's actionability. Conversely, corrupting evidence degraded performance, demonstrating the bottleneck's causal role.
• Latent Denoising: Analysis of the learned precision matrix revealed that the model successfully transformed dense, noisy empirical correlations between rubric dimensions into a sparse, interpretable dependency structure, effectively decoupling correlated concepts.

Significance and Claims

The paper claims that REC-CBM provides a practical, interpretable grading solution that advances trustworthy education. By enabling educators to inspect, verify, and intervene in the reasoning process, the model addresses the "black-box" limitations of current AI grading systems.

The authors emphasize that their work does not merely improve accuracy but ensures that the grading process is mechanistically transparent. The framework allows for:

• Inspection: Educators can view which rubric dimensions and specific token spans contributed to a grade.
• Intervention: Educators can override specific concept scores to observe how the final grade changes, facilitating a feedback loop.
• Trust: The explicit handling of annotation noise and ordinal semantics ensures that the explanations are robust and aligned with the intended rubric structure.

The work concludes that REC-CBM is a viable path toward automated grading systems that are both high-performing and aligned with the pedagogical needs of transparency and accountability.