💻 computer science

Perceive AI: A Dual-Head Parametric Prototype Architecture with Adversarial Consistency Guardrails for Automated Short Answer Assessment

This paper introduces Perceive AI, a novel end-to-end framework for automated short answer grading that leverages a dual-head parametric prototype architecture and adversarial consistency guardrails to achieve robust, prompt-aware scoring while effectively filtering off-topic submissions.

Original authors: MUHAMMAD RIYAN SARWAR

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: MUHAMMAD RIYAN SARWAR

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the landscape of modern education, the short answer question remains a gold standard for measuring true understanding. Unlike multiple-choice tests, which can often be navigated by guessing or pattern recognition, a short answer requires a student to synthesize ideas, construct an argument, and articulate a thought in their own words. For decades, the bottleneck in this process has been human labor. Grading thousands of these open-ended responses is slow, expensive, and prone to inconsistency; two different teachers might assign different scores to the same essay based on their own subjective interpretations. To solve this, researchers have turned to artificial intelligence, specifically systems designed to read and grade text automatically. However, early attempts at this technology often stumbled over a critical flaw: they were too easily fooled. A computer program might award a high grade to a beautifully written paragraph that was completely irrelevant to the question asked, simply because the words were complex and the sentence structure was perfect. This is known as semantic drift, where the system focuses on the style of the writing rather than the substance of the answer.

The challenge, then, is to build an automated grader that does not just read words, but understands the specific relationship between a question and an answer. A new framework called Perceive AI, developed by researchers at the University of Sargodha, attempts to solve this by creating a system that acts more like a vigilant teacher than a simple text matcher. The researchers recognized that standard grading models often fail because they treat every question as if it were the same, or they rely on external databases that are too slow to be practical. Instead, they designed a unified system that learns the unique "shape" of every single question in a curriculum while simultaneously checking if the student's answer actually belongs there. The core of their work is a dual-layered approach: one part of the system evaluates how well the answer fits the grading criteria, while a second, independent part acts as a strict gatekeeper, verifying that the answer is even relevant to the prompt before a grade is ever assigned.

The researchers tested this system on a massive dataset containing nearly 33,000 student responses across 644 different questions, covering a wide range of topics and difficulty levels. The results show that this new architecture significantly outperforms previous methods, particularly in its ability to catch students who try to "game" the system. In traditional models, if a student submitted a lecture transcript or a paragraph about a completely different topic, the AI might still assign a passing grade because the text looked sophisticated. Perceive AI, however, successfully identified these off-topic submissions and rejected them with a 99.10% accuracy rate, assigning them a failing grade as intended. This is achieved through a mechanism that dynamically generates "negative" examples during training, essentially teaching the model to recognize what a wrong answer looks like by mixing up questions and answers within the same batch of data. This forces the system to learn the difference between a good answer to the right question and a good answer to the wrong one.

Beyond just catching cheaters, the system improves the fairness and consistency of grading for legitimate students. By using a method that maps both the question and the grade level into a shared mathematical space, the model can understand that a "B" grade for a difficult physics question should look different from a "B" grade for a simple history question. It does this without needing to store a separate, massive database for every single question, which keeps the system fast and efficient. The researchers found that their model could process answers in milliseconds, making it viable for real-time use in large online classrooms. The system also incorporates a technique called label smoothing, which acknowledges that the line between an "A" and a "B" is often blurry and subjective, preventing the AI from becoming overconfident in its decisions.

The study demonstrates that automated grading can move beyond simple keyword matching to a more robust form of understanding. The key finding is that by combining a standard grading engine with a dedicated consistency check, the system can maintain high accuracy even when faced with adversarial inputs or complex, imbalanced datasets. The researchers observed that the model did not just learn to predict grades; it learned to respect the context of the question. When the consistency gate determined that an answer was irrelevant, it overrode the grading engine to ensure a failing score, effectively acting as a safety net. This approach suggests that the future of educational assessment lies not in replacing human judgment entirely, but in creating tools that handle the heavy lifting of initial screening while flagging edge cases for human review. The work provides a reliable foundation for scaling personalized education, ensuring that the feedback students receive is based on what they actually wrote, not just how well they wrote it.

Technical Summary: Perceive AI

Problem Statement

Automated Short Answer Grading (ASAG) is critical for scalable educational assessment but faces three persistent operational vulnerabilities in current deep learning implementations:

Question-Agnostic Semantic Drift: Standard sequence classification models often learn spurious correlations (e.g., sentence length or vocabulary complexity) rather than prompt-specific relevance. Consequently, a high-quality answer to one question may receive a passing grade when incorrectly submitted for a different prompt.
Memory Overhead vs. Generalization: Existing systems struggle to balance the need for distinct grading criteria across hundreds of unique questions. Maintaining isolated models per question incurs prohibitive memory costs, while external database lookups fail to generalize to semantic variations.
Adversarial Vulnerability: Probabilistic softmax classifiers lack deterministic guardrails, often assigning non-zero confidence scores to off-topic, irrelevant, or adversarial submissions, thereby undermining pedagogical trust.

Methodology

The paper proposes Perceive AI, a unified, end-to-end framework built upon a partially unfrozen DistilBERT backbone. The architecture integrates three interdependent innovations to resolve the aforementioned challenges:

1. Dual-Head Parametric Prototype Architecture

Instead of relying solely on a global linear classifier, Perceive AI employs a hybrid scoring mechanism:

Global Linear Head: Maps pooled contextual vectors to a 5-dimensional grade space (A–F) to capture broad linguistic quality correlations.
Parametric Prototype Memory: An in-model learnable embedding table maps specific prompt identities ( $k_i$ ) and grade tiers ( $c$ ) directly into the latent space. The system computes a theoretical prototype vector $p_{k_i,c}$ as the L2-normalized sum of the question and grade embeddings. Grading is determined by the temperature-scaled cosine similarity between the student's response and these dynamic prototypes.

2. QA-Consistency Guardrail with Cyclic Negative Sampling

To enforce prompt relevance, an auxiliary binary consistency head is introduced.

In-Batch Cyclic Negative Generation: Rather than duplicating datasets, the system dynamically shifts student answers across questions within a minibatch (cyclic roll) to create hard negative pairs (mismatched question-answer pairs) on the fly.
Deterministic Guardrail: During inference, if the consistency head predicts a mismatch probability below a strict threshold ( $\gamma_{thresh} = 0.55$ ), the system overrides the grading output and assigns an automatic 'F' grade. This acts as a zero-tolerance filter for off-target submissions.

3. Composite Multi-Task Learning Objective

The model is optimized via a unified loss function combining four components:

Label-Smoothed Cross-Entropy: Applied to both the global and prototype heads to handle the ordinal and subjective nature of grading boundaries (A vs. B).
Binary Consistency Loss: Optimizes the guardrail head using the cyclic negative samples.
Supervised Contrastive Loss (SupCon): Forces embeddings of responses to the same question to cluster tightly in the latent space while repelling out-of-distribution submissions.

Key Contributions

The paper outlines four primary contributions:

Dual-Head Parametric Prototype Architecture: A hybrid mechanism that conditions classification on prompt identity via learnable prototype embeddings, eliminating the need for external databases or isolated models.
In-Batch Cyclic Negative Guardrail: A training-efficient method to generate hard negatives dynamically, establishing a deterministic inference-time filter that routes irrelevant submissions to failure.
Composite Multi-Task Metric Learning: A four-part loss function integrating label smoothing, binary consistency, and supervised contrastive learning to balance global quality assessment with prompt-specific clustering.
Empirical Validation: Demonstration of the framework's efficacy on a large-scale curriculum dataset.

Experimental Results

The framework was evaluated on a dataset of 32,499 student answers spanning 644 unique questions across 13 curriculum chapters.

Performance: The full Perceive AI model achieved a validation accuracy of 45.83%, a Macro F1 of 0.418, and a Quadratic Weighted Kappa (QWK) of 0.621. These metrics represent an improvement over baseline DistilBERT (41.20% accuracy) and variants using only prototypes or only gating.
Adversarial Robustness: In tests against 1,000 synthetically mismatched question-answer pairs, standard transformers assigned A or B grades to 64.2% of off-topic answers. Perceive AI, utilizing its consistency guardrail, achieved a 99.10% true-negative rejection rate, assigning an 'F' grade to nearly all off-target submissions.
Efficiency: The model exhibits rapid convergence and maintains an inference latency of approximately 14.2 ms on an NVIDIA T4 GPU.

Significance and Claims

The paper claims that Perceive AI provides a reliable and scalable foundation for automated educational assessment by addressing the "question-agnostic" failure mode of standard classifiers. By integrating parametric memory directly into the latent space, the system resolves the trade-off between memory overhead and generalization. Furthermore, the inclusion of a deterministic consistency guardrail is presented as a critical step toward pedagogical fairness, ensuring that automated grading systems do not reward superficial or irrelevant text. The author positions this work as a solution to the operational vulnerabilities that have hindered the production deployment of deep ASAG models.