Toward AI-Resilient Assessment in Computer Science Courses in an AI-Native World

Technical Summary: Toward AI-Resilient Assessment in Computer Science Courses in an AI-Native World

Problem Statement
The paper addresses the fundamental shift in computer science education caused by generative AI, where students can produce code, proofs, and reports indistinguishable from human work. The author argues that the current institutional focus on prohibition, detection, and proctoring fails to address the core educational question: What skill allows a student to achieve outcomes beyond a person who merely has access to AI? The paper posits that for senior-level courses, AI should be treated as a baseline collaborator rather than a forbidden tool. The challenge is to design assessments that grade "AI-resilient skills"—the ability to direct AI to achieve tradeoffs not already supplied by the AI baseline itself—without relying on authorship detection or assuming students will not use AI.

Methodology: A Formal Framework for AI-Resilient Grading
The paper proposes a minimal formal framework where assessment is defined by executable behavior rather than authorship. The core methodology relies on four objects: a real task, an executable evaluator, a declared AI-native Pareto frontier, and a grading rule based on Pareto surplus.

1. Real Task and Evaluator: A task is defined not just by a prompt, but by a tuple including a submission space, a benchmark protocol (public development and hidden evaluation), hard correctness constraints, and a total executable evaluator ($E_T$) that maps submissions to a vector of scores ($R^d$).
2. AI-Native Pareto Frontier: Before grading, a "declared AI-native baseline" is generated using a reproducible AI-agent protocol (including specific models, prompts, budgets, and selection rules). The scores of these AI-generated submissions form a set $S_{AI}$. The AI-native Pareto frontier ($F_{AI}$) is the set of nondominated score vectors within this set. This frontier represents the best-known tradeoffs achievable by the declared AI baseline.
3. Pareto Surplus: A student submission $\pi$ is evaluated by its Pareto surplus ($\delta_T(\pi)$), calculated as the increase in the dominated hypervolume (relative to a reference point $r$) when $\pi$ is added to the frontier.
   • If $\delta_T(\pi) = 0$, the submission is dominated by the AI baseline and offers no new tradeoff.
   • If $\delta_T(\pi) > 0$, the submission achieves a measured tradeoff not covered by the AI baseline.
4. Grading Rule: Grades are assigned based on Pareto surplus. An AI-native submission receives a baseline grade ($g_{AI}$). Any grade above $g_{AI}$ requires a positive Pareto surplus.
5. Extensions for Fairness: To prevent private AI budget (e.g., access to more powerful models or more compute) from becoming the sole determinant of grades, the framework introduces:
   • Server-Mediated Feedback: A feedback oracle that releases only aggregate, coarse-grained information (e.g., pass/fail, score bins) rather than hidden inputs or per-instance errors, preventing students from overfitting to hidden benchmarks via private AI loops.
   • Thresholded Grading: A threshold $\eta_T$ is set based on red-teaming results ($\epsilon_{red}^{bud}$) and evaluator noise ($\epsilon_{eval}$). Only surplus exceeding this threshold ($\delta > \eta_T$) is rewarded, creating a "budget-neutrality margin" where private AI-only improvements within the declared envelope do not yield grade-bearing surplus.
   • Prompt-Skill Red-Teaming: The framework distinguishes between "generic delegation" (which should be part of the baseline) and "frontier-improvement rationales" (where a student identifies a specific bottleneck and hypothesizes a fix). The latter is considered evidence of skill.

Concrete Instantiation: Bloom Filter Assignment
The framework is instantiated as a senior-level assignment for COMP 480/580 at Rice University, centered on approximate-membership structures (Bloom filters).

• Two Regimes: The assignment uses two scales: MB-scale (megabytes) focusing on parameter tuning and low-overhead hashing, and GB-scale (gigabytes) focusing on memory bandwidth, cache locality, parallelism, and vectorization.
• Evaluation: Students must implement systems that satisfy hard correctness constraints (no false negatives) while optimizing a vector of objectives (e.g., compression ratio, false-positive rate, build/query throughput).
• Process: Students receive a frozen AI-native frontier for both scales. They may use AI freely but are restricted to a fixed server feedback budget. To earn a grade above the baseline, they must produce a system that expands the hypervolume beyond the frontier by more than the calculated threshold.

Key Contributions

1. Formal Model: A minimal mathematical model where Pareto frontier expansion serves as an artifact-level certificate of performance beyond an AI baseline.
2. Theoretical Guarantee: A proof that Pareto-surplus grading is AI-resilient with respect to the declared frontier; AI-native submissions receive only the baseline grade, and above-baseline grades strictly require measurable frontier expansion.
3. Protocol Extensions: A framework for handling self-improving AI loops, budget neutrality (via thresholding), server-mediated feedback, and the distinction between productive AI competence and unequal private resources.
4. Automation and Incentives: An explanation of how executable surplus grading automates evaluation and alters incentives against collusion (leaking a frontier-moving solution reduces the leaker's relative grade in a percentile-calibrated system).
5. Concrete Design: A specific, two-regime Bloom filter assignment design intended for deployment in Fall 2026.

Results and Claims
The paper does not report empirical results from student deployments, as the framework is proposed for future implementation. Instead, it presents design guarantees and logical theorems:

• Theorem 1: Pareto surplus is a measurable certificate of frontier expansion; a positive surplus implies the artifact is not dominated by the AI baseline.
• Theorem 2: Under Pareto-surplus grading, AI-native submissions receive exactly the baseline grade, and any grade above baseline certifies measured improvement.
• Theorem 3: Thresholded grading separates surplus from private AI budget and evaluator noise. If the frontier is $\epsilon$-saturated against a declared private AI envelope, AI-only submissions within that envelope receive no surplus grade above the baseline.
• Null-Frontier Outcome: The paper notes that if no student produces positive surplus, it is a diagnostic signal that the task or evaluator failed to capture an AI-resilient skill, prompting a revision of the assignment.

Significance and Positioning
The paper positions itself as a move from "speculation and anxiety" to "concrete objects that can be inspected and debated." Its significance lies in reframing AI-resilient assessment not as a problem of detection, but as a design problem of defining explicit goals and measurable frontiers.

• It argues that once goals are explicit, AI-resilient evaluation becomes a protocol-relative certificate of behavioral improvement.
• It claims that this approach allows faculty to shift their role from manually constructing artifacts to specifying objectives, validating evaluators, and defining what constitutes "frontier movement."
• The author acknowledges limitations: the framework relies on the ability to define executable tasks and multidimensional evaluators, which may not apply to all educational objectives (e.g., ethical reasoning or communication). It also admits that "budget neutrality" is a design goal to be red-teamed, not a metaphysical guarantee.
• The ultimate goal is to provide a framework for critique and refinement before the planned Fall 2026 deployment at Rice University, inviting the community to test the robustness of the baselines, feedback policies, and thresholds.