Accelerating Accurate Assignment Authoring Using Solution-Generated Autograders

Technical Summary: Accelerating Accurate Assignment Authoring Using Solution-Generated Autograders

Problem Statement
The authors identify a critical bottleneck in computing education: the creation of large banks of programming practice problems is hindered by the difficulty of authoring accurate autograders. Traditional autograding relies on manually enumerating test cases using standard testing frameworks (e.g., JUnit, pytest). This process is described as tedious and prone to error, often resulting in autograders that fail to correctly classify submissions. Furthermore, traditional test suites lack a "source of truth," making it difficult to validate the accuracy of the autograder itself or to provide rich feedback on code quality beyond simple behavioral correctness. The authors note that without efficient authoring tools, creating the vast question banks necessary for frequent, low-stakes assessment and randomized quizzes remains infeasible for many instructors.

Methodology: Solution-Generated Autograding
The paper introduces "solution-generated autograding," a paradigm that leverages the existence of a reference solution (provided by the question author) to automatically generate, validate, and enhance the autograder. This approach is implemented in a tool called Questioner, designed for Java and Kotlin.

The methodology proceeds through the following technical stages:

1. Control Class Definition: The author provides a single "control class" containing the problem description, configuration metadata, and the reference solution.
2. Behavioral Analysis and Input Generation: Questioner analyzes the reference solution to determine its signature (parameters and return types). It utilizes built-in random generators for common types (e.g., int, boolean) to create input sequences.
3. Mutation-Based Accuracy Validation: To determine the number of test iterations required for accuracy without manual test case enumeration, Questioner generates a corpus of "incorrect" submissions by applying source-level mutations to the reference solution (e.g., swapping && with ||, changing < to <=).
   • The system runs the reference solution and the mutated (incorrect) versions against the same sequence of random inputs.
   • It counts the iterations required to distinguish all mutated incorrect submissions from the correct reference solution. This count defines the test suite size.
   • If a mutation cannot be distinguished (e.g., due to equivalent behavior), the author can suppress it via comments.
4. Code Quality Evaluation: Beyond behavioral correctness, Questioner extracts metrics from the reference solution to evaluate student submissions. These include:
   • Cyclomatic complexity: To identify overly complex solutions.
   • Execution time and memory usage: Measured via line execution counts and allocation, rather than wall-clock time.
   • Submission length and dead code: To encourage concise, efficient code.
   • Recursion enforcement: Ensuring recursive solutions are used when required.
5. Customization: For cases where default random generation fails (e.g., custom object types or specific edge cases like palindromes), authors can provide fixed input lists or custom generator methods.
6. Class Design Support: The system handles stateful class design problems by instantiating multiple objects with different constructor arguments and calling methods in random sequences, tracking state changes to detect failures that may not be immediately visible.

Key Contributions

• Questioner Tool: A Java/Kotlin implementation of solution-generated autograding that automates the creation of accurate autograders from a single reference solution.
• Mutation-Driven Validation: A novel approach to determining autograder accuracy by using mutated versions of the solution as a ground-truth corpus for incorrect submissions, eliminating the need for manual test case enumeration.
• Integrated Code Quality Metrics: The ability to evaluate submissions on non-functional attributes (complexity, length, dead code) derived directly from the reference solution's execution profile.
• Scalable Implementation: A backend architecture utilizing a Dockerized JVM server, MongoDB, and the Jeed library for high-performance, sandboxed execution.

Results and Experiences
The authors report on four years of deployment (2020–2024) in CS 124, a large introductory course at the University of Illinois.

• Scale: A single instructor authored nearly 800 programming questions (771 in the final bank) covering core CS1 concepts.
• Efficiency: The authoring process allowed for the creation of approximately one question per working day, a pace described as sustainable for a single instructor managing a large course.
• Usage: The system processed over 850,000 submissions in Spring 2024 alone, with a 99.96% successful grading rate.
• Performance: The median evaluation time was 43 ms, with the 99th percentile at 806 ms, enabling interactive student feedback.
• Accuracy: The authors state that despite authoring hundreds of new quiz questions, no quiz question had to be dropped due to autograder inaccuracy. The mutation-based validation ensured that incorrect submissions were reliably identified.
• Adoption: Two external instructors successfully used Questioner to author 59 questions for a public website, guided only by a tutorial.

Significance and Claims
The paper claims that solution-generated autograding fundamentally shifts the autograding workflow from "enumerating test cases" to "providing a solution." The authors argue this is significant because:

1. Speed and Enjoyment: It significantly reduces the time required to author large question banks, making it feasible for individual instructors to maintain extensive problem sets.
2. Accuracy: By using the reference solution as a source of truth and validating against mutated incorrect versions, the system ensures high accuracy in distinguishing correct from incorrect code.
3. Pedagogical Flexibility: It allows for rich feedback on code quality (e.g., complexity, recursion) without requiring the author to manually specify these constraints in test cases.
4. Pedagogical Decision vs. Limitation: The system allows instructors to intentionally obscure test details (if desired) as a pedagogical choice, rather than as a limitation of the autograding tool.

The authors modestly note limitations, such as the current restriction to single-class problems per question (though they mention a Python implementation, Snapact, addresses this) and the potential for adversarial submissions to bypass checks, though they argue such submissions often still meet pedagogical goals. They conclude that this approach enables the creation of large, diverse, and accurate problem banks essential for modern computing education.