💻 computer science

Test-Driven, AI-Assisted Learning: Replacing Lectures with Weekly Closed-Book Tests

This experience report details a 13-week redesign of a Theory of Computation course at HKUST (Guangzhou) that replaces lectures with self-directed, AI-assisted learning and frequent closed-book tests, demonstrating that an AI-driven material production harness can make high-frequency testing scalable while maintaining student accountability and fairness.

Original authors: Jin-Guo Liu, Shang-Qi Lu, Xin-Ran Shi, Long-Li Zheng, Wei Wang

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jin-Guo Liu, Shang-Qi Lu, Xin-Ran Shi, Long-Li Zheng, Wei Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Swapping the "Show" for the "Check"

Imagine a cooking class. In a traditional class, a famous chef stands at the front, chops vegetables, and explains the recipe while the students watch. The students nod along, feeling like they understand. But when they go home to cook, they might burn the food because they never actually held the knife.

The authors of this paper tried a different approach for a tough computer science class (about how computers "think" and solve problems). They asked: "What if we stop the chef from cooking in front of the class, and instead make the students cook every single week, but with a safety net?"

They call this TDAA (Test-Driven, AI-Assisted Learning).

How It Worked: The Two-Part Machine

The course had two main parts working together, like a bicycle with two wheels:

1. The Student's Wheel: "Practice, Then Prove It"

Instead of sitting through lectures, students learned on their own using:

AI Tutors: They could ask an AI (like a super-smart chatbot) to explain concepts whenever they were stuck.
Study Sheets: The teacher provided a "cheat sheet" of what to learn that week.
Practice Problems: They had a set of practice questions to try.

The Catch (The "Quality Gate"):
Every week, there was a 30-minute, closed-book test.

No phones.
No AI.
No notes.
Just the student and the problem.

Think of this like a driver's license road test. You can study the manual and practice in a parking lot all you want, but you have to prove you can drive safely on your own before you get your license. If you fail the weekly test, you know you aren't ready yet. If you pass, you move on.

2. The Teacher's Wheel: The "AI Factory"

You might ask: "If the teacher isn't lecturing, what are they doing? And how do they grade 18 tests every single week?"

The teacher used a special AI "Harness" (a fancy word for a set of automated tools).

The Factory: The teacher told the AI, "Here is the textbook. Make a study sheet, a practice quiz, and a real test for next week."
The Inspector: The AI wrote the materials, but a second AI (the "Reviewer") checked for mistakes.
The Human Boss: Finally, the human teacher looked over everything to make sure it was perfect before releasing it to students.

The Analogy: Imagine a bakery. The AI is the robot arm that mixes the dough and shapes the bread. The human teacher is the head baker who tastes the bread to make sure it's not burnt before putting it in the display case. Without the robot, the baker would be too tired to bake 12 weeks of bread. Without the baker, the robot might serve burnt bread.

What Happened? (The Results)

The paper is a report on one specific class with 18 students. Here is what they found:

Students Liked the "Gate": Surprisingly, students didn't hate the weekly tests. They felt the tests were fair because they had a clear path to prepare (the study sheets and practice problems). They said the tests forced them to actually study instead of procrastinating.
The Teacher's Job Changed: The teacher stopped being a "performer" (lecturing) and became a "coach" and "architect." They spent less time talking and more time fixing the AI's mistakes and helping students who were truly stuck.
It Was Doable: One teacher managed to run this entire 13-week course alone. Without the AI tools, creating that much material and grading it every week would have been impossible.

The Hiccups (Where It Strained)

It wasn't perfect. The paper admits a few things that need fixing:

Hard Topics: When the math got really hard (like complex proofs), the AI-generated explanations were sometimes too short or confusing. Students wanted more examples and pictures, which the AI didn't always provide well.
The "Cheat" Risk: Students realized that if they just memorized the practice problems, they could pass the test without really understanding the deep concepts. The teachers are working on making sure the practice and the test are different enough to prevent this.

The Bottom Line

This paper doesn't claim this method is the "perfect" way to teach everyone forever. It's just a proof of concept.

The main takeaway is a new recipe:

Don't just lecture. Let students learn at their own pace with AI help.
Test them often without help to make sure they actually learned it.
Use AI tools to handle the boring work of making tests and grading, so the human teacher can focus on quality control and helping students.

The authors are sharing their "AI Factory" tools (the code and templates) so other teachers can try this recipe in their own classrooms.

Technical Summary: Test-Driven, AI-Assisted Learning (TDAA)

1. Problem Statement

Traditional lecture-based instruction often creates "staged engagement," where students appear attentive but lack the ability to independently apply concepts. This issue is exacerbated by Generative AI (GenAI), which lowers the cost of obtaining explanations and completing take-home assignments, thereby weakening the signal of individual student mastery. While AI offers potential for personalized feedback, it introduces a critical challenge: how to design a course that leverages AI for learning while maintaining a rigorous, verifiable mechanism to ensure students can perform unaided. Existing assessment models struggle to balance the scalability of AI-assisted production with the necessity of frequent, high-stakes individual accountability.

2. Methodology

The paper presents an experience report on a 13-week redesign of DSAA 3071 (Theory of Computation) at the Hong Kong University of Science and Technology (Guangzhou) in Spring 2026. The study employs a Test-Driven, AI-Assisted (TDAA) model, which replaces traditional lectures with a coupled cycle of student learning and AI-assisted material production.

The Student Cycle (Learning Model)

No Lectures: Students engage in self-paced study using prepared materials and AI agents for just-in-time explanations.
Weekly Closed-Book Gate: Every week concludes with a 30-minute, strictly closed-book test (no AI, notes, or internet) on the previous week's material.
Preparation Artifacts: To ensure the strict gate is perceived as fair rather than arbitrary, students are provided with three aligned artifacts:
1. Learning Sheet: Defines concepts, notation, and examples for the week.
2. Validation Sheet: Practice problems aligned with the learning sheet.
3. Test Sheet: The closed-book assessment, pre-written against the same scope.
Assessment Structure: 30% of the grade comes from weekly tests (capped at 100/130 points to reduce penalty pressure), and 70% from a final exam. There are no take-home projects or homework.

The Production Cycle (The Harness)

To make frequent, high-quality material production and grading feasible for a single instructor, the authors implemented a version-controlled AI agent workspace (the "harness").

Pipeline: A three-stage process involving an AI Writer (drafts materials), an AI Reviewer (flags gaps and errors), and a Human Approver (final sign-off).
Agent Architecture: Agents operate as tool-using loops (reading, editing, writing, searching) within a Git repository. They utilize a project specification (e.g., AGENTS.md), a knowledge base (textbook converted to Markdown), and a set of named skills (e.g., generate-week, review-tests, grade-homework).
Human-in-the-Loop: AI handles drafting, initial review, and routine grading. Humans perform final approval of materials and escalate uncertain grading cases. AI never acts at the closed-book gate.

3. Key Contributions

The paper contributes a reusable design pattern and an open-source starter template for the course-materials harness.

The TDAA Pattern: A coupling of high-frequency, strict closed-book testing (to ensure individual accountability) with AI-assisted material production (to ensure scalability).
The Harness: A public, version-controlled workspace that allows instructors to generate, review, and grade weekly course bundles. It separates the automation of repetitive tasks from the judgment required for pedagogical alignment and error correction.
Operational Feasibility Evidence: Demonstration that a single instructor can sustain a 13-week course with weekly closed-book tests and immediate feedback by offloading drafting and routine marking to agents, provided a human review layer exists.

4. Results

Data was collected from $N=18$ students via an end-of-term survey, weekly test scores, and a git-history audit of the production pipeline.

Student Acceptance (RQ1): Students generally accepted the lecture-free design.
- Perception: 56% reported the format was both "deeper and faster" than traditional lectures. Average agreement scores were high for learning depth (5.50/7) and retention expectations (5.17/7).
- Accountability: The weekly test was ranked as the second most missed component if removed, with students citing it as a necessary push to study and a tool for self-check.
- Attainability: The "gate" was strict but attainable; 80.1% of all test observations scored at least 100/130 points, and weekly medians consistently hit the 100-point credit cap.
Instructor Role (RQ2): The teacher's role shifted from exposition to diagnosis and coaching.
- Students still requested targeted mini-lectures for difficult concepts (e.g., reductions, NP-completeness) and noted limitations in their ability to self-correct AI errors.
- The harness reduced recurring staff time from an estimated 7–10 hours/week (traditional) to 4–5 hours/week (TDAA), primarily by automating drafting and first-pass grading.
Production Quality (RQ2):
- Material Generation: AI-generated learning sheets were usable at scale but required human review. One instance of a "plausible but wrong proof shortcut" was caught by the reviewer.
- Test Reliability: A git audit of 273 answer-key blocks showed 96.3% required no post-release semantic repairs, indicating agents can produce mostly correct formal keys.
- Grading: AI-assisted marking successfully handled routine items, with the harness routing uncertain cases to humans. The 100-point cap mitigated the impact of minor grading uncertainties.
Design Strains (RQ3):
- Scaffolding: Proof-heavy topics (e.g., PSPACE, reductions) were perceived as "too terse," indicating a need for better scaffolding and more worked examples.
- Validation Trap: Students occasionally relied too heavily on the validation sheet for pattern matching rather than deep understanding.
- Improvement Signals: Students requested more examples, a practice problem bank, and AI-workflow guides, rather than fewer tests.

5. Significance and Claims

The paper modestly frames its contribution as an experience report and a reusable design pattern, explicitly avoiding causal claims regarding learning gains due to the lack of a control group and pre/post testing.

Core Claim: The TDAA design demonstrates that high-frequency closed-book testing can preserve individual accountability in an AI-rich environment, while an AI-assisted harness makes the operational burden of such a system feasible for a single instructor.
Significance: It offers a concrete solution to the "assessment crisis" in the GenAI era, showing that AI can be integrated into the production of education without compromising the verification of learning.
Limitations: The study is limited to one small, proof-heavy course ( $N=18$ ) without a control group. The workload figures are planning estimates, and the git audit is descriptive rather than a formal benchmark. The authors note that transfer to large or non-proof-based courses remains a hypothesis for future work.

The authors conclude that the teacher's role has not vanished but has moved toward designing learning paths, overseeing AI-generated materials, and diagnosing confusion that self-paced work fails to resolve. The released harness serves as a practical tool for other instructors to reproduce this model.