AiDER: Auditing and Document Evaluation via Rule Compilation and Small Language Models - An Education Case Study

Imagine you are trying to teach a robot how to grade a student's homework. In the world of Artificial Intelligence, there are two main ways to do this. The first way is to let a giant, super-smart "brain" (called a Large Language Model) read the assignment and just guess the grade based on its gut feeling. It's fast, but it's a black box; if it gets it wrong, you have no idea why, and you can't really check its work. The second way is to use a strict rulebook. You write down exact instructions like "If the student mentions 'gravity,' give 5 points." This is transparent and easy to check, but it's hard to write rules for everything, especially when the instructions are written in messy, human language.

This paper sits right at the intersection of these two worlds. It asks: Can we combine the flexibility of a smart AI with the honesty of a strict rulebook? The researchers are working in the field of "auditable AI," which is all about making sure AI decisions can be explained and checked, rather than just accepted as magic. They are particularly interested in using "Small Language Models" (SLMs)—AI brains that are smaller, cheaper, and can run on a single computer without needing a massive cloud server. The big question is: Can a smaller, simpler AI be smart enough to turn a human's vague instructions into a strict rulebook that a computer can follow perfectly?

The authors introduce a new system called AiDER (Auditing and Document Evaluation via Rule Compilation and Small Language Models). Think of AiDER as a clever translator and a strict referee working together. Instead of letting the AI guess the grade, AiDER forces the AI to do two separate jobs. First, the AI acts as a translator: it takes a natural language requirement (like "Students need to know how they will be graded") and tries to compile it into a strict, executable code rule. Second, a separate, deterministic computer program acts as the referee: it looks at the document, finds the specific evidence the rule asked for, and checks if the rule is met. The AI never actually decides the grade; it just sets up the rules and finds the clues.

To test this, the researchers used four different small AI models (with sizes ranging from 2 billion to 4 billion parameters) and a set of 200 real-world teaching documents, like syllabi and grading rubrics. They tried three different ways to get the AI to write the rules:

1. Human-written rules: The "gold standard" where a human wrote the strict code.
2. Direct compilation: The AI tried to write the code in one go.
3. Validation-loop repair: If the AI wrote a broken rule, the system told it "Error!" and let the AI try again up to five times to fix it.

The results were a mix of "great news" and "cautious optimism." The study found that when the AI successfully wrote a valid rule, even the smallest models were surprisingly good at finding the right evidence to make a decision. However, the AI often struggled to write the perfect rule on the first try. This is where the "repair loop" shined. By letting the AI fix its own mistakes, the system significantly improved how often the rules worked correctly.

Interestingly, the paper suggests that once the strict rule (the "scaffold") is in place, the AI doesn't need much extra help to find the evidence. In fact, adding too many extra instructions from the AI sometimes made things worse, especially for the smaller models. The study concludes that the most important part of the process is getting the rule right. If the rule is clear, even a small, simple AI can do the heavy lifting of finding the proof.

Ultimately, the paper suggests that for tasks like checking educational materials, we shouldn't just ask an AI to "judge" a document. Instead, we should use AI to translate human requirements into strict, checkable rules, and then let a computer verify the facts. This approach makes the whole process transparent: if a decision is wrong, you can trace it back to see if the rule was written poorly, or if the AI missed a piece of evidence, rather than blaming a mysterious "black box" verdict. While the study was limited to one specific course and a small set of documents, it offers a promising blueprint for making AI in education more reliable, fair, and easy to understand.