Self-hosted Lecture-to-Quiz: Local LLM MCQ Generation with Deterministic Quality Control

This paper presents an end-to-end, self-hosted pipeline that converts lecture PDFs into multiple-choice questions using a local LLM and deterministic quality control, ensuring privacy and accountability while releasing a validated 24-question dataset with a detailed warning taxonomy for educational use.

Seine A. Shintani

Published Wed, 11 Ma
📖 5 min read🧠 Deep dive

Imagine you are a teacher who wants to turn a 50-page lecture PDF into a fun, multiple-choice quiz for your students. Usually, you might copy-paste that text into a powerful AI chatbot (like a cloud-based "genius") and ask it to write the questions. But that has two big problems:

  1. Privacy: You are sending your private, unpublished lecture notes to a stranger's computer.
  2. Reliability: The AI might make mistakes, like giving two answers that are both correct, or writing the same question twice, and you might not notice until it's too late.

This paper presents a solution called L2Q (Lecture-to-Quiz). Think of it as building a private, self-contained factory right in your own classroom (or on your own computer) to make these quizzes, without ever calling the outside world for help.

Here is how the process works, explained with some everyday analogies:

1. The Setup: The "Local Factory"

Instead of mailing your lecture notes to a big tech company's cloud, you run the "AI brain" (a Local LLM) right on your own machine.

  • The Analogy: Imagine you have a very smart, but slightly distracted, apprentice working in your kitchen. You give them your secret family recipe (the lecture PDF). They stay in your kitchen the whole time. No one else sees the recipe.

2. The Process: The "Assembly Line"

The system doesn't just ask the AI to "make a quiz." It runs a strict, step-by-step assembly line:

  • Step A: The Plan. The AI reads the lecture and makes a tiny shopping list of topics to cover.
  • Step B: The Drafting. The AI writes the questions. It's told to write them in a very specific format (like filling out a strict form), so the computer can read them easily.
  • Step C: The "Bouncer" (Quality Control). This is the most important part. Before a question is allowed to leave the factory, it goes through a Bouncer (a set of strict computer rules).
    • The Rule: "Do you have exactly 5 options?"
    • The Rule: "Is there only ONE correct answer?"
    • The Rule: "Did you accidentally write the same wrong answer twice?" (e.g., Option C and Option E both equal 5).
    • The Rule: "If the answer is a decimal, did you tell the student how many digits to round to?"

3. The "Retry" Mechanism

If the Bouncer catches a mistake, the AI doesn't just get a warning; it has to redo the question immediately.

  • The Analogy: It's like a baker who drops a cake. The Bouncer says, "That cake is lopsided." The baker doesn't argue; they just throw it in the trash and bake a new one until it's perfect. The system keeps trying (up to 3 times) until the question passes the test.

4. The Result: A "Static" Quiz

Once the questions pass the Bouncer, they are saved as a simple text file (like a spreadsheet).

  • Why this matters: When your students take the quiz later, they do not need an AI. They just open the file. The AI is gone. The quiz is just a piece of paper (or a digital file) that is safe, fast, and doesn't cost money to run every time a student takes it.

The "Seed Sweep" Experiment

The author tested this on three short, made-up lectures about science (entropy, thermodynamics, etc.).

  • They ran the factory 15 times with different random settings (like shuffling the deck of cards).
  • They generated 120 questions.
  • The Good News: Every single question passed the strict "Bouncer" checks. The system was very stable.
  • The "Warning" Layer: Even though the questions were technically perfect, the system flagged 8 of them for a human to double-check. For example, one question asked for a number but didn't say "round to two decimal places." The computer couldn't fix that logic, so it raised a flag for the teacher to fix.

Why is this a big deal? (The "AI2L" Lens)

The paper connects this to a philosophy called AI2L (AI to Learn).

  • Privacy: Your lecture notes never leave your computer.
  • Accountability: You can see exactly how the question was made and check the math.
  • Green AI: You don't need to keep paying for expensive cloud servers every time a student takes a quiz. You generate the quiz once, and then it's free to use forever.

The Catch (Limitations)

The author is honest about what this can't do yet:

  • It's not a genius teacher: The computer can check if the math adds up, but it can't tell if the question is actually fair or if it tests the right concept. A human teacher still needs to look at the final list and say, "Yes, this is a good question."
  • It struggles with pictures: If your lecture PDF has complex charts or handwritten diagrams, the system might miss them because it only reads the text.

In a Nutshell

This paper describes a privacy-safe, self-correcting machine that turns your lecture notes into a quiz. It acts like a tireless intern that drafts the questions, a strict inspector that catches math errors, and a final editor that packages everything up so you can hand it to your students without needing any internet connection or AI subscriptions.