TeachMateGPT: A Multi-Agent Knowledge-Grounded Framework for Pedagogical Assessment Generation from Science Curriculum Materials

Imagine you are a teacher trying to build a quiz for your science class. You have a thick, official textbook that holds all the answers, but you don't have time to flip through every page to find the perfect questions. You turn to a super-smart computer robot (an Artificial Intelligence) to help you. You ask it, "Make me a quiz about acid rain," and the robot starts typing. But here's the catch: sometimes these robots are like over-eager students who memorized a few facts but then start making things up to sound smart. They might invent a fake science rule or pull a fact from a different chapter entirely. This is called "hallucinating." To stop this, scientists use a trick called Retrieval-Augmented Generation (RAG). Think of RAG as giving the robot a library card and a strict rule: "You can only write your quiz if you can point to the exact page in the textbook where you found the answer."

However, even with a library card, the robot can still get confused. If the textbook is organized in a complex way, or if the robot grabs a tiny, isolated sentence without the surrounding context, it might still write a bad question. The big question for researchers is: How do we build a robot that doesn't just grab any page, but understands the structure of the lesson, knows when it doesn't have enough information to answer, and checks its own work before showing it to the teacher? This is the problem a new study tries to solve, specifically for science teachers in Bangladesh who use a very specific national textbook.

---

Meet TeachMateGPT: The Robot Librarian with a Safety Net

The paper introduces a new system called TeachMateGPT. You can think of it not as a single robot, but as a team of specialized workers in a high-tech factory, all working together to turn a teacher's request into a perfect, textbook-accurate quiz. The system is designed specifically for the National Curriculum and Textbook Board (NCTB) Class 8 science textbook in Bangladesh, a resource that is crucial for students but often difficult for AI to navigate perfectly.

Here is how the factory works, step-by-step:

1. The Smart Index (COPE)
First, the system has to organize the textbook. Imagine the textbook is a giant, messy attic. A normal robot might just grab a random box of stuff. TeachMateGPT uses a special tool called COPE (Curriculum-Oriented Pedagogical Embedding). Instead of just cutting the text into random chunks, COPE understands the "family tree" of the book. It knows that a "Chapter" contains "Lessons," which contain "Sections," which contain "Definitions." It builds a map where every piece of information is linked to its parents and neighbors. If the robot needs to know about a specific animal, it doesn't just find the word "animal"; it finds the whole paragraph about where that animal fits in the family tree of life. This ensures the robot understands the context, not just the words.

2. The Gatekeepers (The Routing Agents)
Before the robot even looks for answers, a team of "Gatekeeper" agents checks the teacher's request.

• The Intent Agent asks: "Is this a real science question, or did the teacher just say 'Hello'?"
• The Ambiguity Agent asks: "Is the question clear? If the teacher says 'Make a quiz about animals,' the agent stops and asks, 'Which chapter? Which type of animal?'"
• The Safety Agent makes sure no harmful or off-topic requests get through.
  If the request is too vague or unsafe, the system politely stops and asks for more details. It refuses to guess.

3. The Detective Team (Hybrid Retrieval)
Once the request is clear, the system goes hunting for evidence. It uses two types of search at the same time:

• The Semantic Detective: Looks for the meaning of the words (e.g., finding "acid rain" even if the text says "rain with high acidity").
• The Lexical Detective: Looks for exact scientific terms that might be missed by meaning alone.
  If the system finds the evidence, it uses a "Fail-Closed" rule. This is like a strict teacher who says, "If you don't have enough proof, you get a zero." If the system can't find enough textbook pages to support a question, it simply refuses to generate one, rather than making something up.

4. The Writers (Specialist Agents)
Once the evidence is gathered, different "writers" take over.

• The MCQ Specialist writes multiple-choice questions (like "A, B, C, or D?").
• The Creative Question Specialist writes the longer, story-based questions common in board exams (where a student reads a scenario and answers four parts).
  These writers are forced to use only the evidence the detectives found. They cannot use their own "memory" to invent facts.

5. The Inspector (SAVER)
Before the quiz is handed to the teacher, a final inspector named SAVER (Source-Attributed Verification and Evidence Ranking) checks the work. It looks at every single question and asks three questions:

1. Faithfulness: Did you actually find this fact in the textbook?
2. Relevance: Does this match what the teacher asked for?
3. Hallucination Risk: Did you make anything up?
   If the score is too low, the system flags the question for the human teacher to review. It doesn't automatically delete it; it just says, "Hey, check this one, I'm not 100% sure."

What Did They Find?

The researchers tested this system by generating 198 science questions (143 multiple-choice and 55 creative questions) covering all 14 chapters of the Class 8 textbook. They then had three real science teachers grade the results.

The results were quite impressive. Compared to a standard, "vanilla" AI system that just grabs text and writes:

• Faithfulness (how true the facts are to the source) jumped from 0.68 to 0.96. This means the new system is almost perfectly honest about where it got its information.
• Answer Relevancy (how well the answer fits the question) went from 0.60 to 0.89.
• Teacher Utility (how useful the teachers found the questions) skyrocketed from a score of 2.00 to 4.80 on a 5-point scale.

The study also showed that if you remove the "Smart Index" (COPE), the quality drops significantly. If you remove the "Inspector" (SAVER), the system starts making up facts again. This proves that both understanding the book's structure and double-checking the work are essential.

The Limits and The Future

The authors are careful to note what their system cannot do yet.

• It's Text-Only: The system reads the words of the textbook, but it struggles with pictures. If a question requires looking at a diagram of a circuit or a cell, the system can't "see" the image. It has to rely on the text description, which might miss the point of the visual question.
• It Needs Human Eyes: The system is designed to be a helper, not a replacement. It flags questions for teachers to check, but it doesn't make the final decision. The teachers must still review the work before giving it to students.
• Specific to One Book: This system was built specifically for the Class 8 science textbook in Bangladesh. It might not work perfectly for a different grade, a different subject, or a different country's curriculum without significant changes.

In short, TeachMateGPT suggests that by giving AI a better map of the textbook, a strict rule to stop guessing, and a built-in inspector, we can create a tool that helps teachers save time without sacrificing accuracy. It's a step toward a future where AI is a reliable co-pilot for education, rather than a risky autopilot.