SCOPE and SCION: A Benchmark and an Auditable Reference Pipeline for Schema Induction and Fusion from Text
This paper introduces SCOPE, a benchmark for evaluating schema induction from raw text, and SCION, an auditable reference pipeline that constructs and fuses schema graphs from training data, demonstrating superior performance over existing baselines while providing fully transparent, evidence-linked outputs.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to build a massive, super-organized library for a city that has never had one before. You have millions of books (raw text) scattered on the floor, but you don't have a catalog system yet. In the world of computer science, specifically in Information Extraction and Knowledge Graphs, this catalog is called a "schema." Think of a schema as the library's master blueprint: it's the list of allowed categories (like "Person," "Organization," or "Event") and the rules for how they connect (like "Person works for Organization"). Usually, computer programs are like super-fast librarians who can only work if someone hands them a finished blueprint first. But in the real world, creating that blueprint is slow, expensive, and often different for every library. If you want to organize news about finance, you need a different blueprint than if you want to organize news about biology. The big question researchers are asking is: Can we teach a computer to look at the messy pile of books and build its own perfect blueprint, or even merge it with an old one, without a human architect holding its hand?
This paper introduces two new tools to solve exactly that problem: SCOPE and SCION.
First, the authors created SCOPE, which is like a giant, standardized "test kitchen" for schema building. Before this, it was hard to compare different computer programs because everyone used different messy test data. SCOPE gathers 24 different public datasets (like news archives and scientific papers) and turns them into a strict, fair test. It gives the computer programs only the raw text (the "train" split) and asks them to build a schema from scratch. The computer isn't allowed to peek at the "gold" answer key until the very end. This setup forces the programs to prove they can actually learn the rules of the library from the books themselves, rather than just memorizing a list they were given.
Then, the authors built SCION, a new "reference pipeline" (a step-by-step recipe) for how to build these schemas. Instead of just letting a powerful AI chatbot wander off and write whatever it wants, SCION acts like a strict, auditable project manager. Here is how it works:
- The Candidate List: First, SCION scans the text to find potential ideas (like "Person" or "Buyer") and puts them in a candidate bucket. It doesn't let the AI invent new things out of thin air; the AI can only pick from this bucket.
- The Contract: The AI is forced to follow a strict "JSON contract." It must output its blueprint in a specific, machine-readable format and must point to the exact sentences in the text that prove why it chose those categories.
- The Safety Net: If the AI tries to break the rules or its output is messy, SCION has a "fallback" plan. It can switch to a simpler, deterministic method to ensure the blueprint is never broken. It also keeps a detailed log of every decision, so humans can audit exactly how the blueprint was made.
The paper tested this recipe against other methods, including older techniques and powerful AI models working alone. The results suggest that SCION-lite (a compact version of their recipe) performs better than the others. It achieved the highest scores in matching the "gold" blueprints, with a Literal F1 score of 0.7518 and a Graph F1 score of 0.7888. This means it was better at figuring out the right categories and how they connect, even when the text was messy or in different languages.
Crucially, the paper also argues against the idea that we should just let AI "hallucinate" or freely invent schemas. The authors show that when AI is allowed to wander without constraints, it often creates inconsistent or unreliable blueprints. By forcing the AI to stick to the "candidate bucket" and prove its choices with evidence, SCION makes the process auditable and controllable. They even created a smaller, open-source version called SCION-RL (using a model called Qwen3-8B) that reduces the need for expensive, proprietary AI models, proving that this strict, evidence-based approach works even with smaller tools.
In short, the paper suggests that the best way to teach computers to organize the world's information isn't to let them dream up their own rules, but to give them a strict, evidence-based framework where they must justify every category they create. It's the difference between a chaotic brainstorming session and a disciplined engineering team building a library that actually works.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.