Quality Assessment of Public Summary of Training Content for GPAI models required by AI Act Article 53(1)(d)
This paper presents a research-based framework to evaluate the transparency and usefulness of public training summaries required by the EU AI Act for General-Purpose AI models, applying it to five existing summaries to identify issues and guide providers and authorities toward higher-quality disclosures.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you just bought a fancy new blender. The box says it's a "General Purpose Blender," but you have no idea what ingredients went into the smoothie it made. Did it use organic strawberries? Did it accidentally blend in a plastic spoon? Did it use fruit that was stolen from a neighbor's garden?
In the world of Artificial Intelligence, the "blender" is the AI Model, and the "ingredients" are the data used to train it.
The European Union's AI Act is a new set of rules designed to make sure these blenders are safe and fair. One specific rule (Article 53) says: "AI companies must publish a 'Public Summary' of their ingredients." They have to fill out a specific form (a template) telling everyone exactly what data they used.
However, just because a company fills out a form doesn't mean they did a good job. They might write "We used fruit" (too vague), or they might hide the fact that they used plastic spoons (bad faith).
This paper is like a Quality Control Inspector for those ingredient lists. The authors created a "Report Card" system to grade how well AI companies are following the rules.
🕵️♂️ The Problem: The "Black Box" of AI
Right now, many big AI companies are keeping their training data secret. The EU wants to change that so that:
- Artists and Writers can see if their work was used without permission.
- Regular People can understand how the AI works.
- Lawyers can enforce rights if something illegal happened.
But there's a catch: The EU gave companies a template to fill out, but they didn't say how to check if the answers are actually good. It's like giving a student a test sheet but no answer key.
📝 The Solution: The "Report Card" Framework
The authors (researchers from Trinity College Dublin and Mozilla) built a Quality Assessment Framework. Think of this as a giant checklist with 242 specific questions.
They judge the AI companies' summaries on two main things:
Transparency (The "Flashlight" Test):
- Is the information clear?
- Is it complete, or are there holes?
- Is it consistent (does the story change halfway through)?
- Analogy: If you shine a flashlight on the document, can you see everything clearly, or is it foggy and blurry?
Usefulness (The "Tool" Test):
- Can a regular person actually use this info to find out if their art was stolen?
- Is the document easy to find and read?
- Analogy: Is this document a working screwdriver, or is it a piece of plastic that looks like a screwdriver but doesn't turn any screws?
📊 The Results: Who Passed? Who Failed?
The researchers tested 5 AI models that had published these summaries (as of early 2026). Here is how they did:
- 🏆 The A-Students (Apertus & Bria): These companies filled out the form almost perfectly. They were clear, honest, and easy to understand. Their "Report Cards" got an A or A+.
- 👍 The B-Students (SmolLM & Bielik): These were good, but had some messy handwriting or missing details. They got a B+. They did the job, but they could be more precise.
- 📉 The F-Student (Microsoft's Phi-4): This one was a disaster. The document was vague, missing huge sections, and gave answers like "We follow the law" instead of actually listing the data. It got a D for transparency and an F for usefulness. It was like submitting a blank piece of paper with a sticky note saying "Trust us."
- 🚫 The Missing Students: The biggest AI giants (Google, OpenAI, Meta) hadn't published any summaries yet. They were effectively skipping class.
🛠️ Why This Matters
The authors argue that if we don't have high-quality summaries, the whole law is useless.
- If the summary is bad, a copyright lawyer can't prove the AI stole their client's work.
- If the summary is hidden, a citizen can't know if their data was used.
💡 The Takeaway
The paper isn't just grading homework; it's trying to fix the system.
- For AI Companies: "Stop being lazy! Fill out the form properly, or you'll get a bad grade and maybe a fine."
- For the EU: "The template needs to be clearer, and maybe you should have a central website where all these summaries live, so people don't have to hunt for them like treasure."
- For You: "We built a public website where you can see these grades. If you want to know if an AI is trustworthy, check its report card first."
In short: This paper is the "Consumer Reports" for AI training data. It's here to make sure the AI companies aren't just pretending to be transparent, but are actually being honest about the ingredients in their digital smoothies.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.