Open Technical Problems in Open-Weight AI Model Risk Management

Imagine the world of artificial intelligence as a massive, bustling library. For a long time, the most powerful books in this library were kept behind glass cases, locked up by a few giant corporations. You could read them, but you couldn't take them home, and you certainly couldn't tear out pages or rewrite the chapters. These are "closed-weight" models. But recently, a new trend has exploded: "open-weight" models. These are like digital copies of the library's most powerful books that anyone can download, take home, and keep forever. The best part? You can photocopy them, glue them together, or even rewrite the text to make them do new things. This is incredibly exciting for researchers and inventors who want to build amazing new tools.

However, there's a catch. When you give everyone a copy of a book and let them rewrite it, you lose control. If someone writes a dangerous story in their version, they can't be stopped from sharing it with the whole world. Unlike the locked books, you can't just call the librarian to take the dangerous version back; once it's out, it's out. This paper, written by a huge team of experts from universities and safety organizations, asks a critical question: How do we keep this open library safe when anyone can change the books? They aren't trying to lock the library back up; instead, they are trying to invent new kinds of "magic ink" and "security guards" that stay safe even when the book is being rewritten. They want to figure out how to build AI that is both open for everyone to use and tough enough to resist being turned into a weapon.

The Big Problem: The Unstoppable Copy Machine

The authors explain that open-weight AI models are becoming incredibly powerful, sometimes just 6 to 12 months behind the super-secret models kept by big companies. The problem is that while closed models can be patched or pulled back if they start acting weird, open models are like digital viruses. Once someone downloads them, they can be modified, shared, and spread forever.

Think of a closed model like a smartphone app. If the app has a bug that lets hackers steal your photos, the company can push an update to fix it, or even remove the app from the store. But an open-weight model is like a recipe for a cake. If you find a dangerous ingredient in the recipe, you can't stop people who already have the recipe from baking the cake. Even worse, someone can take that recipe, add a "poison" ingredient (like a hidden instruction to hurt people), and share the new, dangerous recipe with millions of others. The original baker has no way to stop this.

The paper highlights that current safety tools, like filters that block bad words, are useless against open models. Why? Because if you have the recipe (the model's code), you can just erase the filter. It's like trying to stop someone from baking a cake by putting a "Do Not Eat" sign on the kitchen counter; if they have the whole kitchen, they can just ignore the sign.

The 16 Challenges: A Toolkit for a Wild West

The paper doesn't just point out the problem; it lays out 16 specific technical challenges that scientists need to solve to make open AI safe. They organize these into five main categories, like different stations in a factory that builds these AI models.

1. The Ingredients (Training Data Curation)
Before an AI learns, it eats a massive amount of data, like a student reading millions of books. The first challenge is: How do we make sure the student never reads the bad books?
The authors suggest that if we filter out dangerous information (like instructions on how to build bombs or create illegal images) before the AI starts learning, the AI might never learn those bad tricks in the first place. It's like teaching a child to be kind by only giving them books about kindness, rather than trying to teach them to be kind after they've already learned to be mean. However, the paper notes this is tricky. Sometimes, removing a specific topic might accidentally make the AI worse at something else, or the AI might still figure out the bad stuff from innocent-looking clues.

2. The Unbreakable Armor (Tamper-Resistant Training)
Even if we start with a safe AI, someone might try to "fine-tune" it—teach it new, bad tricks later on. The second challenge is: How do we build an AI that refuses to learn bad tricks, even when someone tries to force it?
Imagine trying to teach a dog to bite, but the dog is so well-trained that it physically cannot learn the command. The paper says current methods are weak. If someone tries to retrain a safe AI to be dangerous, they can usually do it in just a few hundred steps. The authors want to find ways to make the AI "immune" to this, so that even if someone tries to hack it, the AI stays safe. They also explore "unlearning"—a way to make the AI forget specific bad things it might have learned, but current methods are easily reversed.

3. The Stress Test (Model Tampering Evaluations)
Before we release a new AI, we need to test it. The third challenge is: How do we test an AI to see if it can be broken by someone trying to hack it?
Right now, most tests just ask the AI normal questions. The authors say we need to hire "red teams"—groups of hackers whose job is to try to break the AI by tweaking it in every possible way. They want to know: How many steps does it take to turn a helpful AI into a harmful one? How much does it cost? The paper finds that we don't have good tests for this yet. We need to know exactly how easy it is to break these models so we can fix them before they are released.

4. The Slow Release (Staged Deployment)
Instead of dropping a new AI on the world all at once, the fourth challenge is: Can we release it in stages to watch what happens?
Imagine a new video game. Instead of letting everyone play it immediately, the developers let a small group of testers play it first. If they find a bug, they fix it before the whole world gets it. The authors suggest doing this with AI. We could let a few trusted researchers use the model first, watch how they use it, and only open it up to everyone once we are sure it's safe. They also look at technical tricks like "split deployment," where part of the AI runs on your computer and part runs on a secure server, making it harder for someone to steal the whole thing.

5. The Detective Work (Model Provenance and Forensics)
Finally, once the AI is out in the wild, the fifth challenge is: How do we know where a specific AI came from and what changes were made to it?
If you find a dangerous AI model on the internet, how do you know who made it? The authors suggest using "watermarks"—invisible digital fingerprints baked into the model's code. If someone tries to change the model, the fingerprint should still be there, or at least we should be able to trace the family tree of the model to see who modified it. This helps researchers track how dangerous models are spreading and who is responsible.

What the Paper Found (and What It Didn't)

The authors looked at the technical reports of the top 10 most popular open AI models released in 2025, as well as several famous image and video generators. They found a worrying gap.

• Data Cleaning: Most companies talked a little bit about cleaning their data (removing bad stuff before training). Some gave a paragraph, a few gave a whole section, but many didn't say much.
• The Missing Pieces: Almost no one talked about making their models resistant to tampering. None of the top models mentioned "unlearning" algorithms or "tamper-resistant" training. Only one model (gpt-oss) had a dedicated paper about testing how hard it was to break their model.
• The Silence: There was almost no mention of "staged deployment" or "provenance" (tracking the model's history) in the reports of the biggest models.

The paper suggests that this silence means one of three things: either these safety techniques aren't being used yet, they are being used but not talked about, or they simply don't work well enough yet. The authors argue that we need to stop just releasing models and start building a rigorous science around how to keep them safe.

The Bottom Line

This paper is a call to action. It says that while open AI is amazing and brings huge benefits, we are currently flying blind when it comes to keeping it safe from bad actors. We can't just rely on the companies to be nice; we need to invent new mathematical tools that make the AI itself tough to break.

The authors are hopeful but realistic. They don't claim to have solved the problem. Instead, they have mapped out the territory, showing us exactly where the holes are in our armor. They suggest that the future of safe AI depends on openness—not just sharing the code, but sharing the research, the methods, and the failures. If we can all work together to build these new defenses, we might just be able to keep the library open and safe for everyone. But if we don't, the dangerous versions of these models could spread faster than we can stop them.