Choosing Where and How to Moderate: End-to-End Trade-offs in Filter Placement and Response Rewriting

Imagine you are the editor of a massive, chaotic newspaper that prints millions of stories every second. Some stories are brilliant, funny, and helpful. Others are dangerous, mean, or just plain nonsense. Your job is to make sure the final copy handed to the reader is safe and useful. But here's the tricky part: you have a team of "Safety Guards" (computer programs) who can spot the bad stuff. The big question isn't just if the guards can find the bad stuff; it's when and how they should do it.

Should the guards check the reader's request before the story is even written? Should they wait until the story is finished and then check it? Or should they do both? And if they find a problem, should they just throw the whole story in the trash, or try to fix it and give the reader a safe version? This paper dives into that exact puzzle for AI chatbots. It treats the AI like a writer and the safety filters like editors, testing different ways to arrange them to see which setup gives the best result: the most helpful answers with the fewest dangerous slips.

---

The Great AI Safety Dance: Where to Stand and What to Do

The authors of this paper, a team from Microsoft Responsible AI and Goodfire, decided to stop looking at safety filters in isolation. Usually, scientists just measure how good a filter is at spotting bad words, like a spellchecker counting typos. But the real world is messier. If a filter is too strict, it might block a perfectly good story just because it used a word that sounds risky. If it's too loose, a dangerous story might slip through.

So, the team set up a massive experiment to test four different "dance moves" for their safety system. They wanted to find the sweet spot where the AI is most helpful (showing you a great answer) but still safe (not showing you anything harmful).

The Four Dance Moves

They tested four specific setups, which they call configurations:

1. Input Only: The guard checks the reader's question before the AI writes anything. If the question looks risky, the guard stops the whole thing immediately. No story gets written.
2. Response Only: The guard lets the AI write the story first. Then, the guard checks the finished story. If it's bad, the story gets thrown away.
3. Input + Response: The guard checks both the question and the answer. If either one looks risky, the story gets blocked.
4. Response + Rewrite: This is the fancy new move. The guard checks the finished story. If it's risky, instead of just throwing it away, a "fixer" (a second AI) tries to rewrite the story to make it safe. Then, the guard checks the new version one last time. If it passes, you get the fixed story!

The Big Discovery: Wait Until the End!

The team ran these tests on two different stages: a private, human-labeled dataset of 1,250 conversations (the "Internal Benchmark") and a public dataset of over 5,000 toxic chats (the "Public ToxicChat").

Here is the surprising result: Waiting until the end (Response Only) was the best strategy for keeping things helpful.

When they just blocked bad answers after they were written, the system kept 85.68% of the conversations helpful and on-topic. But when they tried to block bad questions before the AI wrote anything (Input Only), they accidentally threw away half of the good conversations! It turns out that modern AI models are already pretty good at ignoring bad questions and writing safe answers on their own. So, stopping the process early was like a bouncer kicking people out of a club before they even got a chance to show their ID, even though many of them were just fine.

However, there was a catch. While "Response Only" was the most helpful, it let slightly more harmful content through compared to checking both the input and the output. If you have a super strict safety budget (meaning you can't let any bad stuff through), checking both the input and the output (Input + Response) was the safest, but it made the system much less helpful.

The Magic of the "Fixer"

The team then asked: "Can we have the best of both worlds? Can we keep the 'Response Only' strategy for its helpfulness, but fix the few bad answers that slip through?"

They tried the Response + Rewrite strategy. When the guard caught a bad answer, instead of blocking it, they sent it to a "fixer" AI. This fixer would rewrite the answer to remove the bad parts while keeping the good parts.

The results were impressive. By using this rewrite trick, they recovered almost all the traffic that would have been blocked.

• On the internal test, the helpfulness jumped from 85.68% to 95.04%.
• The number of blocked conversations dropped from 9.60% down to just 0.24%.

Crucially, the number of harmful answers that actually reached the user stayed exactly the same as the "Response Only" strategy. The rewrite didn't let more bad stuff through; it just saved the good stuff that was almost getting thrown away.

Speed and the "Rewrite" Cost

Of course, nothing is free. Rewriting takes time. The team measured how long it took to fix a story.

• If they used a giant, slow AI to decide what to rewrite and how, it took about 13.8 seconds. That's an eternity in chat time!
• But, they found a clever shortcut using smaller, specialized "probes" (tiny, fast detectors) to decide what to do. This cut the time down to just 0.47 seconds.

This means you can have a system that is super helpful and safe without making the user wait forever.

The Fine Print: What the Rewrite Missed

The authors didn't just look at the numbers; they read the rewritten stories to see what was actually happening. They found that the "fixer" was good at generalizing. For example, if a story mentioned a specific dangerous app, the fixer would replace the name with "a safe platform" and still give good advice on how to stay safe.

However, they also found a boundary. In some sensitive cases, like stories about self-harm, the rewrite sometimes removed specific, helpful resources (like crisis hotline numbers) just to be extra safe. The paper notes that while the system is great at balancing safety and helpfulness, it's not perfect. Sometimes, in the rush to be safe, it might accidentally leave out a piece of support information that a human would want to keep.

The Bottom Line

This paper doesn't tell us there is one single "perfect" rule for all AI safety. Instead, it gives us a map. It shows that:

1. Don't block too early: Letting the AI write first usually results in more helpful answers.
2. Rewrite, don't just block: If you catch a bad answer, try to fix it. It saves a lot of good conversations without increasing the danger.
3. Speed matters: You can fix things quickly if you use the right tools.

The authors conclude that there is no universal "one-size-fits-all" rule. Instead, companies need to look at their own specific needs. If they can tolerate a tiny bit of risk to get more helpful answers, they should use the "Response Only" strategy with a "Rewrite" fixer. If they need to be absolutely zero-risk, they might need to check the input too, even if it means blocking more good conversations. It's all about finding the right balance for your own club.