UNSPECIFIC: General Constraint Synthesis for Breaking Copy-and-Paste Shortcut in LLM Instruction Following

Imagine you are teaching a robot to write stories. You give it a list of rules, like "The hero must be brave" or "The story must end with a rainstorm." This is called instruction following. In the world of Artificial Intelligence, Large Language Models (LLMs) are these robots, and researchers are constantly trying to see if they can follow complex, long lists of rules without getting confused or making things up.

But here's the tricky part: how do you test if the robot is actually thinking about the rules, or if it's just taking a shortcut? Sometimes, the test itself is flawed. If you ask a robot to write a story based on a specific example it just read, it might just copy-paste the example's details into its own story. It's like a student who, instead of writing an essay about "a time you helped a friend," just copies a paragraph from a friend's homework that says, "I helped my friend carry groceries." The teacher sees the words "helped" and "friend" and thinks, "Great job!" but the student didn't actually do the work. This paper is all about fixing that shortcut so we can see if the robots are truly smart or just really good at copying.

---

The Great Copy-Paste Shortcut

Meet UNSPECIFIC, a new framework designed to catch AI models when they try to take the easy way out. The researchers from the University of Massachusetts, Amherst, noticed a major problem with how we currently test AI. Usually, to create a test, we take a real article (like a news story) and ask an AI to turn it into a list of instructions. Then, we ask a different AI to write a new story following those instructions.

The problem? The first AI often gets simplified. Instead of thinking of general rules, it just grabs specific details from the original article. For example, if the original story is about "John and Mary fighting over yogurt at Walmart," the instructions might say, "The characters must be named John and Mary" and "They must fight at Walmart." When the second AI gets these instructions, it doesn't need to be creative; it just copies "John," "Mary," and "Walmart" right into its story. It's a "copy-and-paste shortcut" that makes the AI look like it's following instructions perfectly, when it's really just mimicking.

How UNSPECIFIC Identifies the Shortcuts

To stop this, the UNSPECIFIC team came up with a three-step magic trick to make the tests fair and tough.

Step 1: The "Common Ground" Game
Instead of using just one story to make the rules, the researchers feed the AI two similar stories. Imagine you have two different stories about people having arguments. One is about John and Mary at a grocery store; the other is about Chris and Julie at a gas station. If you ask the AI to find rules that apply to both stories, it can't say "The characters must be John and Mary" because that only fits the first story. Instead, it has to come up with a general rule like, "The main characters must have an argument." This forces the rules to be broader and more natural, like something a real human would ask for, rather than a specific detail from a single text.

Step 2: The "Too Easy" Filter
Even with general rules, some are still too easy. Maybe the rule is "The story should have a beginning." Well, every story has a beginning! The AI satisfies this without even trying. UNSPECIFIC checks this by having the AI write a "base story" before it even sees the rules. If the AI accidentally satisfies a rule just by writing a normal story, that rule is flagged as "too easy." The system then swaps that easy rule for a harder one, like "The story must start with a character waking up to a strange noise," forcing the AI to actually think.

Step 3: The "Summary Test"
This is the most clever part. After the AI writes its story, the researchers ask it to summarize the story into a short paragraph—about 25% of the original length. If the AI only satisfied the rules by tacking on random sentences at the end (like "And by the way, the story has a happy ending"), those details get cut out in the summary. If the rule is still satisfied in the summary, it means the AI actually wove the rule into the core of the story. If the rule disappears, the AI was just taking a shortcut with superficial details.

What They Found

When they put this new system to the test, the results were eye-opening. They found that many AI models were indeed taking the copy-paste shortcut.

• The Difficulty Jump: When using the new UNSPECIFIC method, the satisfaction rate for a powerful model called GPT-5 Mini dropped from 89.7% (under the standard single-article baseline) to 78.2%. This doesn't mean the AI got worse; it means the test finally caught it when it wasn't trying hard enough.
• The Copy-Paste Gap: The researchers discovered that a huge chunk of the "success" in previous tests was not fully accurate. When they looked at the summaries, many models that seemed to follow instructions perfectly actually failed the "core narrative" test. They had satisfied the rules only on the surface, not in the heart of the story.
• Naturalness Wins: The new constraints felt much more like real human requests. In fact, the gap between how humans rated the naturalness of these new instructions versus the old ones improved by 30%.

The paper suggests that simply making instructions harder isn't enough; we have to make sure the instructions are general enough that the AI can't just copy-paste the answer. By using the "two-story" method and the "summary test," UNSPECIFIC provides a much clearer picture of whether an AI is truly understanding instructions or just playing a game of "find the matching words."

In the end, UNSPECIFIC isn't just about making AI work harder; it's about making sure we know how they are working. It stops the models from taking the shortcut and forces them to actually do the writing, ensuring that when we say an AI is "following instructions," it really means it understands what it's doing.