SCHEDBench: A Benchmark for Evaluating LLM Constraint Faithfulness in Natural-Language Combinatorial Scheduling
This paper introduces SCHEDBench, a comprehensive natural-language benchmark demonstrating that large language models fail to maintain reliable constraint faithfulness and feasibility across semantically equivalent surface-form variations in diverse combinatorial scheduling tasks.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are the conductor of a massive, chaotic orchestra. You have hundreds of musicians, a limited number of instruments, and a strict rulebook: the violinist can't play until the drummer finishes a solo, and the trumpet section needs a break every twenty minutes. Your job is to write a schedule that tells everyone exactly when to play so the concert happens without a single clash. This is the world of combinatorial scheduling. It's a branch of math and computer science dedicated to organizing complex tasks where resources are scarce and rules are tight. If you get the timing wrong, the whole show falls apart.
For years, scientists have been teaching computers to solve these puzzles using strict, mathematical code. But recently, a new kind of computer brain called a Large Language Model (LLM) has arrived. These are the same AI systems that write essays, chat with you, and tell jokes. They are amazing at understanding human language. But here's the big question: Can an AI that is great at chatting also be great at following a strict, logical rulebook to solve a scheduling puzzle? And more importantly, if you ask the AI the same question but phrase it in a slightly different way—like swapping "violinist" for "musician" or changing the order of the rules—will it still get the right answer? This paper dives into that exact mystery, testing whether these AI brains are truly logical or if they just get confused by how the question is dressed up.
The Paper's Big Test: SCHEDBench
The researchers behind this study, Shrenil Shaun Sharma and Avi Sharma, built a giant testing ground they call SCHEDBench. Think of it as a "driving test" for AI, but instead of driving a car, the AI has to drive a complex schedule. They didn't just make up random problems; they took 1,132 real-world scheduling puzzles from famous libraries used by engineers and mathematicians. These puzzles cover everything from managing factory jobs and hospital nurse shifts to organizing university class timetables.
To make it a true test of "language understanding," they didn't feed the AI raw numbers. Instead, they translated every single puzzle into natural English sentences. They then created different versions of the exact same puzzle. In one version, the rules might be listed in order from A to Z. In another, the rules might be shuffled. In a third, the "jobs" might be called "batches" instead of "orders," or the "machines" might be renamed "workstations." The math and the logic remained exactly the same, but the words changed.
They asked 13 different AI models (including big names like GPT-5, Claude, and Llama) to solve these puzzles. The goal was simple: Could the AI generate a valid schedule that followed all the rules, no matter how the question was phrased?
The Findings: The AI Gets Confused by the Dress-Up
The results were surprising. The paper found that AI models are not reliable when it comes to these scheduling puzzles. Even though the math behind the puzzles didn't change, the AI's performance dropped significantly when the wording changed.
Here is what they discovered in plain terms:
- The "Dress-Up" Effect: When the researchers changed the surface details of the problem—like reordering the list of rules or swapping out names for synonyms—the AI often failed to find a solution it could have found before. It's as if the AI is so focused on the specific words it sees that it forgets the underlying logic.
- The Order Matters (A Lot): The biggest culprit was the order of the rules. When the researchers shuffled the sequence of the constraints (e.g., listing the nurse's break time before the shift start time instead of after), the AI's ability to solve the puzzle became much worse. This suggests the AI isn't truly "understanding" the whole problem at once; it might be getting tripped up by the sequence in which it reads the instructions.
- Not Just Random Mistakes: The researchers were careful to prove this wasn't just the AI having a "bad day" or random guessing. They ran the tests multiple times with different random seeds (like rolling dice to pick the starting point). They found that the drop in performance was real and consistent, not just noise.
- The Best Performers Still Struggle: Even the most advanced models, like GPT-5.5, showed a sharp decline in performance when the puzzles were "dressed up." On a specific subset of 470 simpler puzzles, GPT-5.5 solved about 84% of them in their simplest form. However, when the rules were shuffled or the wording changed, that success rate dropped to 61.5% on that same subset. When tested on the full set of 1,132 puzzles with all variations active, the success rate for GPT-5.5 fell further to 55.9%. This shows that even the best models struggle to maintain accuracy when the presentation of the problem changes.
What This Means
The paper explicitly rules out the idea that these AI models are perfectly stable logical engines. They are not. The study shows that for complex tasks like scheduling, how you ask the question matters just as much as the question itself.
The authors suggest that while these AI models are getting better at many things, they still struggle to maintain "constraint faithfulness." In other words, they can't always promise to stick to the rules if the rules are presented in a slightly different style. The paper doesn't claim this is a permanent flaw that can never be fixed, but it does show that current models are not yet ready to be trusted with critical scheduling tasks where a small misunderstanding could lead to a massive failure.
In short, if you ask an AI to schedule a factory, and you change the order of the instructions, it might suddenly forget how to do its job. The paper concludes that we need to be very careful about how we talk to these models when the stakes are high, because they are still learning how to separate the meaning of a rule from the words used to describe it.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.