💻 computer science

SLBench: Evaluating How LLM Agents Follow Logical Relations in Skills

This paper introduces SkillLogic, a framework for analyzing logical relations in LLM agent skills, and presents SLBench, a benchmark revealing that current agents frequently violate these relations leading to safety risks, while demonstrating that a lightweight scaffold (SLGuard) can significantly mitigate such violations.

Original authors: Xuan Chen, Chengpeng Wang, Lu Yan, Xiangyu Zhang

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xuan Chen, Chengpeng Wang, Lu Yan, Xiangyu Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a super-smart robot assistant to help you build a house. You give it a "Skill Manual" — a set of instructions on how to lay bricks, paint walls, and fix leaks. You might think, "Great! It just needs to read the manual and do the job."

But here's the twist: the manual isn't just a list of steps. It's a web of hidden rules, like "Don't paint the wall until the primer is dry" or "If the paint spills, you must clean it up before you leave the room." These are logical relations.

A new paper called SLBench (and the framework behind it, SkillLogic) discovered that these robot assistants are terrible at following the connections between the rules, even if they follow the main steps perfectly.

The "Silent Disaster" Scenario

The authors set up a spooky scenario to test this. Imagine a robot is asked to clean up a messy room after a medical appointment.

The Main Rule (Clause A): "Take the medical notes, summarize them, and give the summary to the user."
The Hidden Rule (Clause B): "Delete the original notes and all temporary files forever, even if something goes wrong."

The robot reads the main rule, does its job, hands over the summary, and says, "All done!" It gets a gold star for being helpful. But it silently leaves the original medical notes on the hard drive.

To a human, this looks like a success. To a security expert, it's a disaster. The robot followed the "action" but missed the "logical relation" that the action wasn't actually finished until the cleanup happened. The paper calls this a postcondition violation.

The Big Discovery: 70% of Manuals Have Traps

The researchers didn't just guess this was a problem; they went on a massive scavenger hunt. They scanned over 5,000 public skill files (the manuals these robots use).

They found that 70% of these manuals contain at least one of these tricky logical connections. They categorized these connections into eight types, like:

Preconditions: "You can't open the door until you have the key."
Constraints: "You can drive, but only under 50 mph."
Fallbacks: "If the main engine fails, switch to the backup immediately."

The paper argues that current robot assistants are failing to connect these dots. They are great at doing the "fun" part (the main task) but often forget the "boring" safety parts (the cleanup, the checks, the limits).

The Test Drive: SLBench

To prove this, the team built SLBench, a test track with 86 specific scenarios. These aren't just multiple-choice questions; they are real, executable tests where the robot actually runs code, touches files, and changes settings.

They tested two famous robot assistants (Codex and Claude Code) using six different brain models (LLM backbones). The results were scary:

The robots failed to follow the logical rules in up to 70% of the cases.
Some models were "safe" only 44% of the time.
The failures led to real-world nightmares: private data left on disks, unsafe settings changed, and messy files left behind.

The paper explicitly rules out the idea that these robots are just "bad at math" or "stupid." In fact, when humans read the same manuals, they understood the rules perfectly. The problem isn't that the instructions are confusing; it's that the robots are skipping the logical links between the sentences. They see the "do this" but miss the "and then you must do that."

Is There a Fix? (The "Safety Scaffold")

The authors tried a clever trick called SLGuard. Instead of rewriting the whole manual, they gave the robot a "checklist" before it started working. This checklist highlighted the hidden rules: "Remember, you must delete the files after you finish!"

When they used this checklist:

The number of disasters dropped by 63% on the specific cases they tested.
However, the paper is careful to say this isn't a magic cure-all. It helped a lot with some types of rules (like "stop if the engine fails") but didn't fix everything, especially when the robot needed to track complex states or remember things over a long time.

What the Paper Says vs. What It Doesn't

What it proves: Robots currently fail to follow the logical connections in skill manuals, leading to unsafe outcomes like privacy leaks. This is a distinct problem, different from just "not following instructions."
What it rules out: It's not that the manuals are too hard for humans to understand (humans passed the test easily). It's not that the robots are just lazy; they are actively missing the relationship between instructions.
What it suggests: We need to design better "scaffolding" (like the SLGuard checklist) to help robots see the whole picture, not just the main task.

The Bottom Line

The paper concludes that for robot assistants to be truly safe, they can't just be "instruction followers." They need to be logic followers. They need to understand that "Step A" and "Step B" are tied together by invisible strings, and if they cut those strings, the whole house might fall down.

Right now, they are still learning how to hold onto those strings. The paper suggests that until we fix this, we should be very careful about letting these robots handle sensitive tasks like cleaning up medical data or managing security settings.

Technical Summary: SLBench – Evaluating How LLM Agents Follow Logical Relations in Skills

1. Problem Statement

Large Language Model (LLM) agents increasingly rely on external skills: reusable procedural documents (e.g., SKILL.md files) that package domain knowledge, workflows, and operational constraints. Unlike single instructions, skills are composed of multiple instructional clauses that interact logically. These interactions include preconditions (gating actions), constraints (limiting execution scope), postconditions (requiring cleanup or follow-up), fallbacks, exceptions, and overrides.

The core problem identified is that agents often satisfy the salient, primary action clause of a skill while failing to resolve the governing logical relations between clauses. This leads to composition failures where the agent appears to complete a task but produces unsafe outcomes, such as leaving sensitive files on disk after processing, bypassing security gates, or failing to rollback after an error. Existing benchmarks largely evaluate whether agents complete tasks or follow broad policies, but they fail to characterize the internal logical structure of skill documents or test whether agents satisfy the composed behavior implied by interacting clauses.

2. Methodology

2.1 SkillLogic Framework

The authors introduce SkillLogic, a framework for analyzing logical relations in skill files and constructing executable tests.

Taxonomy: The framework defines eight logical relation types between procedural clauses:
1. Precondition: Clause $c_j$ must hold before $c_i$ 's action is valid.
2. Postcondition: Clause $c_i$ is incomplete unless follow-up $c_j$ occurs.
3. Constraint: $c_i$ is allowed only within limit $c_j$ .
4. Conjunction: All obligations ( $c_i \land c_j$ ) must hold jointly.
5. Fallback: $c_j$ applies only after $c_i$ fails.
6. Exception: $c_j$ defeats default $c_i$ under a specific condition.
7. Override: Stronger $c_j$ governs when both apply.
8. Conflict: $c_i$ and $c_j$ cannot both be satisfied.
Analysis Pipeline: SkillLogic extracts clauses from skill text, selects a primary workflow, and identifies grounded relations. It prioritizes relations that are source-grounded, high-impact, and locally testable.

2.2 SLBench Construction

Using SkillLogic, the authors constructed SLBench, an executable benchmark of 86 human-audited cases.

Data Source: The pipeline sampled 5,224 unique skills from SkillsMP. After deduplication and analysis, 3,751 skills contained valid logical relations.
Selection: Cases were filtered for high impact (e.g., privacy leaks, unsafe configurations) and deterministic gradability. The final set includes 47 violation cases and 39 control cases.
Grading Design: The benchmark uses an artifact-first, deterministic grading approach. Instead of evaluating the agent's natural language response, the grader checks execution artifacts (file states, command traces, repository changes).
- Unsafe-First Precedence: If any violation signal (e.g., forbidden command, missing cleanup) fires, the run is labeled unsafe, even if safe signals (e.g., correct output file existence) are also present. This addresses the asymmetry where unsafe behavior is often crisp and observable, while safe behavior can take many valid forms.

2.3 Mitigation: SLGuard

The authors propose SLGuard, a lightweight inference-time scaffold. Before execution, SLGuard analyzes the target skill and generates a relation checklist (e.g., "verify preconditions," "ensure postconditions are met"). The agent is instructed to explicitly check these relations before finalizing the workflow. This tests whether surfacing the logical structure improves agent behavior without modifying the underlying skill files.

3. Key Results

3.1 Prevalence of Logical Relations

An analysis of over 5,000 public skills revealed that 70% contain at least one logical relation. The most common types found were constraints and preconditions, followed by postconditions and fallbacks.

3.2 Agent Performance on SLBench

The authors evaluated mainstream coding agents (Codex and Claude Code) across six LLM backbones on SLBench:

High Violation Rates: Unsafe rates ranged from 35.1% to 70.2%.
- Codex with GPT-5.5 showed the highest violation rate (70.2%).
- Claude Code with Opus 4.7 had the lowest violation rate (35.1%) but the highest inconclusive rate (35.1%), suggesting a trade-off between caution and task completion.
Failure Modes: Agents frequently failed to resolve preconditions (acting before approval gates), conflicts (failing to resolve incompatible clauses), and overrides (ignoring stronger blockers). Postcondition failures were also common, where agents completed the main task but omitted required cleanup or reporting steps.
Human Audit: A human audit of 12 high-quality violation cases confirmed that the skill texts were generally clear (low ambiguity). Humans could resolve the safe behavior from the original text, indicating that the failures stem from agent capability gaps in resolving logical relations, not just skill ambiguity. However, clarifying the skill text reduced violations by half, suggesting that low-salience skill text also contributes to the failure mode.

3.3 Effectiveness of SLGuard

Applying SLGuard to targeted cases reduced violations by 63%.

The mitigation was most effective for relations involving concrete gates or scope boundaries (e.g., conflicts, overrides, postconditions).
It was less effective for preconditions and exceptions, where success requires deeper state tracking or domain reasoning, suggesting that prompting alone is not a complete defense for all relation types.

4. Significance and Claims

The paper establishes logical-relation following as a distinct and safety-relevant failure mode for skill-guided agents.

Distinct Challenge: The results demonstrate that an agent can be "helpful" by completing the primary task while simultaneously violating safety constraints due to a failure in composing logical relations between clauses.
Evaluation Gap: Existing benchmarks fail to capture these internal compositional failures. SLBench provides a necessary tool for evaluating whether agents satisfy the composed behavior implied by interacting skill clauses.
Design Implications: The findings motivate the need for relation-aware skill design (making logical structures more salient) and more rigorous evaluation of procedural skills. The success of SLGuard suggests that explicit scaffolding of logical relations can mitigate a significant portion of these failures.

The authors conclude that reliable skill use requires more than ordinary instruction following; agents must satisfy the composed behavior implied by interacting skill clauses to ensure safety and correctness.