Technical Summary: Harness-IF

Problem Statement

Existing instruction-following (IF) benchmarks and coding-agent evaluations suffer from a fundamental misalignment in how they measure compliance. Traditional IF benchmarks (e.g., IFEval, FollowBench) concentrate rules within user prompts, failing to capture the multi-surface nature of deployed agents. Conversely, coding-agent benchmarks (e.g., SWE-bench) prioritize final task success, obscuring whether an agent followed specific operational rules or simply arrived at the correct outcome through default behavior.

Consequently, current metrics cannot distinguish between compliance (following a specific instruction) and coincidence (achieving a result that happens to align with the instruction due to the model's unprompted defaults). Furthermore, the "instruction stack" of a coding agent—comprising system prompts, tool descriptions, skill definitions, project files, and user instructions—lacks a standardized evaluation framework to determine which surface takes precedence when rules conflict or how agents adhere to rules distributed across these surfaces.

Methodology

Harness-IF introduces a benchmark designed to measure instruction following at the level of individual rules and delivery surfaces. The methodology is built on three core components:

1. Instruction Surfaces and Controlled Relocation

The framework defines six instruction surfaces:

• Harness Default (HD): Platform-inserted defaults.
• System Prompt (SP): Developer-defined role/behavior.
• Tool Description (TD): Tool usage instructions.
• Skill Description (SD): Skill usage rules.
• Project File (PF): Repository-level docs (e.g., CLAUDE.md).
• User Instruction (UI): The user's request.

Harness-IF employs a controlled-relocation design where the semantic meaning of a rule remains fixed while its delivery surface is varied. This allows for matched comparisons of how an agent follows the same rule when presented in different contexts (e.g., as a system prompt vs. a project file).

2. Data Construction

• Library: A curated library of 642 atomic constraints across seven families (professional writing, output control, code style, workflow, quantitative limits, conditional logic, tool use).
• Items: 60 realistic multi-turn coding items were assembled from this library. Each item injects 25–35 rules, instantiated across the five configurable surfaces (SP, TD, SD, PF, UI).
• Execution: The benchmark was run across 12 frontier models for 3 rounds, generating 2,160 evaluation runs.
• Verdicts: Instead of a single task-level pass/fail, the system generates a verdict (pass, fail, or not applicable) for every applicable rule in every run, resulting in 256 distinct rules receiving execution-grounded verdicts.

3. Metrics: Against-Prior Accuracy (AP-Acc)

To separate compliance from coincidence, the authors introduce Against-Prior Accuracy (AP-Acc).

• Prior Control: Rules are labeled as "against-prior" if they oppose the model's observed unprompted default behavior. This is determined via a zero-injection probe (re-running tasks without the rule to observe defaults) and curated annotations.
• Calculation: AP-Acc scores only the subset of rules labeled as opposing the default.
• Comparison: This is contrasted with standard Accuracy (Acc), Filtered Accuracy (F-Acc), and Discrimination-Weighted Accuracy (DW-Acc).

Key Results

1. The "Compliance Gap"

Across all 12 evaluated models, performance on against-prior rules is consistently lower than aggregate accuracy.

• Aggregate Accuracy: Ranges from 72.1% to 85.9%.
• AP-Acc: Ranges from 66.1% to 78.6%.
• The Gap: Every model performs worse on against-prior rules by 3.6 to 7.4 percentage points (mean gap: 5.81).
• Implication: Aggregate scores systematically overstate compliance. The degree of overstatement varies by model (a twofold spread), meaning prior control can alter model rankings (exchanging three adjacent rank pairs) even if the top model remains unchanged.

2. Failure Signatures

• Action vs. Inaction: Failures are heavily skewed toward shortfall rules (omissions of demanded behavior), which account for 77.1% of failures. "Overstep" rules (forbidden actions) account for only 20.8%.
• Difficulty by Modality: Commanding rules (require/forbid) are the most difficult (76.0% accuracy), while Preference rules are the easiest (90.6%).
• Difficulty by Family: Output control is the most difficult family (70.9% accuracy), followed by workflow and code style.

3. Instruction Surface Precedence (E0 Pilot)

A controlled conflict pilot (E0) tested which instruction surface prevails when rules conflict.

• Finding: Precedence does not follow simple "prompt depth" (i.e., the user instruction being last does not guarantee it wins).
• Hierarchy: System prompts (SP), project files (PF), and user instructions (UI) tied for the highest precedence. Tool descriptions (TD) and skill descriptions (SD) were significantly lower.
• Stability: This ordering was robust across bootstrap resampling and model variations, suggesting a pooled tendency rather than a universal hierarchy.

Significance and Claims

Harness-IF claims to provide the first unified framework for measuring operational instruction following in coding agents at the rule level. Its primary contributions are:

1. Rule-Level Measurement: It shifts the evaluation unit from "task success" to "rule compliance," enabling the detection of specific failures that aggregate scores obscure.
2. Decoupling Compliance from Coincidence: By introducing AP-Acc, the benchmark isolates true instruction following from behavior that merely aligns with model defaults. The paper argues that without this control, benchmark scores are inflated by a model-specific margin.
3. Surface-Aware Evaluation: It demonstrates that instruction delivery surface matters. The framework reveals that agents do not treat all instructions equally and that the "instruction stack" has a non-trivial precedence structure that differs from simple prompt ordering.
4. Diagnostic Granularity: The benchmark exposes distinct failure signatures across rule families (e.g., output control is a primary failure point) and modalities, offering actionable insights for improving agent reliability.

The authors conclude that coding-agent evaluation must treat instruction compliance as an execution-level object, preserving rule provenance and separating against-prior behavior from aggregate success to accurately assess agent capabilities.