Technical Summary: CHILL-Harness

Problem Statement

Modern Large Language Model (LLM) agents increasingly tackle long-horizon tasks requiring sustained context management, tool interaction, verification, and execution control. To translate latent model capabilities into reliable behavior, agent harnesses serve as the operational infrastructure coordinating these elements. However, current harnesses predominantly rely on hand-crafted rules, fixed thresholds, or globally configured workflows. This static approach creates a mismatch with the dynamic nature of task demands, execution environments, and evolving execution states. Consequently, existing systems often suffer from:

• Unnecessary computational overhead: Redundant reasoning and delayed corrections.
• Reduced task success: Workflow disruptions that degrade performance in adverse cases.
• Lack of adaptability: Existing methods either rely on domain-specific engineering or optimize collaboration structures without determining whether a specific orchestration intervention causally improves the current workflow.

The core challenge is to formulate a framework that enables adaptive orchestration in harness systems, assessing whether a concrete adjustment improves the ongoing workflow while preserving task success.

Methodology: CHILL-Harness

The authors propose CHILL-Harness (Counterfactual Harness Intervention Learning for Long-Horizon Agents), a causal framework that treats adaptive harness orchestration as a causal learning problem. It distinguishes between the "factual" workflow (the agent's current plan) and "counterfactual" alternatives (admissible interventions). The framework consists of two coupled components:

1. Causal Intervention Effect Learning (CIEL)

CIEL serves as the effect-estimation component. It unifies heterogeneous orchestration behaviors (e.g., additional reasoning, workflow correction, execution recovery) into a unified intervention space.

• Unified Induction: It maps interventions to admissible workflows, categorizing them into actions like Keep, Deliberate, Revise (inspect/verify/visual), Stabilize, and AnswerSynthesis.
• Effect Estimation: CIEL estimates the context-conditioned intervention-relative advantage ($\Gamma_t$) of a workflow alternative over the factual workflow using confidence-weighted execution evidence from offline paired records.
• Factorized Decision Learning: To manage complexity, CIEL decomposes control into three specific decision heads:
  • Counterfactual Deliberation Effect (CDE): Predicts the required deliberation route (Skip, Light, Full) to avoid over-deliberation when the factual workflow is sufficient.
  • Counterfactual Revision Effect (CRE): Determines if a candidate workflow provides sufficient support to replace the factual one, preventing harmful replacements.
  • Completion Attribution Effect (CAE): Predicts whether to terminate execution based on evidence, preventing premature termination or futile continuation.
• Training: The system is trained using a composite objective ($L_{CIEL}$) that combines workflow-effect estimation loss with weak supervision from logged execution traces for the decision heads. Crucially, efficiency-related decisions receive positive supervision only when task success is preserved.

2. Advantage-Realizing Causal Orchestration (ARCO)

ARCO is the realization component that translates CIEL's estimates into actionable decisions.

• Route Selection: Based on the CDE prediction, ARCO selects a deliberation route (e.g., Skip bypasses candidate generation; Full enables broader evaluation).
• Candidate Valuation: For admissible candidates exposed by the route, ARCO calculates a combined score of operational utility (progress, cost, risk) and the estimated causal intervention effect.
• Conservative Authorization: A candidate workflow is executed only if:
  1. The CRE predictor supports the change.
  2. The estimated advantage ($\Delta_{cf}$) exceeds an intervention-dependent authorization margin ($\delta$).
• Completion: Termination is authorized only when both the CAE predictor and explicit task evidence (from a verifier) confirm completion.

Success-Preserving Objective

The framework incorporates a constrained optimization objective. It minimizes trajectory cost (tokens, latency) subject to a bounded degradation in pass rate ($\epsilon_{pass}$). This is realized through outcome-conditioned weak supervision that penalizes resource cost, task failure, and specifically "success-breaking" adaptations (where a reference success is turned into a failure).

Key Contributions

1. Causal Formulation: The paper formulates adaptive harness orchestration as a causal learning problem over factual and admissible alternative workflows, moving beyond fixed or globally optimized policies.
2. CIEL Framework: Introduction of Causal Intervention Effect Learning, which unifies heterogeneous behaviors and learns context-conditioned intervention advantages from confidence-weighted evidence, enabling selective intervention.
3. ARCO Mechanism: Development of Advantage-Realizing Causal Orchestration, which realizes workflow adaptations only when supported by sufficient estimated advantage, ensuring efficient and reliable execution under success-preserving constraints.
4. Empirical Validation: Extensive experiments demonstrating that the framework preserves or improves task success while significantly reducing resource consumption.

Experimental Results

The authors evaluated CHILL-Harness on three heterogeneous long-horizon benchmarks: GAIA (information seeking), SWE-bench Verified (software engineering), and Terminal-Bench 2.0 (terminal interaction).

• Effectiveness: CHILL-Harness consistently preserved or improved task success rates compared to baselines. It achieved the highest success rates on GAIA (71.3%) and SWE-bench Verified (65.6%), and remained competitive on Terminal-Bench 2.0 (56.1%).
• Efficiency: The framework substantially reduced resource consumption:
  • Token Reduction: 28.4% (GAIA), 13.1% (SWE-bench), and 23.1% (Terminal-Bench) compared to the lowest-token baselines.
  • Time Reduction: 46.6% (GAIA), 25.9% (SWE-bench), and 2.6% (Terminal-Bench) compared to the fastest baselines.
  • Tokens per Solved Task: CHILL-Harness achieved the highest number of solved tasks per million tokens across all benchmarks.
• Ablation Study: An "Always-Full" variant (forcing full deliberation) resulted in significant cost increases and, in some cases, reduced success rates. This validates the necessity of the route-before-generation strategy (CDE) to suppress low-value deliberation before incurring planner overhead.

Significance and Claims

The paper claims that CHILL-Harness provides a unified framework that moves agent development beyond fixed harness engineering toward learned, causal, and success-preserving orchestration. By explicitly modeling the causal contribution of orchestration interventions, the system can distinguish genuine improvements from spurious associations, thereby enabling efficient reasoning without compromising reliability. The authors posit that this approach offers a foundation for building more efficient and reliable long-horizon agents that adapt dynamically to execution states rather than relying on static, hand-crafted policies.