Technical Summary: Trajectories That Segment Themselves

1. Problem Statement

Long-horizon coding-agent trajectories present a fundamental mismatch with standard training units. Current approaches rely on three flawed credit assignment mechanisms:

• Per-action credit: Individual actions (e.g., grep, pytest) lack intrinsic value; their utility depends entirely on the causal hypothesis they serve.
• Terminal episode labels: A single success/failure bit conflates productive exploration with abandoned directions, merging well-grounded phases with failed commitments.
• Fixed windows: Arbitrary time-step boundaries (e.g., $K$-turn windows) often split coherent investigations or merge the abandonment of one cause with the adoption of another.

These boundaries are derived from logging mechanics rather than the agent's evolving epistemic state, resulting in noisy targets that reward surface form over causal reasoning.

2. Methodology

2.1 Agent-Declared Semantic Self-Segmentation

The paper introduces a collection-time protocol where the acting agent exposes its own boundaries via a declarative contract.

• The Mechanism: During a tool-using rollout, the agent adopts a falsifiable causal hypothesis ($h_k$) and declares it. The agent then gathers evidence, edits code, or runs checks under this hypothesis. The phase ends when the agent adopts a new hypothesis ($h_{k+1}$) or the trajectory terminates.
• The Cost: This adds a minimal overhead (mean 52 tokens per declaration, ~9.8% of agent output) compared to alternatives requiring gold patches, environment-specific vocabularies, replayable environments, or post-hoc segmenters.
• The Collection Protocol: An agent attempts a task. If it fails hidden tests, a reviewer model intervenes. The reviewer sees the reference fix but is forbidden from naming it. Instead, the reviewer must provide: (1) a negation of the current wrong approach and (2) the smallest positive direction to investigate next. This forces the agent to abandon the refuted hypothesis and adopt a new one, creating a clear "wrong-cause-then-correction" transition.

2.2 Four Supervised Targets from One Collection

A single rollout yields four distinct training views, leveraging the asymmetry of information:

1. Audit (2,721 rows): The auditor sees the completed phase and bound evidence to judge if the direction was sound and identify missing evidence. This recovers lessons from spans an episode label would discard.
2. Propose (1,405 rows): The model sees only the state at the boundary (masked) and must derive a checkable next hypothesis or state missing evidence.
3. Fix (553 rows): The model sees an outcome-confirmed cause and source to turn a localized diagnosis into a patch.
4. Preference Pairs (2,551 rows): The model sees the masked boundary state and learns to prefer an advancing opening over a stalling or refuted one.

2.3 Phase-DPO

The paper implements a standard Direct Preference Optimization (DPO) on the preference pairs. The objective is to learn a policy $\pi_\theta$ such that $\pi_\theta(y^+ | x) > \pi_\theta(y^- | x)$, where $y^+$ is a preferred opening hypothesis and $y^-$ is a rejected one, given the masked state $x$. The phase serves as the credit unit, fixing where the context is cut and which text is the target.

3. Key Contributions

1. Dependency-Free Boundaries: The segmentation is supplied by the acting policy itself under a declarative contract, requiring no external gold patches, milestone vocabularies, or post-hoc segmenters. The boundary arrives within the same rollout that produces the trajectory.
2. Addressable Units: Unlike windows defined by position, a declared hypothesis is an addressable handle. This enables finer-grained credit assignment, process supervision without step annotation, and guidance by name (negating specific conjectures rather than whole trajectories).
3. Semantic Coherence: The boundaries track the agent's reasoning, not logging artifacts.
   • Attribution: When declarations are scrubbed, an independent model attributes action blocks to their governing hypothesis at 2.18x chance (paired sign test $p=0.0002$), outperforming equal-length windows.
   • Non-Derivability: A code-blind human annotator matches 24/40 boundaries (vs. 11.5 expected by chance), while a mechanical test-event rule fails to beat chance at either strict or permissive thresholds.
4. Construction Interface: The protocol manufactures "wrong-cause-then-correction" transitions rarely found in recorded work, yielding audit supervision from failed regions that standard episode labels discard.

4. Results

4.1 Unit Validity

• Attribution: Action blocks remain attributable to their governing hypothesis even after scrubbing the declaration text, confirming the boundary is semantic, not lexical.
• Recoverability: A code-blind reader can recover boundaries significantly better than random, though they over-segment (1.55x), indicating the boundaries are not trivially reproducible by simple heuristics.

4.2 Downstream Optimization (Phase-DPO)

• Orientation Sensitivity: Training on 2,551 pairs with a 27B SFT policy produces a label-inverted arm that does not reproduce the separation, confirming the signal is not an artifact of the optimizer shell.
• Generalization Limits:
  • On adversarially built held-out items (87 instances), the trained adapter changed 0/91 decisions.
  • On matched-construction items (60 items from the same generator family), the adapter changed 4/60 decisions (all wrong-to-right), while controls changed none.
  • Conclusion: The fitted preference is real but construction-bound; it learns how the pairs were written rather than a generalizable capability.

4.3 Rollout Pilot

A 20-task pilot on SWE-bench Verified showed:

• Resolve Rate: Base (13/20), SFT (14/20), Phase-DPO (14/20). No statistically significant improvement in solve rate.
• Efficiency: Phase-DPO showed a tendency toward shorter trajectories (14.0% reduction in messages vs. base), but confidence intervals included zero.
• Case Studies: Specific cases showed DPO making better local choices (e.g., preserving internal defaults) or worse ones (committing to wrong loci early), highlighting that better phase selection does not guarantee global success.

5. Significance and Claims

The paper modestly claims that the collection-time segmentation protocol and the data interface are the primary contributions.

• What is established: A semantic boundary can be obtained cheaply without external dependencies. This boundary creates a coherent unit where actions are attributable to a governing hypothesis, allowing for the extraction of lessons from failed search directions (audit supervision) and the construction of preference pairs.
• What is not established: The paper does not claim that training on these units results in a policy with superior end-to-end coding capabilities or higher resolve rates. The optimization signal is orientation-sensitive but currently bound to the construction of the training pairs.
• Future Direction: The paper posits that the next variable to vary is the diversity of the pair corpus, not the boundary definition itself. The boundary is a necessary condition for the unit, but the unit's utility depends on the quality and variety of the data it structures.

The work reframes the "credit unit" problem: by letting the agent declare its own causal commitments, the system manufactures training data that distinguishes between search directions rather than just outcomes, offering a path to more granular and interpretable agent training.