Contradiction-Aware Strategy Synthesis in Agent Skills via Loop Engineering: A Controlled Empirical Comparison with Conventional Search

Technical Summary: Contradiction-Aware Strategy Synthesis in Agent Skills via Loop Engineering

Problem Statement
The paper addresses two critical failure modes in current study-abroad and scholarship advising: (1) Information Overload, where conventional web searches return unstructured, unevenly trustworthy lists of links that fail to synthesize an applicant's specific profile; and (2) Invalid Premises, where applicants pursue structurally impossible goals (e.g., a fully-funded, English-medium medical degree in Austria) without the search engine flagging the contradiction. Conventional search engines are optimized for source relevance, not decision quality, and lack mechanisms to validate goal feasibility against country-specific constraints. While "agent skills" (encapsulated workflows) claim to solve this, rigorous, input-controlled empirical comparisons against conventional baselines have been absent.

Methodology
The study employs a strict controlled A/B testing protocol across ten experiments (nine real applicant profiles across nine countries/domains and one sparse-profile robustness test).

• Controlled Inputs: Both the "Skill" arm and the "Normal Web Search" arm receive the identical natural-language prompt describing the applicant and their goal. The Skill arm additionally ingests a structured JSON profile encoding the same facts. This design isolates the value of the synthesis loop from any advantage in information access.
• Loop Engineering Framework: The Skill executes a five-stage iterative refinement process:
  • L1 (Retrieval & Structuring): Collects applicant and country facts.
  • L2 (Contradiction & Constraint Detection): Tests the stated goal against reality to detect impossibilities or conditions.
  • L3 (Quantification & Benchmarking): Assigns costs, acceptance rates, net funding gaps, and scores.
  • L4 (Strategy Synthesis & Actionability): Ranks options and constructs a timeline.
  • L5 (Verification): Programmatic reconciliation of numbers and consistency checks.
• Metrics: The study measures output-dimension coverage across seven decision-critical classes: ranked funding combinations, quantified net gaps, institution comparisons, named awards, charts, timelines, and goal-validity verdicts. An ablation study quantifies the marginal contribution of each loop.

Key Contributions

1. Controlled Comparison Protocol: The first multi-domain, input-controlled study comparing a structured agent skill against a conventional web search, eliminating the "better input" confound.
2. Empirical Corpus: A dataset of 20 artifacts (10 Skill reports vs. 10 Baseline searches) spanning diverse funding regimes (tuition-free, high-tuition, stipend-default).
3. Loop-Level Ablation: Formalization of the advising process into loops, demonstrating that Loops L2–L4 unlock approximately 69% of decision-grade artifacts, proving that iteration depth, not mere data access, drives value.
4. Programmatic Verification: An L5 pass that machine-verifies all generated reports, confirming structural claims (page counts, chart presence, verdict inclusion) rather than asserting them.

Results
The results demonstrate a categorical superiority of the Skill over the baseline:

• Decision-Critical Coverage: The Skill achieves 100% coverage on all seven measured dimensions across all ten experiments. The baseline achieves ~10% coverage (limited to scattered award names) and 0% on all synthesis-dependent dimensions (ranked combinations, quantified gaps, timelines, verdicts).
• Contradiction Detection (The Decisive Test): The Skill correctly classifies goal validity (Impossible, Conditional, Selective, or Consistent) in 100% of cases. The baseline achieves 0%, failing to flag structurally impossible goals (e.g., Experiment A: English-medical in Austria) and instead returning generic links that sustain the false premise.
• Artifact Generation: The Skill produces a 16-page, chart-rich PDF with 10 ranked funding combinations, a 12-month timeline, and an "Honest Verdict." The baseline returns an unstructured list of ~10 links with no synthesis.
• Robustness: In the sparse-profile experiment (Experiment C), the Skill gracefully degrades to a structured skeleton with placeholders and a "supply data" verdict, whereas the baseline returns the same generic links as for rich inputs.

Significance and Claims
The paper claims that for high-stakes, contradiction-prone study-abroad goals, the structured agent skill strictly dominates conventional search.

• Value Creation: The primary value is not in retrieving information (which both arms access) but in the synthesis labor internalized by the Skill (Loops L2–L4). This transforms raw data into an actionable plan, removing the burden of assembly from the applicant.
• Ethical Implication: The Skill's ability to detect and explicitly state "impossible" goals prevents the financial and temporal harm caused by the baseline's "confidently incomplete" output.
• Categorical Gap: The "100% relative advantage" is not a rhetorical figure but a direct observation of the baseline's inability to produce decision-grade artifacts (plans, gaps, verdicts) compared to the Skill's consistent production of them.
• Future Direction: The authors propose that future scholarship tools should adopt the seven-dimension coverage score as a standard metric and extend the loop-engineering framework to other high-stakes domains like immigration and medical triage.

The paper concludes that the Skill should be the default first step for applicants, with conventional search serving only as an L1 input source, as the Skill provides a "corrected plan" versus the baseline's "wasted cycle."