🤖 AI

Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1

This paper demonstrates that a cost-effective, non-fine-tuned open-weight model (DeepSeek V3.2) equipped with specialized agentic harnesses—specifically an Explorer-Definer Pipeline and a Reflective Orchestrator—can significantly boost ARC-AGI-1 performance from a 15.50% baseline to 67.25% pass@2 by explicitly separating pattern discovery from program synthesis and adaptively re-exploring transformations, all without heavy test-time compute or benchmark-specific training.

Original authors: Kabir Moghe, Peter Chin

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kabir Moghe, Peter Chin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a giant, complex puzzle where the rules change every single time you pick up a new piece. This is what the ARC-AGI benchmark is: a test designed to see if a computer can figure out abstract patterns and logic without having seen that specific puzzle before.

For a long time, solving these puzzles required one of two expensive approaches:

The "Brute Force" Method: Hiring a super-intelligent (and very expensive) AI to try millions of random guesses until it gets lucky. This costs a lot of money and computer power.
The "Specialist" Method: Taking a small AI and teaching it only how to solve these specific puzzles by showing it thousands of examples. This works well, but the AI becomes a specialist who can't do anything else.

The Big Idea of This Paper
The authors, Kabir Moghe and Peter Chin from Dartmouth College, asked a different question: Can we get a "generalist" AI (one that hasn't been specifically trained on these puzzles) to solve them well, using a clever team structure instead of brute force or special training?

They built a cost-effective "agent harness"—essentially a management system that organizes how the AI thinks. They tested this on a standard, open-source AI model (DeepSeek V3.2) that was not specifically trained on these puzzles.

Here is how their system works, using a simple analogy:

The Analogy: The Detective Agency

Imagine you are a detective trying to solve a crime scene (the puzzle). You have a team of agents with different jobs.

1. The Old Way (One-Shot Baseline)

You call one detective, show them the crime scene, and ask, "Who did it?" They guess immediately.

Result: They get it right only 15.5% of the time. They are guessing in the dark.

2. The "Think Before You Speak" Way (Chain-of-Thought)

You call the detective again, but this time you force them to write down their reasoning before giving the answer.

Result: Accuracy jumps to 30%. Just writing down the thought process helps.

3. The New Way: The "Explorer-Definer Pipeline"

Instead of one detective, you hire a team.

The Explorers (The Pattern Hunters): You send out 5 different agents to look at the crime scene independently. They don't try to solve it yet; they just look for clues, patterns, and weird symmetries. They write detailed reports on what they see.
The Definer (The Architect): A master architect reads all 5 reports. They take the best ideas, combine them, and write a specific "recipe" (a computer program) to solve the puzzle.
The Check: They test this recipe on the known clues. If it fails, they tweak the recipe.
Result: This team approach gets 57.5% of the puzzles right, costing only $0.25 per puzzle. They didn't need to hire a super-expensive AI or train a specialist; they just organized the work better.

4. The "Reflective Orchestrator" (The Boss with a Second Chance)

The authors noticed a problem with the Pipeline: Sometimes the Explorers miss the big picture entirely. The Architect then tries to fix the recipe, but they are stuck trying to fix a broken foundation. It's like trying to paint a masterpiece on a cracked canvas.

So, they added a Boss (The Orchestrator).

If the Architect's recipe fails, the Boss doesn't just ask for a tweak. The Boss says, "Okay, this approach is wrong. Let's send out a new, smaller team of Explorers specifically to look for the clues we missed."
This allows the system to re-explore the problem from a fresh angle when it gets stuck.
Result: This smart loop gets 67.25% of the puzzles right, for about $0.62 per puzzle.

What Did They Discover?

The paper makes a few key discoveries about why this works:

It's about "Generation," not "Selection":
The team found that the main reason they were failing wasn't that they couldn't pick the right answer from a list. It was that they weren't generating enough different types of ideas in the first place.
- Analogy: It's not that the team is bad at choosing the right key; it's that they didn't bring enough different keys to the door.
- The "Orchestrator" fixed this by forcing the team to generate new keys (new ideas) when the old ones didn't work.
The "Think" Tool is Crucial:
They tested what happens if they remove the "Think" tool (a private scratchpad where the AI can jot down notes before speaking).
- Result: Accuracy dropped by nearly 6%.
- Analogy: It's like forcing a detective to solve the case out loud without being allowed to write notes. They get confused and make mistakes. The private "thinking space" is essential for complex reasoning.
Cost vs. Performance:
They achieved these high scores without spending thousands of dollars per puzzle (like the "Brute Force" methods) and without training a new AI from scratch (like the "Specialist" methods). They did it by building a smarter workflow for an existing, affordable AI.

The Bottom Line

This paper proves that you don't always need a bigger, smarter, or more expensive AI to solve hard logic puzzles. Sometimes, you just need a better manager to organize how the AI thinks. By breaking the problem into steps (looking for patterns, then building a solution) and allowing the system to "re-think" when it gets stuck, they boosted performance from 15% to nearly 67% using a very small budget.

Note: The authors specifically tested this on a public set of 400 puzzles. They did not claim this works for medical diagnosis, self-driving cars, or other real-world applications yet; they only proved it works for this specific abstract reasoning test.

Technical Summary: Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1

Problem Statement

The paper addresses the high cost and computational intensity associated with achieving strong performance on the Abstraction and Reasoning Corpus (ARC-AGI-1). Current state-of-the-art approaches generally fall into two regimes:

Heavy Test-Time Compute: Using frontier Large Language Models (LLMs) with extensive evolutionary search, exhaustive sampling, or extended chain-of-thought reasoning, often costing tens to hundreds of dollars per task.
Benchmark-Specific Training: Fine-tuning small models or training them from scratch directly on ARC data, often combined with aggressive test-time augmentation.

The authors investigate a third, under-explored regime: achieving significant performance gains on ARC-AGI-1 using an open-weight model in non-thinking mode (DeepSeek V3.2) with no benchmark-specific fine-tuning and a strict budget (under $1 per task). The central hypothesis is that the gap between cheap one-shot LLM calls (~16% accuracy) and performant systems (≥50%) is not solely a function of model capability or scale, but can be recovered through specific architectural engineering.

Methodology

The study evaluates four architectures on the public 400-task ARC-AGI-1 evaluation set, all running on the same model (DeepSeek V3.2) with identical grid rendering and scoring protocols.

1. Baselines

One-Shot Baseline: A single LLM call with no reasoning scaffolding, returning a structured JSON output.
Structured Chain-of-Thought (CoT) Baseline: A single call requiring the model to output a natural language reasoning field before the grid, enforcing externalized deliberation.

2. The Explorer-Definer Pipeline

This architecture decomposes the task into two sequential phases mediated by compressed natural-language artifacts:

Phase 1 (Parallel Exploration): A fleet of $N$ PatternExplorer agents independently analyzes training pairs. Each agent uses a private think tool for internal deliberation and a note_pattern tool to record observations. Their outputs are compressed into a dense summary.
Phase 2 (Transformation Definition): A TransformationDefiner agent receives the training pairs and the compressed exploration summaries. It uses a think tool and generates executable Python code to implement the transformation.
Selection & Refinement: Candidates are ranked by exact-match accuracy on training pairs. A dedup-by-test-prediction mechanism merges candidates producing identical test outputs. If a candidate passes ≥50% of training pairs but fails others, the definer is re-invoked with structured failure feedback to refine the code.

3. The Reflective Orchestrator

Motivated by the finding that the pipeline is "generation-bound" (accuracy is limited by the diversity of candidate transformations, not the selection mechanism), the Orchestrator extends the Pipeline's definer with adaptive capabilities:

Mid-Loop Re-exploration: If the current transformation hypothesis fails significantly on training pairs, the Orchestrator can spawn a focused fleet of $K=2$ new PatternExplorer agents.
Negative-Leaning Guidance: These new explorers are conditioned on a structured summary of "dead ends" and ruled-out rule families, steering them toward unexamined regions of the transformation space.
Termination: The loop exits upon reaching 100% training accuracy, an agent-mediated done call, or hitting safety fuses (max iterations, consecutive errors, or spawn limits).

Key Contributions

New Cost-Accuracy Frontier: The paper introduces two agentic architectures operating in the sub-$1-per-task region:
- Explorer-Definer Pipeline: Achieves 57.50% pass@2 at $0.25/task.
- Reflective Orchestrator: Achieves 67.25% pass@2 at $0.62/task.
- These represent lifts of ~42 and ~52 percentage points, respectively, over the one-shot baseline, without benchmark-specific training.
Diagnostic-Then-Test Methodology: The authors establish a falsifiable diagnostic: the pipeline is generation-bound, not selection-bound. Analysis shows that training-pair selection captures ~95% of the achievable candidate ceiling. This predicts that further gains require broader candidate generation rather than better ranking. The Orchestrator is explicitly designed as an architectural test of this prediction, confirming it via a near-identical lift in both naive pass@2 and unbiased pass@1.
Robustness and Ablation Analysis:
- Explorer Count ( $N$ ): Identified as the dominant cost-accuracy lever; accuracy saturates as $N$ increases, while definer sampling ( $M$ ) shows diminishing returns.
- Think Tool: Removing the definer's private think scratchpad reduces pass@2 by 5.75 percentage points, confirming its critical role in deliberation.
- Refinement: Disabling train-feedback refinement yields a negligible drop (-0.75 pp), suggesting the selection layer absorbs most of the benefit.
- Cross-Model Validation: The lift trajectory holds on Qwen3-235B, indicating the harness architecture, not the specific model, drives the gains.
Methodological Discipline: The paper advocates for token-normalized cost reporting, explicit pass@k definitions, and bootstrap confidence intervals to make architectural claims auditable.

Results

Performance: The Reflective Orchestrator achieves 67.25% pass@2 on the 400-task public set. This is a +9.75 percentage point improvement over the Pipeline (57.50%) and a +51.75 point improvement over the one-shot baseline (15.50%).
Cost Efficiency: The Orchestrator operates at $0.62 per task, a fraction of the cost of frontier-model search regimes (often $2–$ 400/task).
Statistical Significance: All architectural transitions (Baseline → CoT → Pipeline → Orchestrator) show statistically significant lifts (95% CI excludes zero) via paired bootstrap analysis.
Behavioral Insights: Trace analysis reveals that ~75% of unique solves by the Orchestrator (where the Pipeline failed) utilized the adaptive re-exploration (spawn) mechanism, validating the hypothesis that "wrong-abstraction" failures are the dominant residual error mode.

Significance and Claims

The paper claims that a substantial fraction of ARC-AGI-1 performance is accessible through agent harness engineering alone, without requiring specialized training or massive compute budgets.

Architectural Levers: The work demonstrates that the gap between simple prompting and high performance can be bridged by three distinct architectural levers:
1. Within-call deliberation (structured CoT).
2. Across-call decomposition (separating pattern discovery from program synthesis).
3. Adaptive re-exploration (allowing synthesis to trigger fresh exploration upon failure).
Generation vs. Selection: The study provides empirical evidence that for this class of problems, accuracy is constrained by the diversity of generated candidates (exploration) rather than the quality of selection.
Modesty: The authors explicitly state they do not claim a new state-of-the-art (SOTA) for the official leaderboard, noting that their results are on the public 400-task set rather than the semi-private holdout. However, they argue that their approach occupies a sparse and valuable region of the cost-accuracy plane, proving that high performance is not exclusively the domain of expensive, specialized systems.