It's Not the Size: Harness Design Determines Operational Stability in Small Language Models

This paper demonstrates that for small language models (2-3B parameters), a structured 4-stage harness pipeline significantly outperforms raw prompting or minimal wrappers in operational stability and task success, while also revealing that insufficient scaffolding can lead to structural format collapse in certain models.

The Gist

Imagine you have a very smart, but slightly scatterbrained, assistant. This assistant is small (they only have a "2B" or "3B" brain size, which in AI terms means they are "Small Language Models"). You want them to do a bunch of complex jobs, like writing reports, searching the web, or following multi-step instructions.

The paper asks a simple question: Does the way you give instructions to this assistant matter more than how "smart" the assistant is?

The answer is a resounding yes. The authors call the way you give instructions a "harness." Think of a harness like the gear you put on a horse. You can have a fast horse, but if you don't give it a bridle and reins (the harness), it might run in circles, get tired, or ignore your commands.

Here is the breakdown of their experiment and findings using everyday analogies:

1. The Three Ways to Give Instructions (The Harnesses)

The researchers tested three different ways to talk to these AI assistants:

• The "Raw Prompt" (Model-Only): This is like shouting a task at your assistant while they are eating lunch. "Hey, write me a report!" No structure, no rules, just a raw request.
• The "Minimal Shell" (Wrapper Tags): This is like putting the task inside a fancy box with a label that says "TASK START" and "TASK END." It looks organized, but it doesn't actually help the assistant think through the steps.
• The "4-Stage Pipeline" (The Full Harness): This is like giving the assistant a detailed checklist:
  1. Plan: "First, think about what you need to do."
  2. Execute: "Now, do the work."
  3. Verify: "Check your work. Did you make a mistake?"
  4. Recover: "If you made a mistake, fix it and try again."

2. The Big Surprise: "More Help" Can Sometimes Be "Less Help"

The researchers found something weird and counter-intuitive.

For two of the models, the "Minimal Shell" (the fancy box) actually made the assistant perform worse than the "Raw Prompt."

• The Analogy: Imagine asking a friend to bake a cake. If you just say "Bake a cake," they might do a decent job. But if you hand them a rigid, confusing form with boxes to fill out before they can even mix the flour, they might get overwhelmed, forget the recipe, and burn the cake.
• The Result: The extra "wrapper tags" added mental clutter (cognitive load) that confused the small models, causing them to time out or fail more often than if they had just been given a simple command.

3. The "Scaffold Collapse" (When the Assistant Drops the Format)

One of the most interesting findings involved the LLaMA 3.2 model.

• The Situation: When asked to write a report in a specific format (like a JSON list), this model would often get confused and just write a normal paragraph instead, ignoring the rules.
• The Term: The authors call this "Scaffold Collapse."
• The Analogy: Imagine a construction worker who is great at laying bricks (generating content) but keeps forgetting to use the blueprints (the format). Without a foreman (the harness) standing over them saying, "Check the blueprint, you're building it wrong," they just build whatever they feel like. The harness didn't make them smarter at laying bricks; it just forced them to follow the blueprint.

4. Why the "4-Stage Pipeline" Won

The full pipeline (Plan → Execute → Verify → Recover) was the clear winner, especially for complex tasks.

• Planning: This acted like a "mental anchor." Before the model started writing, the "Plan" step forced it to remember constraints (like "keep this under 200 characters"). Without this step, the model would forget the limit and write a novel.
• Recovery: This was the safety net. If the model got stuck or timed out, the "Recover" step let it try again.
• The Result: With the full pipeline, the models achieved near-perfect success rates (95%+), whereas without it, they struggled significantly.

5. The "Verification" Catch

The researchers also measured how often the "Verify" step caught mistakes.

• The Stat: The system caught about 62.5% of the errors and fixed them.
• The Catch: Sometimes the "Verify" step was fooled. For example, if the model was asked to count characters, the model would guess the number wrong, and the verifier would also guess wrong, thinking the job was done when it wasn't.

6. The "Tool" Problem (A Flaw in the Experiment)

The paper included a task where the AI had to search the web.

• The Issue: The "Raw" and "Minimal" versions of the AI didn't have access to the search tool at all, so they failed automatically. The "Pipeline" version did have the tool, but it failed because the search engine (DuckDuckGo) blocked them for asking too many questions too fast.
• The Lesson: The authors admit this part of the test was flawed because they were comparing "having a tool" vs. "not having a tool," rather than comparing "good harness" vs. "bad harness."

Summary: What Does This Mean?

The main takeaway is simple: For small AI models, how you organize the task is more important than the model's size.

• Don't overcomplicate it: Adding fancy labels (minimal shells) can sometimes confuse small models more than helping them.
• Structure is key: Breaking a task down into "Plan, Do, Check, Fix" allows even a "small" brain to do complex jobs reliably.
• The Harness is the Hero: The "harness" (the system of instructions) acts as both a safety net (fixing mistakes) and a guide (preventing mistakes before they happen).

The paper concludes that if you want small, efficient AI models to work well in the real world, you need to spend more time designing the "harness" (the workflow) than just worrying about which model you pick.

Technical Summary

Technical Summary: Harness Design Determines Operational Stability in Small Language Models

Problem Statement

While Small Language Models (SLMs, 2–3B parameters) offer a cost-effective alternative to large models for edge and on-premises deployment, their operational stability in multi-step tasks remains under-researched. Existing literature focuses heavily on static benchmark accuracy (e.g., MMLU, HumanEval), leaving a gap in understanding how execution structures affect output completeness and format compliance in real-world scenarios. Furthermore, while frameworks like Reflexion and ReAct suggest that execution flow matters, systematic evidence regarding the efficacy of such structures specifically for 2B–3B models is scarce.

Methodology

The authors conducted an experimental analysis comparing three levels of "harness engineering" (execution structure) across three SLMs: Gemma4 E2B (2B), Qwen3.5:2B (2B), and LLaMA 3.2 (3B). The study utilized 24 tasks across six categories, including structured knowledge extraction, search-and-grounding, workflow completion, and constraint-sensitive tasks.

Harness Conditions:

1. Model-Only: Raw prompts with task descriptions only.
2. Minimal-Shell: Tasks wrapped in structural tags ([TASK START], [OUTPUT]) without execution flow logic.
3. 4-Stage Pipeline: A structured workflow consisting of Plan → Execute → Verify → Recover. The recovery stage allows for up to two retries with error feedback upon verification failure or timeout.

Evaluation Metrics:

• Task Success Rate (TSR): Percentage of tasks achieving full success (score=2).
• Valid TSR (VTSR): Percentage of tasks achieving at least partial success (score≥1).
• Verification Catch Rate (VCR): The ratio of defects caught by the verification stage that were subsequently retried.
• Thresholds: Operational stability was defined as meeting TSR ≥ 0.65 and VTSR ≥ 0.80.

Ablation Study:
The study isolated specific pipeline components (Planning, Verification+Recovery, Recovery) to quantify their individual contributions to performance gains.

Key Findings and Results

1. Operational Stability via Harness Design

The study demonstrates that SLMs can meet operational thresholds within constrained task scopes, but success is heavily dependent on harness design rather than parameter count alone.

• Gemma4 E2B: Achieved TSR=0.952 and VTSR=1.000 under the 4-stage pipeline, significantly outperforming the model-only baseline (TSR=0.762).
• LLaMA 3.2 (3B): The model-only condition failed to meet stability thresholds (TSR=0.429) due to "scaffold collapse," where the model abandoned JSON structures under complex format requirements. The pipeline harness restored stability, though with lower absolute gains compared to Gemma.

2. The Non-Monotonic Effect

A critical finding is the non-monotonic phenomenon: in both Gemma4 and Qwen3.5, the minimal-shell condition yielded lower TSR than the model-only condition.

• Observation: Minimal-shell TSR (0.714) < Model-only TSR (0.762) for Gemma; Minimal-shell (0.857) < Model-only (0.952) for Qwen.
• Hypothesis: The authors suggest that wrapper tags may increase cognitive load in complex workflows, inducing additional timeouts, though this requires further controlled experimentation to prove causation.

3. Component Contributions (Ablation)

• Planning: Acts as a "format anchor" for quantitative constraints. Removing the planning stage caused a drop in TSR, specifically in tasks with character limits (e.g., T1-04), where the model failed to adhere to limits without explicit decomposition.
• Recovery: The recovery mechanism was the primary contributor to performance gains, accounting for approximately 24.7% of the total gain. It successfully converted timeout failures and format violations into successes.
• Verification: The Verification Catch Rate (VCR) across all pipeline runs was 0.625. Notably, LLaMA 3.2 had a VCR of 1.000 (catching all defects) but could not always fix them, while Qwen's VCR was 0.000 (failing to catch defects).

4. Failure Mode Classification

The paper categorizes failure modes based on harness fixability:

• Fixable (Reactive): Timeouts and rule-detectable constraints (e.g., prohibited words) are successfully addressed via recovery retries.
• Fixable (Proactive): "Scaffold collapse" (format abandonment) is addressed by the planning and prompt structuring stages.
• Unfixable: Quantitative constraints (e.g., exact character counts) often fail due to inherent LLM counting limitations, and hallucinations remain beyond the scope of harness correction.

Significance and Claims

The paper claims that harness design is a decisive factor in the operational stability of SLMs, often outweighing the benefits of raw model size.

• Scaffold vs. Retry: The authors distinguish between two independent harness functions: reactive recovery (fixing errors after they occur) and proactive scaffolding (preventing errors by structuring the prompt and execution flow). The study argues these roles are distinct and non-interchangeable.
• The "Less is More" Warning: The non-monotonic results suggest that "half-measures" in harness design (e.g., adding tags without a full pipeline) can degrade performance compared to raw prompting.
• Limitations: The authors modestly acknowledge limitations, including a small task set (24 tasks), lack of variance estimation (single runs), and specific design flaws in the Web Search task (T6) which compared tool-equipped vs. tool-less conditions rather than orchestration quality.

In conclusion, the paper posits that for SLMs to be reliably deployed in operational environments, the engineering of the execution harness is as critical as the model selection itself. Future work is suggested to focus on rule-based verification to improve catch rates and redesigning tasks to isolate orchestration effects from tool access.