Technical Summary: Code-Guided Reasoning for Small Language Models

Problem Statement

Small Language Models (SLMs) are frequently deployed in practical systems not as isolated answerers, but as components within scaffolds involving external tools, code execution, and repeated model calls. However, standard Multiple-Choice Question Answering (MCQA) benchmarks evaluate SLMs primarily through direct prompt-response interactions, failing to capture performance under these "scaffolded" conditions.

The core question addressed by this work is: Does the same small solver behave differently when moved from a direct natural-language option prompt into an executable Python scaffold, and can this difference be audited? The authors note that while code alone does not guarantee reliability, current LLMs function as both code generators and domain reasoners. Deployed workflows often combine these roles, yet evaluation protocols rarely distinguish between a model's direct capability and its performance when mediated by a generated executable controller.

Methodology: Code-Guided Reasoning (CGR)

The paper introduces Code-Guided Reasoning (CGR), an evaluation protocol and trace package designed to measure the impact of executable reasoning scaffolds on SLMs.

The CGR Protocol

For each MCQA item, the system generates a three-channel output:

1. Direct Baseline: The target solver answers the question directly, emitting a single option letter.
2. Assisted Solver: A generator LLM writes an item-specific Python scaffold. This scaffold stores variables, branches logic, calls the target solver helper multiple times, extracts answers, and applies tiebreakers. The final "assisted answer" is the option selected by this executable logic.
3. Generator-Side Answer: The generator LLM also selects an answer within the scaffold's logic (independent of the solver's output). This serves as a diagnostic channel to distinguish between scaffolded solver improvement and generator-side knowledge.

Execution Flow

• Input: An MCQA item (question, options, gold label).
• Generation: A generator LLM produces a Python function with a fixed return contract: (solverLLM_answer, genLLM_answer, genLLM_difficulty).
• Execution: The scaffold runs, calling the target solver via a helper interface (llm_model). It may make multiple calls (e.g., for analysis and verification) and uses an extractor to parse the first standalone capital letter (A–Z) from responses.
• Scoring: The three channels (Direct, Assisted, Generator-side) are scored separately against the gold label.

Datasets and Models

The evaluation utilizes a locally prepared, normalized MCQA bundle containing 20,498 retained result rows across nine dataset configurations (including MMLU-Pro, OpenBookQA, SuperGPQA, MedQA, Time-MQA, AIME, and others). Six distinct solver models were evaluated, ranging from compact recent models (Gemma 4 E2B, Nemotron-3-Nano-4B) to earlier instruction-tuned and code-oriented models (Granite 4H Small, Granite 8B Code, Llama 3.2 11B, Mistral Small 3.1 24B).

Partitioning Strategy

To ensure interpretability, the authors define a non-zero-baseline partition. This includes only dataset–solver pairs where the solver achieved at least one correct answer in the direct baseline. Pairs with zero direct accuracy are treated as a separate "zero-baseline diagnostic partition," as large gains there may reflect prompt-format rescue or extraction mismatches rather than genuine solver improvement.

Key Contributions

1. Evaluation Setting: The study establishes executable MCQA scaffolding as a distinct evaluation condition, observing the same target solver under direct prompting and within a generated Python scaffold, with separate scoring for direct, assisted, and generator-side answers.
2. Trace Package: The authors release the CGR trace package, a resource containing 20,498 result rows, generated programs, response metadata, and audit fields. This allows for the inspection of scaffold contracts, call counts, extraction failures, and answer-channel discrepancies.
3. Quantitative Results: The paper reports a specific, bounded quantitative improvement: in the non-zero-baseline partition, assisted macro accuracy rises from 38.11% (direct) to 66.21% (assisted), a +28.10 percentage-point difference. Under a stricter gate (direct accuracy > 30%), the gain is +14.11 points.

Results and Observations

• Performance Gains: The assisted path significantly outperforms the direct baseline in the non-zero-baseline partition. The improvement is most pronounced when direct baseline accuracy is low but non-zero (e.g., MedQA, AIME, MMLU-Pro).
• Budget Disparity: The assisted inference path uses approximately 7.36 times the solver-token budget of the direct baseline (mean 7.23 calls vs. 1.01 call per item).
• Extraction Fragility: Answer extraction is a significant failure point. The assisted path has a 15.67% extraction failure rate (returning 'X' for no valid letter) compared to 0.44% for the direct baseline.
• Negative Cases (Time-MQA): The Time-MQA dataset exhibits regressions for stronger direct solvers. For models with high direct accuracy on Time-MQA, the scaffolded approach sometimes reduces performance, suggesting that decomposing time-series questions can introduce inconsistency or distract from relevant signals.
• Audit Findings:
  • Some generated programs violate the "no-hard-coding" instruction, containing literal answer assignments.
  • The prompt limit of 10 solver calls is not enforced at runtime; some rows exceed this limit.
  • Generator-side answers often differ from assisted solver answers, confirming the necessity of keeping channels separate.

Significance and Claims

The paper positions CGR not as a universal improvement method or a new model architecture, but as an evaluation protocol that makes the scaffolded condition measurable and auditable.

• Modest Claims: The authors explicitly state that the results are descriptive, not causal. The observed gains are associated with a larger inference budget and specific extraction behaviors, not necessarily the executable structure alone. The paper does not claim CGR is equal-cost, universally beneficial, or safe for clinical/operational deployment without expert validation.
• Interpretability: The primary value of CGR is the "paired record" it provides. It allows researchers to determine why a score changed: Was it due to useful decomposition, answer-format repair, extra calls, or generator-side knowledge?
• Future Directions: The authors identify the need for future work to include matched-budget baselines (e.g., direct self-consistency), stricter option validation, sandboxed execution, and repeated sampling to isolate the causal effect of the scaffold from the extra inference budget.

In summary, CGR demonstrates that moving an SLM into an executable scaffold can yield substantial accuracy gains on MCQA tasks, but these gains come with increased computational costs, extraction fragility, and domain-specific variability that must be carefully audited.