STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs

1. Problem Statement

Current Large Language Model (LLM) evaluation relies heavily on aggregate benchmark scores (e.g., accuracy on GSM8K or MATH). While these metrics indicate overall performance, they suffer from significant limitations:

• Monolithic Evaluation: They treat complex tasks as single units, obscuring why a model fails. Two models with identical scores may have entirely different underlying weaknesses (e.g., one lacks a specific reasoning step, while another fails only when combining multiple steps).
• Black Box Nature: Standard evaluation does not reveal intermediate reasoning failures or the specific "compositional skill gaps" where a model struggles to coordinate multiple skills.
• Limitations of Existing Methods:
  • Chain-of-Thought (CoT): Generates intermediate traces but is difficult to scale systematically and often produces unfaithful explanations that do not reflect internal reasoning.
  • Task Decomposition: Breaks tasks down but is primarily used to improve performance rather than to diagnose specific failure modes and skill interactions.

The authors argue that to truly understand LLM capabilities, we need a diagnostic framework that distinguishes between skills a model possesses in isolation versus skills it can only perform with guidance, and specifically identifies where the composition of skills breaks down.

2. Methodology: Scaffolded Task Design (STaD)

The authors propose STaD, a framework inspired by the cognitive science concept of scaffolding (Vygotsky). Instead of using scaffolding to improve performance, STaD uses it as a diagnostic tool to reveal the minimum level of support required for a model to succeed.

Core Framework Components

STaD operates using three distinct LLM roles:

1. Teacher Model ($f_{teach}$):
   • Decomposition: Breaks a benchmark question $q$ into a sequence of $K$ minimal reasoning sub-tasks ($s_1, ..., s_K$).
   • Computation: Computes ground-truth intermediate answers for each sub-task.
   • Generation: Creates Scaffolded Variations. For a task with $K$ steps, it generates $K-1$ variations where the first $j$ steps are pre-solved and injected into the prompt, leaving the model to solve only the remaining steps.
2. Judge Model ($f_{judge}$):
   • Verifies the consistency of the Teacher's decomposition by ensuring that the Teacher can correctly solve the original task even when provided with its own intermediate answers (preventing hallucination or inconsistent logic).
3. Target Model ($f_{target}$):
   • The model being diagnosed. It is evaluated on the original task and the generated scaffolded variations.

Key Metrics and Analysis

• Minimum Scaffolding Level ($k$): Defined as the smallest number of pre-solved steps required for the target model to answer correctly.
  • $k=0$: Solvable independently.
  • $k>0$: Solvable only with support (indicates a gap).
  • $k=-1$: Unsolvable even with full support (indicates a fundamental intractability).
• Skill Mapping: Sub-tasks are mapped to higher-level reasoning skills (e.g., "Temporal Reasoning," "Algebraic Translation") using clustering and semantic embedding. This allows analysis at the skill level rather than just the task level.
• Bottleneck Identification: The framework identifies the first step in a sequence where the model fails, even if it possesses the individual skills required for that step in isolation.

3. Experimental Setup

• Datasets: Three reasoning benchmarks covering varying difficulty levels:
  • ToT Arithmetic: Temporal reasoning (dates, times, durations).
  • GSM8K: Grade school math word problems.
  • Math-Hard: Complex mathematical problems (algebra, combinatorics, calculus).
• Models:
  • Teacher: GPT-OSS-120B (for generating decompositions and scaffolds).
  • Target Models: Six models of varying sizes and families (Qwen2.5, Granite 3.3, Llama 3.1/3.2) ranging from 2B to 8B parameters.
• Process: The framework filters out instances where the Teacher model cannot consistently solve the scaffolded variations, ensuring high-quality test cases.

4. Key Results and Findings

A. Performance vs. Scaffolding

• GSM8K: Models generally perform well independently (high baseline). Scaffolding yields minimal gains, suggesting most failures are due to isolated skill deficits rather than composition issues.
• ToT & Math-Hard: Significant performance gaps exist between original and scaffolded tasks. Many tasks that models fail initially become solvable with intermediate support, revealing that the failure was due to compositional gaps (inability to coordinate steps) rather than a lack of knowledge.

B. Compositional Skill Gaps (The "Bottleneck" Effect)

The most critical finding is that skills present in isolation often fail in combination.

• Example: A model might correctly perform "Subtraction" and "Multiplication" individually. However, when a task requires Subtraction followed immediately by Multiplication, the model fails.
• Bottleneck Analysis: In ToT Arithmetic, the most frequent bottlenecks were not basic arithmetic but complex interactions like "Overlap and intersection of time" combined with "Discrete slot counting."
• Model Specificity: Models with similar aggregate scores have vastly different bottleneck profiles.
  • Example: Qwen3B and Granite8B had similar ToT scores, but Qwen3B struggled with "Overlapping time" combinations, while Granite8B struggled more with "Unit conversion" combinations.

C. Learnable vs. Intractable Tasks

STaD distinguishes between two types of failure:

1. Learnable (Scaffoldable): Tasks where providing intermediate steps allows the model to reach the correct answer. These indicate a need for better training on skill coordination.
2. Intractable: Tasks that remain unsolved even with full scaffolding. These suggest deeper issues, such as an inability to synthesize information, plan a decomposition strategy, or understand the core problem structure.

D. Skill Granularity

• Isolation vs. Context: Individual skill testing (evaluating skills in isolation) often overestimates model competence. Skills like "Arithmetic on durations" or "Calendar arithmetic" appeared moderately strong in isolation but were major bottlenecks when embedded in multi-step reasoning chains.

5. Key Contributions

1. Conceptual Framework: Formalizes Scaffolded Competence, shifting the evaluation focus from "what a model can do alone" to "what it can do with support," thereby revealing latent compositional gaps.
2. STaD Framework: A systematic, scalable pipeline for generating controlled task variations to pinpoint exact reasoning failure points. The authors release the code and datasets.
3. Empirical Insights:
   • Demonstrated that aggregate scores mask distinct failure modes across models.
   • Proved that failures often stem from skill interactions (coordination) rather than missing primitives.
   • Identified specific "bottleneck" skill combinations (e.g., Time Overlap + Discrete Counting) that are universally difficult for current models.

6. Significance and Implications

• Actionable Training: Instead of generic fine-tuning, STaD suggests generating synthetic data specifically targeting skill combinations that cause failures (e.g., training models to coordinate "Time Parsing" with "Arithmetic").
• Better Evaluation: Proposes a new evaluation protocol that reports both isolated skill accuracy and compositional bottlenecks, offering a more nuanced view of model capabilities.
• Diagnostic Precision: Moves beyond "the model is bad at math" to "the model fails when it must combine Unit Conversion with Duration Arithmetic."
• Future Directions: Highlights the need for training strategies that explicitly target skill coordination and reasoning planning, not just skill acquisition.

In conclusion, STaD provides a "microscope" for LLM reasoning, revealing that the primary barrier to advanced reasoning in current models is not necessarily a lack of knowledge, but a failure to compose known skills effectively within a specific context.