🤖 machine learning

Boule or Baguette? A Study on Task Topology, Length Generalization, and the Benefit of Reasoning Traces

This paper introduces PITA, a large-scale propositional logic dataset, to demonstrate that reasoning trace models excel at length generalization on broad, shallow tasks but struggle on narrow, deep ones, revealing fundamental scaling limitations and strengths of intermediate reasoning steps.

Original authors: William L. Tong, Ege Cakar, Cengiz Pehlevan

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: William L. Tong, Ege Cakar, Cengiz Pehlevan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Boule or Baguette? A Simple Explanation

Imagine you are trying to teach a robot how to solve a puzzle. You have two ways to teach it:

Direct Prediction (The "Baguette" Approach): You show the robot the puzzle and say, "Just guess the answer immediately."
Reasoning Traces (The "Boule" Approach): You show the robot the puzzle and say, "Think out loud. Write down every step you take to solve it, then give me the answer."

For a long time, everyone assumed the second method (thinking out loud) was always better. But this new study, titled "Boule or Baguette?", discovered something surprising: It depends entirely on the shape of the puzzle.

The authors use a French bread metaphor to explain this:

Boule: A round, wide loaf of bread.
Baguette: A long, thin, narrow stick of bread.

Here is what they found, broken down into simple concepts.

1. The Two Shapes of Problems

The researchers created a massive library of logic puzzles (over 23 million of them) to test AI models. They categorized these puzzles into two "shapes":

The "Boule" (Round & Wide): These are problems with many different variations but short steps.
- Analogy: Imagine a game where you have to guess a secret code. There are millions of possible codes (very wide), but you only have to check 3 or 4 numbers to find the right one (shallow).
The "Baguette" (Long & Thin): These are problems with few variations but very long, deep steps.
- Analogy: Imagine a maze. There are only a few mazes (narrow), but to get through one, you have to walk 1,000 steps without getting lost (deep).

2. The Big Discovery: Who Wins Where?

The researchers tested two types of AI models on these shapes:

The "Direct" Model: Just guesses the answer.
The "Reasoning" Model: Writes out a step-by-step proof before guessing.

The Result:

On the "Boule" (Wide/Shallow) problems: The Reasoning Model crushed the competition. By writing out its steps, it learned the patterns of the many different variations perfectly. It was like a student who writes out their math homework; they understood the concept and could solve any variation.
On the "Baguette" (Narrow/Deep) problems: The Direct Model actually won! The Reasoning Model got confused. Why? Because writing out 1,000 steps of logic is hard. The model got lost in its own long chain of thought, made a mistake early on, and couldn't recover. It was like a student trying to solve a 1,000-step math problem on a napkin; they ran out of space or got tired and gave up.

3. Why Does This Happen? (The "Signal in the Noise" Problem)

The paper explains this using a concept called Long Context.

The Reasoning Model's Weakness: When a problem is very deep (a long Baguette), the model has to remember a huge amount of information to get to the end. Imagine trying to find a single specific word in a book that is 1,000 pages long. The "signal" (the important clue) gets buried under all the "noise" (the other 999 pages of text). The model gets distracted by its own long explanation and forgets the original goal.
The Direct Model's Strength: On these deep, narrow problems, the Direct Model doesn't get distracted. It doesn't try to write a novel; it just looks for a shortcut or a pattern it memorized. Since the problem is narrow (few variations), the shortcut works!

4. The "Transitive Inference" Test (The Mini-Game)

To prove this wasn't just a fluke with their big dataset, they created a tiny, simple game based on Transitive Inference (a classic logic test).

The Game: If A is bigger than B, and B is bigger than C, is A bigger than C?
They made the game wide (many different letters) and deep (long chains like A>B>C>D>E...).
The Outcome: The results were exactly the same. The "Thinking Aloud" model was great at the wide game but terrible at the deep game. The "Just Guess" model was okay at the wide game but surprisingly good at the deep game.

5. What Does This Mean for the Future?

This study changes how we should think about AI "Reasoning" (like the "Deep Think" features in new chatbots).

Don't just force AI to "think" for everything. If a task is complex and requires a long chain of logic, forcing the AI to write out every step might actually make it dumber because it gets lost in the details.
Match the tool to the task.
- If you have a broad problem (like creative writing or general knowledge), let the AI "think out loud." It helps it generalize and learn.
- If you have a deep, narrow problem (like a specific, long mathematical proof), sometimes it's better to let the AI skip the chatter and go straight to the answer, or find a better way to handle long contexts.

The Takeaway

The paper's title, "Boule or Baguette?", is a question: Do you want a round, wide loaf (broad reasoning) or a long, thin stick (deep reasoning)?

The answer is: You need both, but you have to know which one you're eating.

Use Reasoning Traces for broad, varied tasks.
Be careful with Reasoning Traces for deep, long tasks, because the AI might get lost in its own thoughts.

The authors conclude that while "thinking out loud" is a superpower, it has a limit. If the path is too long and narrow, the AI needs a different kind of help to get to the end.

1. Problem Statement

Recent advances in Large Language Models (LLMs) have demonstrated that generating Reasoning Traces (RTs) (e.g., Chain-of-Thought) significantly improves performance on complex reasoning tasks. However, the conditions under which RTs are beneficial versus detrimental remain poorly understood. Specifically, there is a lack of clarity regarding:

How RTs interact with task topology (the structural complexity of a problem).
The limits of length generalization (the ability to solve problems requiring longer reasoning chains than seen during training).
Whether RTs universally outperform Direct Prediction (DP) models (which output an answer without intermediate steps) or if their efficacy depends on specific task characteristics.

The authors hypothesize that the performance gap between RT and DP models is not random but is dictated by the depth (number of reasoning steps) and breadth (number of unique examples) of the task.

2. Methodology

A. The PITA Dataset

To rigorously test these hypotheses, the authors introduce PITA (Propositional Inference Task and Analysis), a large-scale synthetic dataset containing over 23 million statements in propositional logic and 95 billion tokens.

Formalization: Statements and proofs are formalized in Lean, a theorem prover, to eliminate natural language ambiguities.
Task: Models must determine if a propositional statement is True or False.
Training Paradigms:
- RT Models: Trained on the statement followed by the full proof (reasoning trace) before the final classification token.
- DP Models: Trained only on the statement, outputting the classification token directly.
Task Topology Metrics:
- Depth: The number of unique proof states (steps) required to solve a statement.
- Breadth: The number of unique statements for a given size (number of atoms).
Dataset Splits: PITA is divided into four splits with varying topologies:
1. FULL & IMPLY: "Boule-shaped" (Broad and Shallow). High breadth, low depth.
2. OR & PHP: "Baguette-shaped" (Narrow and Deep). Low breadth, high depth. (PHP involves the Pigeonhole Principle, requiring exponentially long proofs).

B. Transitive Inference (TI) Theoretical Model

To generalize findings beyond PITA, the authors construct a simplified Transitive Inference (TI) task based on syllogisms.

Structure: Symbols are arranged in $B$ parallel branches, each of depth $D$ .
Goal: Determine if a relation $A \to C$ holds given $A \to B$ and $B \to C$ .
Theoretical Analysis: The authors analyze single-layer Transformers using max-margin theory. They derive scaling laws relating model width ( $H$ ), task depth ( $D$ ), and task breadth ( $B$ ) to generalization accuracy.

3. Key Contributions

Definition of Task Topology: The paper formalizes "depth" and "breadth" as critical metrics for reasoning tasks, moving beyond simple token length.
The "Boule vs. Baguette" Phenomenon: The discovery of a reversal in generalization performance based on task topology:
- Boule (Broad/Shallow): RT models significantly outperform DP models.
- Baguette (Narrow/Deep): DP models significantly outperform RT models (sometimes by >50%).
Theoretical Explanation: The authors provide a theoretical framework explaining why this reversal occurs:
- Broad Tasks: RTs provide an inductive bias that allows the model to generalize perfectly across diverse examples by enumerating intermediate steps. DP models tend to overfit to spurious correlations in the training distribution.
- Deep Tasks: RTs suffer from long-context processing limitations. As depth increases, the reasoning trace becomes a long sequence where the relevant signal (the connection between start and end) is weakly dispersed among many tokens. This leads to statistical indistinguishability and failure to terminate. DP models avoid this "dilution" by mapping input directly to output.
PITA Dataset Release: A massive, open-source benchmark for logical reasoning and proof generation.

4. Key Results

Empirical Findings on PITA

Breadth-Dominated Splits (FULL, IMPLY): RT models achieved higher generalization accuracy than DP models, especially as model size increased. RTs successfully generalized to longer proofs than seen in training.
Depth-Dominated Splits (OR, PHP): DP models vastly outperformed RT models. On the PHP split (deepest), the 7B DP model outperformed the 7B RT model by ~50 percentage points.
Failure Mode of RTs on Deep Tasks: The primary cause of RT failure was non-termination. Models failed to generate the final classification token within the context window, often getting lost in the long chain of reasoning.

Theoretical Findings on TI

Scaling Laws:
- DP Models: Training accuracy scales linearly with depth ( $D \propto H$ ). Generalization decays rapidly with breadth ( $\alpha_g \propto B^{-2}$ ).
- RT Models: Training accuracy scales with the square root of depth ( $D \propto \sqrt{H}$ ) due to context dilution. However, generalization remains perfect across breadth ( $\alpha_g \propto 1$ ) because the reasoning trace creates a permutation-invariant mapping between train and test examples.
Inductive Bias: RTs enforce a solution structure that transfers perfectly from training to testing in broad tasks, whereas DP models learn "shortcuts" that fail when the task space broadens.

5. Significance and Implications

Rethinking Reasoning Traces: The paper challenges the assumption that "more reasoning is always better." It demonstrates that for deep, narrow tasks (like complex math proofs or long-horizon planning), forcing a model to generate a trace can be harmful due to context window limitations and signal dilution.
Architectural Guidance: The results suggest that for deep tasks, improving long-context processing (e.g., better attention mechanisms, retrieval) is more critical than simply adding reasoning traces. Conversely, for broad, diverse tasks, RTs are essential for robust generalization.
Benchmarking: PITA provides a rigorous, formal testbed for evaluating logical reasoning, moving beyond natural language benchmarks that may conflate reasoning with language modeling capabilities.
Future Directions: The authors highlight the need for models that can dynamically switch between direct prediction and reasoning traces based on the estimated depth and breadth of the task.

In summary, the paper establishes that the efficacy of reasoning traces is not intrinsic but topology-dependent. Reasoning traces act as a powerful generalization tool for broad, shallow problems but become a liability for deep, narrow problems due to the statistical difficulties of processing long, dispersed contexts.