💻 computer science

The Shape of Reasoning: Topological Analysis of Reasoning Traces in Large Language Models

This paper introduces a topological data analysis framework that evaluates large language model reasoning traces by capturing their geometric structures, demonstrating that these topological features provide a more accurate and label-efficient assessment of reasoning quality than traditional graph-based metrics.

Original authors: Xue Wen Tan, Nathaniel Tan, Galen Lee, Stanley Kok

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xue Wen Tan, Nathaniel Tan, Galen Lee, Stanley Kok

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Judging the Journey, Not Just the Destination

Imagine you are a teacher grading a student's math homework. Usually, you just look at the final answer. If it's correct, you give an "A." But what if the student got the right answer by guessing, or by using a magical trick that doesn't actually work? You wouldn't know, because you only looked at the result.

This is the problem with Large Language Models (LLMs). They are great at giving answers, but we don't really know how they got there. Sometimes they give the right answer for the wrong reasons. The authors of this paper want to fix this by looking at the "reasoning trace"—the step-by-step path the model took to get to the answer.

The problem is, checking these paths by hand is slow, boring, and subjective. Trying to automate it with simple "connect-the-dots" graphs (like counting how many times the model loops back) is too simple; it misses the nuance of complex thinking.

The Solution: Looking at the "Shape" of Thought

The authors propose a new way to judge these reasoning paths using Topology.

The Analogy: The Coffee Mug and the Donut
In topology (a branch of math), a coffee mug and a donut are considered the same shape. Why? Because if you imagine them made of stretchy rubber, you can squish and stretch the mug into a donut without tearing it or gluing anything. They both have exactly one hole.

The authors argue that good reasoning has a specific "shape" that stays the same, even if the specific words or steps change. Just like the mug and donut share a fundamental "hole-ness," a high-quality reasoning path shares a fundamental structural "shape" that distinguishes it from a confused or flawed path.

How They Did It: The "DNA Matcher" and the "Rubber Sheet"

The researchers built a four-step process to measure this shape:

The Test: They used hard math problems from the American Invitational Mathematics Examination (AIME). These problems have known, expert-written solutions (the "Gold Standard").
The Match: They asked AI models to solve these problems. Then, they used a biological tool called the Smith-Waterman algorithm (usually used to match DNA strands) to line up the AI's steps with the expert's steps. This tells them how well the AI's path matched the "correct" path.
The Shape Scan (The Magic Part): They turned the AI's steps into points in a high-dimensional space. Then, they treated these points like a cloud of dust on a rubber sheet. They slowly inflated the sheet (a process called a Vietoris-Rips filtration) to see how the points connected.
- H0 (The Clumps): This measures how the points group together. Do they form tight, coherent clusters?
- H1 (The Loops): This measures if the path goes in circles or takes detours.
- By watching how these clumps form and merge, and how loops appear and disappear, they created a "persistence diagram"—a map of the reasoning's shape.
The Comparison: They compared this "shape map" against the quality of the reasoning (how well it matched the expert).

What They Found: Shape Matters More Than Structure

The researchers compared their new "Shape" method against the old "Graph" method (which just counts loops and paths).

The Graph Method: Like counting how many times a driver took a wrong turn. It's okay, but it misses the big picture.
The Topology Method: Like looking at the entire map of the drive to see if the route was a smooth, efficient highway or a chaotic mess of back-and-forth detours.

The Result: The "Shape" method was much better at predicting whether the reasoning was good or bad.

When they used only the old graph metrics, they could barely predict the quality.
When they used the topological "shape" features, their prediction power jumped significantly.

The "Good" Shape:
High-quality reasoning traces looked like a coherent main road with short, useful side-checks.

They didn't wander off into huge, late-stage detours.
They formed tight, stable clusters of ideas (the "clumps" stayed together).
They avoided getting stuck in endless loops.

The Takeaway

This paper suggests that we shouldn't just look at the structure of an AI's thinking (like a flowchart); we should look at its geometry (the shape of the path).

Just as a master chef's knife skills have a specific rhythm and flow that a novice lacks, high-quality reasoning has a specific geometric "shape" that persists across different problems. By measuring this shape, we can automatically tell if an AI is truly reasoning or just guessing, without needing a human to read every single step.

What the paper does NOT claim:

It does not say this will fix AI hallucinations immediately.
It does not claim this works for medical diagnosis or legal advice yet.
It does not say this method works for every type of problem, only that it was tested on math problems.

The paper simply proves that topology is a better ruler for measuring the quality of AI thinking than the old graph-based rulers.

Technical Summary: The Shape of Reasoning

Problem Statement

Evaluating the quality of reasoning traces in Large Language Models (LLMs) remains a significant challenge characterized by reliance on labor-intensive manual annotation, expert rubrics, and slow pairwise judgments. Current automated approaches predominantly utilize graph-based proxies to quantify structural connectivity (e.g., branching, convergence). However, the authors argue that these abstractions are often overly simplistic for inherently complex reasoning processes and fail to clarify what constitutes high-quality reasoning. Furthermore, existing datasets often lack step-by-step solutions, forcing researchers to use final answer correctness as an imperfect proxy for reasoning quality. There is also a growing concern regarding "faithfulness," where models may arrive at correct answers through flawed or spurious reasoning processes that do not reflect their actual decision-making.

Methodology

The authors propose a novel evaluation framework grounded in Topological Data Analysis (TDA) to objectively quantify the geometry of reasoning traces. The methodology proceeds in four stages:

Data Generation: The study utilizes the American Invitational Mathematics Examination (AIME) dataset from the Art of Problem Solving (AoPS) portal, which provides detailed, community-contributed solution steps. The authors generate LLM reasoning traces for these problems using an answer-blind prompt.
Alignment: To address the scarcity of aligned step-by-step data, the authors adapt the Smith-Waterman algorithm (typically used for biological sequence alignment) to align LLM-generated reasoning steps with expert reference solutions in embedding space. This involves segmenting both traces and reference solutions, embedding them using all-mpnet-base-v2, and computing local alignment scores and coverage metrics.
Topological Feature Extraction: The aligned step embeddings are treated as point clouds in high-dimensional space. The authors construct Vietoris-Rips filtrations and compute persistence diagrams to extract topological invariants. They focus on homology dimensions $k \in \{0, 1\}$ $k \in {0, 1}$ , representing connected components and 1-cycles (loops). Three families of features are extracted:
- Vietoris-Rips summary statistics.
- Betti-curve descriptors.
- Persistence landscape descriptors.
Graph Baselines: For comparability, the authors also compute standard graph-theoretic metrics (e.g., diameter, clustering coefficient, loop count, small-world index) on the same embeddings, following prior work by Minegishi et al. (2025).

Key Contributions

Novel Alignment Application: The application of the Smith-Waterman algorithm to align LLM reasoning traces with expert solutions, enabling systematic evaluation on curated AIME problems with multiple valid solution paths.
TDA Framework for Reasoning: The introduction of a topological data analysis framework designed to objectively quantify reasoning quality by capturing invariant geometric structures that persist across diverse solution strategies.
Empirical Validation: Evidence demonstrating that topological features provide stronger discriminative signals for distinguishing high-quality from flawed reasoning traces compared to standard graph metrics.

Experimental Results

The study evaluated the predictive power of topological features versus graph metrics in predicting Smith-Waterman alignment scores (a proxy for reasoning quality) across eight different LLM settings (including Qwen3, DeepSeek-r1, and GPT-OSS variants).

Predictive Power: Topological Data Analysis (TDA) features consistently outperformed graph-only features as standalone predictors. The mean $R^2$ for TDA was 0.236 compared to 0.064 for graph metrics. The mean adjusted $R^2$ was 0.112 for TDA versus 0.032 for graph metrics.
Feature Stability: Adding graph features to TDA models resulted in modest gains in raw fit but inconsistent or negative gains in complexity-adjusted fit (adjusted $R^2$ ), suggesting that topology is the more stable signal.
Significant Features: Regression analysis on clustered TDA features identified specific topological characteristics associated with high-quality reasoning:
- Positive Association: Higher $H_0$ Betti spread (Cluster 2) and $H_1$ Betti width (Cluster 12).
- Negative Association: Lower $H_0$ Betti width (Cluster 3) and a weak negative association with $H_1$ max birth/max death (Cluster 16).
- Interpretation: High-quality traces tend to maintain a coherent main line, include short and varied local checks, and avoid late, large-scale detours.
Relation to Graph Metrics: The authors found that global cohesion and efficiency metrics (clustering, path length, diameter, small-world index) are strongly explained by topological features (specifically $H_0$ descriptors), whereas loop counts are less predictable by TDA, suggesting they depend more on idiosyncratic local connection patterns.

Significance and Claims

The paper claims that effective reasoning is better captured by higher-dimensional geometric structures rather than purely relational graphs. By treating reasoning traces as geometric objects, the proposed TDA framework offers a label-efficient, automated assessment method that is robust to surface-level variations in problem-solving approaches.

The authors posit that topological features provide a practical signal for future reinforcement learning algorithms, potentially enabling the optimization of reasoning processes based on structural quality rather than just final outcomes. They emphasize that while their features summarize the geometry of step embeddings rather than the symbolic structure of reasoning trees, these geometric proxies correlate strongly with alignment quality and offer a principled way to identify invariant properties of valid reasoning paths.

Limitations

The authors acknowledge that their findings are currently limited to the AIME dataset, which restricts the diversity of reasoning styles and domains (focusing on competitive mathematics). They also caution that topological features are geometric proxies dependent on the choice of embedder, segmentation, and distance metric, and do not necessarily correspond to literal detours or merges in a human-readable proof. Future work is suggested to curate datasets across other domains (commonsense, science, programming) and to ground topological events in more interpretable operations.