Technical Summary: REASONINGLENS

Problem Statement

The emergence of Large Reasoning Models (LRMs) such as Deepseek-R1, GPT-5, and Qwen3 has led to the generation of exceptionally long Chain-of-Thought (CoT) traces. While these models demonstrate advanced capabilities in decomposing problems and self-correcting, the scaling of reasoning length introduces a significant "transparency burden." Critical logical dependencies are often buried under massive, unstructured procedural text, creating a "wall of text" that obscures structural transparency. This opacity hinders manual inspection, error diagnosis, and safety assurance. Existing visualization methods are largely heuristic and descriptive, offering superficial text rendering without capturing underlying reasoning structures or providing a comprehensive framework for deep-level analysis. Consequently, there is a lack of tools capable of guiding the design of structured taxonomies to bridge the gap between simple visualization and actionable diagnostic auditing.

Methodology

To address these challenges, the authors present REASONINGLENS, an open-source framework designed for hierarchical visualization and diagnostic auditing. The system operates through a principled pipeline comprising three core components:

1. Hierarchical Visualization

The framework transforms raw, unstructured CoT traces into multi-layered, interactive reasoning graphs. This process relies on a meticulously designed taxonomy of reasoning behaviors categorized into two levels:

• Exploration-Level: High-level strategic moves (e.g., Decomposition, Backtracking, Validation) that dictate the global reasoning trajectory and topology transitions.
• Exploitation-Level: Low-level procedural execution units (e.g., Knowledge Retrieval, Procedural Execution, State Assertion) that instantiate the plan.

The system extracts "atomic planning units" from the text using linguistic markers (e.g., "but," "wait") and constructs a macro-level exploration graph connected by structural edges. Within each macro-node, a local execution subgraph is generated to encode fine-grained procedural dependencies.

2. Agentic Diagnosis

To tackle implicit reasoning errors masked by complexity, REASONINGLENS employs a multi-agent framework comprising three modules: Memory, Verification, and Suggestion.

• Error Taxonomy: The system utilizes a unified taxonomy of five error types: Overthinking, Safety violations, Knowledge Errors, Logical Errors, and Formal Errors.
• Detection Mechanism: The Memory module performs incremental inspection via memory compression to localize errors while preserving trace consistency. The Verification module leverages external tool invocations to resolve ambiguities.
• Actionable Feedback: The Suggestion module maps identified errors to specific mitigation strategies, drawing from a repository of training-free approaches and post-training techniques.

3. Systemic Profiling

Moving beyond instance-level inspection, this module aggregates trajectory sets into a unified behavioral representation. It compresses similar reasoning paths while preserving distinct heuristic features to synthesize model-level profiles. These profiles analyze three axes: exploration habits (depth–breadth trade-offs), verification reliability, and stability bottlenecks.

Evaluation Benchmark: LENSBENCH

To validate the framework, the authors constructed LENSBENCH, a unified benchmark containing 130 instances across five representative categories of reasoning failures. The dataset features gold annotations for both exploration-level hierarchical structure and fine-grained error types, created through controlled error injection and human verification.

Key Results

The framework was evaluated on LENSBENCH using five backbone evaluators (DeepSeek-V4-Pro, MiniMax-M2.7, Qwen3.5-27B, Gemma-4-26B-A4B, and Qwen3-32B).

• Hierarchical Visualization: The module demonstrated consistent robustness across model scales. It achieved average Node Type Accuracy (NTA) of 75.0 and Graph Edit Similarity (GES) of 69.7, validating that the hierarchical design functions effectively independent of the backbone model's strength.
• Agentic Diagnosis: Diagnostic performance scaled with the strength of the backbone model. Overall diagnostic F1 scores ranged from 66.3 (Qwen3-32B) to 82.3 (DeepSeek-V4-Pro).
  • Safety Detection: Remained consistently high across all models, suggesting current safety alignments effectively reinforce boundaries.
  • Complex Errors: Detection of Knowledge and Logical errors was strongly model-dependent, with lower F1 scores on weaker models (e.g., 39.0 for Knowledge Error on Qwen3-32B), indicating that diagnosing deep reasoning failures relies heavily on the model's internal reasoning capacity.

Significance and Contributions

The paper argues that structured visualization must be purpose-driven to serve as both a structural scaffold for understanding internal logic and a diagnostic foundation for localizing failures. The major contributions are:

1. Formalized Framework: The formalization of a multi-granularity diagnostic framework tailored to the transparency burden of large reasoning models.
2. Open-Source Toolkit: The release of REASONINGLENS, enabling hierarchical visualization, automated diagnosis, and systematic profiling for LRMs.
3. Unified Benchmark: The construction of LENSBENCH, which assesses both structural visualization fidelity and CoT error-annotation accuracy, providing a foundation for future reasoning analysis tools.

The authors conclude that REASONINGLENS transcends mere heuristic rendering, offering a formalized foundation for diagnosing the internal logic of LRMs and providing a scalable path toward transparent and verifiable machine deliberation. The work is presented as a modular foundation for interpreting, debugging, and optimizing the next generation of reasoning-centric AI.