Technical Summary: Framework of Thoughts (FoT)

Problem Statement

Current prompting schemes for Large Language Models (LLMs)—such as Chain of Thought (CoT), Tree of Thoughts (ToT), and Graph of Thoughts (GoT)—have demonstrated significant improvements in reasoning capabilities. However, the authors identify three critical limitations in existing approaches:

1. Static and Manual Structures: Most schemes rely on manually defined, static graph structures. Users must specify task-specific prompts and decomposition logic a priori. This lack of adaptability prevents generalization to dynamic or unseen problem types where the ideal reasoning structure cannot be known in advance.
2. Insufficient Optimization: Existing schemes often suffer from untapped accuracy potential due to a lack of systematic hyperparameter and prompt optimization. The high cost of testing various combinations often leads to suboptimal configurations, making it unclear whether performance differences stem from architectural superiority or lucky parameter choices.
3. Inefficient Execution: Many schemes execute LLM prompts sequentially and redundantly. They fail to leverage parallel execution or caching, resulting in unnecessary inference costs and significant waiting times, particularly when the same operations or inputs are repeated.

Methodology: Framework of Thoughts (FoT)

To address these limitations, the authors introduce Framework of Thoughts (FoT), a general-purpose foundation framework designed for implementing and optimizing dynamic reasoning schemes. FoT is not a specific prompting scheme itself but a modular infrastructure that supports chains, trees, and graphs.

Core Architectural Components

• Dynamic Graph Structures: FoT distinguishes between two graph types:
  • Execution Graph ($G_X$): Models the sequence of operations, their inputs/outputs, and the full history of execution. Unlike static frameworks, this graph can be modified and evolve dynamically during execution based on operation outputs.
  • Reasoning Graph ($G_R$): A simpler representation of the logical dependencies between thoughts (nodes) generated by completed operations.
  • Operations: The building blocks of the framework. An operation is a function that takes input thoughts and an execution graph, returning updated thoughts and a potentially modified execution graph. This allows for automatic derivation of graph structures.
• Safe Parallel Execution: To enable concurrency without race conditions in a dynamically evolving graph, FoT employs a Controller and Scheduler. It enforces strict constraints on graph modifications:
  • Operations can only view their ancestors, descendants, and themselves.
  • Operations cannot modify ancestors (already executed) or non-exclusive descendants (to prevent conflicts).
  • Operations can only modify their "exclusive descendants" and add connections within this safe subgraph.
• Efficient Caching: FoT implements two caching strategies to reduce redundant LLM calls:
  • Process Cache: Temporarily stores results within a single problem instance execution.
  • Persistent Cache: Stores results across multiple problem instances, allowing the system to recall outputs for identical inputs in subsequent tasks.
• Optimization Tools: The framework integrates Optuna for hyperparameter tuning and DSPy for prompt optimization. These tools allow developers to optimize for accuracy, runtime, or cost, with caching making the optimization process computationally feasible.

Key Contributions

1. A Dynamic Foundation Framework: FoT enables the implementation of reasoning schemes where graph structures are not pre-defined but evolve automatically during execution. This contrasts with frameworks like GoT or LangGraph, which are primarily designed for static flows.
2. Performance Optimization Mechanisms: By integrating parallel execution and persistent caching, FoT unlocks latent performance potential, significantly reducing inference costs and runtime.
3. Systematic Optimization Pipeline: The inclusion of built-in hyperparameter and prompt optimization tools addresses the issue of suboptimal manual tuning, allowing schemes to reach their true performance ceiling.
4. Empirical Validation: The authors re-implemented three popular schemes—Tree of Thoughts (ToT), Graph of Thoughts (GoT), and ProbTree—within FoT to demonstrate its universality and efficiency gains.

Experimental Results

The authors evaluated FoT across five tasks: Game of 24 (Go24), Sorting, Document Merging (DM), HotpotQA, and MuSiQue.

• Efficiency Gains:
  • Runtime: Parallel execution combined with persistent caching accelerated execution by 1.9x to 35.4x depending on the task and scheme. The average acceleration across all tasks was 10.7x compared to baseline sequential implementations.
  • Cost: Caching reduced inference costs by 14% to 46% on tasks with recurring patterns (Go24, DM, MuSiQue).
• Optimization Impact:
  • Using FoT's optimization tools, the authors improved task scores (accuracy or F1) while simultaneously reducing costs for all evaluated schemes.
  • Feasibility of Optimization: The optimization process itself is resource-intensive. However, with caching, the cost of optimization was reduced to 9%–36% of the cost without caching, and the duration was reduced by factors of 8.7x to 50.2x. This demonstrates that systematic optimization is often only viable when supported by efficient caching and parallelization.

Significance and Claims

The paper claims that FoT represents a shift from static, manually engineered reasoning flows to adaptive, dynamic reasoning systems.

• Adaptability: By allowing graph structures to evolve, FoT paves the way for reasoning schemes that can handle previously unseen or highly heterogeneous problem types without requiring manual re-engineering.
• Efficiency as a Prerequisite for Optimization: The authors emphasize that the "efficient-by-default" setup (parallelization + caching) is not merely an incremental improvement but a prerequisite for making the systematic optimization of reasoning schemes viable. Without these features, the time and budget required to tune prompts and hyperparameters are often prohibitive.
• Generalizability: The framework is designed to be open and modular, supporting any operation definable in Python (LLM calls, tool use, code execution), thereby facilitating the development of future dynamic and efficient reasoning schemes.

The authors conclude that while they implemented semi-automatic and manual schemes to demonstrate the framework, the true potential of FoT lies in enabling fully automatic schemes that dynamically derive reasoning structures for complex, real-world problems.