Sophrosyne: Agentic Exploration of Relational Data Systems Needs Moderation

Technical Summary: Sophrosyne – Agentic Exploration of Relational Data Systems Needs Moderation

Problem Statement
Text2SQL agents powered by Large Language Models (LLMs) increasingly rely on tool calls to explore data systems before formulating SQL queries. To ensure security and scoped access, data systems expose these agents via explicit API surfaces. The authors identify a fundamental tradeoff in the design of these APIs:

• Coarse-grained APIs: Provide the entire database schema upfront. While this gives the agent full visibility, it incurs prohibitive token costs as the agent must process irrelevant schema elements for every query.
• Fine-grained APIs: Allow progressive exploration (e.g., listing tables first, then inspecting specific schemas). While token-efficient, the lack of global schema visibility forces agents to "over-explore." Agents, unable to know a priori which tables are relevant, often inspect irrelevant tables and incorporate them into query construction, leading to inaccurate SQL generation.

The paper posits that the majority of modern data systems (e.g., Amazon Redshift, GCP BigQuery, Supabase) expose fine-grained APIs, inadvertently making agents prone to over-exploration and reduced accuracy.

Methodology: The Sophrosyne System
The authors propose Sophrosyne, a data system environment designed to mitigate over-exploration within fine-grained API contexts without sacrificing the cost benefits of progressive exploration.

• Core Mechanism (Directives): Sophrosyne augments the standard API response (specifically the list_tables call, which is typically the agent's first tool call) with directives. These directives explicitly inform the agent which tables are likely irrelevant to the current natural language query, thereby constraining its future exploration.
• Implementation:
  • Offline Phase: A hybrid search index (HIcol) is constructed over individual column names. This index combines an IVF index for semantic similarity and a BM25 index for keyword matching, using results fused via reciprocal rank fusion.
  • Online Phase: When an agent requests a list of tables, Sophrosyne:
    1. Infers potential relevant column names from the user's query (using an inexpensive model).
    2. Searches the HIcol index with these inferred names to identify the most relevant tables.
    3. Computes the set of irrelevant tables (those not in the search results).
    4. Returns the standard table list alongside a directive listing the irrelevant tables.
• Evaluation Setup: The system was evaluated using an MCP server for SQLite exposing a fine-grained API. The agent (based on OpenCode) was tested on 157 queries from the BIRD LiveSQLBench benchmark using three models: GPT-5.4-mini, GPT-5.4, and Claude Sonnet 4.5.
• Baseline Comparisons: The study compared three configurations:
  1. No Directives: Standard fine-grained API exposure.
  2. Sophrosyne: Uses the computed (imperfect) directives.
  3. Sophrosyne-Oracle: Uses perfect directives derived from ground-truth SQL queries (representing an upper bound on performance).

Key Results

• Reduction in Over-Exploration: The introduction of directives significantly reduced the number of exploratory tool calls. Sophrosyne reduced over-exploration by up to 4.6× compared to the baseline. The Oracle variant reduced it even further.
• Accuracy Improvements: By preventing agents from incorporating irrelevant tables into their logic, Sophrosyne improved SQL execution accuracy. The best results showed an accuracy boost of up to 12.4% (approximately 4 percentage points) over the baseline.
• Token Efficiency: While fine-grained exploration is already more efficient than coarse-grained, Sophrosyne further reduced token consumption by limiting the schema elements the agent processes. For example, on Sonnet-4.5, input tokens were reduced from 8.74M to 8.00M.
• Cost Analysis: The overhead for computing directives (inferring columns and embedding them) was approximately 8 cents per run. For most models (Sonnet-4.5 and GPT-5.4), the savings from reduced token usage outweighed this overhead. For GPT-5.4-mini, the overhead slightly exceeded the base cost savings, though accuracy still improved.

Significance and Claims
The paper claims to be the first study to categorize data system API surfaces into coarse- and fine-grained types and to identify over-exploration as a critical symptom of the tradeoff inherent in fine-grained API design.

The authors argue that curbing over-exploration is essential for the effective use of fine-grained APIs. Sophrosyne demonstrates that data system environments can actively guide agent behavior through lightweight directives, balancing the need for secure, scoped access with the need for accurate query generation. The work suggests that future data agents and systems should not just passively expose APIs but should actively provide context (like directives) to moderate agent exploration, a concept the authors name "Sophrosyne" (the Greek virtue of prudence and moderation).

The paper concludes that while perfect directive computation is challenging, even imperfect directives yield significant gains, and future work could extend this to multi-source environments and performance hinting.