💬 NLP

StocksTalk: A Voice-Enabled Conversational Agent for Structured Query Generation over Web Data

StocksTalk is a transparent, voice-enabled conversational agent that transforms spoken financial requests into validated, executable SQL queries by combining streaming speech recognition, retrieval-augmented constraint extraction, and interactive human-in-the-loop verification to outperform baseline LLM approaches in accuracy and stability.

Original authors: Akshat Parmar, Vikranth Udandarao, Abhay Shakya, Tanmay Hire, Avinash Anand, Rajiv Ratn Shah, Daniel Wang Zhengkui

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Akshat Parmar, Vikranth Udandarao, Abhay Shakya, Tanmay Hire, Avinash Anand, Rajiv Ratn Shah, Daniel Wang Zhengkui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of finance, making a decision often feels like searching for a needle in a haystack, but the haystack is made of millions of numbers, and the needle is a specific pattern of growth or value. For decades, researchers have tried to teach computers to understand human language so that anyone could ask a question and get a precise answer from a database. This field, known as text-to-SQL, aims to translate the messy, flexible way people speak into the rigid, exact language that computer databases require. However, most of these systems have been tested in quiet, perfect conditions with written text. They struggle when faced with the reality of human speech, which is often filled with background noise, accents, and the natural tendency to speak in incomplete thoughts. In the high-stakes arena of investing, where a single misunderstood number can lead to a costly mistake, the gap between what a person says and what a computer executes is a dangerous chasm.

A team of researchers at IIIT-Delhi and the Singapore Institute of Technology has built a new system called StocksTalk to bridge this gap. They created a voice-enabled assistant designed specifically to turn spoken investment ideas into valid, working database queries. Unlike previous tools that might guess at a user's intent and hope for the best, StocksTalk is built to be transparent and cautious. It treats the process of finding stocks not as a single leap of logic, but as a series of careful steps where the computer checks its own work and invites the human to do the same. The system listens to a user speak, breaks the sentence down into specific financial rules, checks those rules against a real-time market database, and then shows the user exactly what it plans to do before it actually runs the search.

The researchers tested this system with a carefully prepared set of 150 spoken prompts, covering three different types of investment strategies: looking for fast-growing companies, finding stocks that pay regular dividends, and hunting for undervalued assets. They recorded these prompts in two very different environments: one in a quiet room and another in a noisy cafeteria to simulate real-world distractions. The results showed that while a standard, unassisted artificial intelligence model could generate a query about 81 percent of the time without crashing, it often produced answers that were logically inconsistent or financially nonsensical. By adding layers of verification and a method to retrieve specific financial definitions during the process, the system improved its ability to generate correct queries to nearly 98 percent. More importantly, the system's ability to stay on track during a back-and-forth conversation improved dramatically, jumping from about 51 percent accuracy to nearly 89 percent when the human was allowed to verify the steps along the way.

The core of the system's success lies in how it handles the uncertainty of human speech. When a user says something like "find large technology stocks with improving margins," the system does not immediately fire off a search. Instead, it first converts the voice into text, then uses a retrieval system to look up exactly what "improving margins" means in the context of the specific database it is using. It then constructs a draft query and displays it to the user on a dashboard. This dashboard shows the extracted rules, the mathematical operators it chose, and the final code it intends to run. The user can see if the computer misunderstood a number or a category and correct it before the search begins. This "human-in-the-loop" approach proved essential, particularly for complex requests involving multiple conditions, where the system alone would have made errors in nearly one out of every three attempts.

The study also highlighted that the biggest hurdle remains the quality of the voice recognition itself. When the audio was recorded in a noisy environment, the system's ability to correctly identify specific numbers and financial terms dropped significantly, leading to more errors in the final query. This suggests that while the logic and verification layers are robust, the initial step of hearing the user clearly is still the most fragile part of the chain. The researchers found that their method of breaking the problem into smaller, verifiable stages prevented the system from compounding small mistakes into large failures. For instance, in a multi-turn conversation where a user refines their search, the system without human verification would often lose track of the original constraints, whereas the interactive version maintained the correct path almost 90 percent of the time.

Looking ahead, the team envisions expanding this approach to include real-time news and earnings reports, allowing the system to answer questions like "which companies beat their profit estimates last quarter?" by pulling in fresh data as it arrives. They also plan to let the system compare new findings against a user's existing portfolio, enabling more personalized advice. The work demonstrates that for complex, high-stakes tasks, the most reliable path forward is not to build a machine that tries to be perfect on its own, but to build a machine that knows when to pause, show its work, and ask for help. By making the reasoning process visible and editable, StocksTalk turns a black box of artificial intelligence into a collaborative tool, ensuring that the final answer is not just a guess, but a verified fact ready for the real world.

Technical Summary: StocksTalk

Problem Statement

The paper addresses the challenge of mapping unconstrained, noisy spoken natural language into executable, structured SQL queries for financial screening. While text-to-SQL systems have advanced, they typically assume clean textual input and fixed schemas, failing to account for the specific complexities of real-world financial domains. These complexities include:

Noisy Input: Transcription errors and ambiguity inherent in spoken interaction.
Domain Specificity: The need to ground natural language in precise financial metrics (e.g., P/E ratio, debt-to-equity) and normalize units (percentages vs. absolute values).
Logical Consistency: The requirement to prevent logically inconsistent query constructions and align temporal qualifiers with available data fields.
Interpretability: The lack of transparency in existing "black-box" agents, which often fail to expose intermediate reasoning steps to the user.

The authors frame this as an interactive structured prediction problem under uncertainty, where the system must infer a valid SQL query aligned with a predefined schema (Screener.in) while preserving semantic intent.

Methodology

StocksTalk is a modular, Web-native pipeline composed of four integrated stages, designed to transform spoken queries into validated SQL queries with human-in-the-loop (HITL) verification.

1. Speech Interaction Layer

The system utilizes a streaming speech-to-text interface (ElevenLabs STT) to convert spoken utterances into structured text. This layer preserves discourse cues necessary for retrieval and tracks multi-turn dialogue states, producing a stable intermediate representation for downstream processing.

2. Retrieval-Augmented Intent Understanding

Transcribed utterances are processed by a Retrieval-Augmented Generation (RAG) engine built on GPT-4o.

Knowledge Base: A curated financial knowledge base containing metric definitions, operator conventions, and sector taxonomies for the Screener.in schema is retrieved at inference time.
Function: This layer grounds constraint extraction to reduce hallucinations, parses intent to extract constraints (valuation thresholds, sector preferences, growth filters), and retains context across multi-turn conversations.

3. Structured Query Induction

Extracted constraints are mapped into executable SQL-like queries via a constrained generation module.

Constrained Decoding: The system enforces logical and financial coherence, identifies relational operators, and prevents malformed structures.
Explanatory Synthesis: The generated SQL is exposed for auditability, allowing users to verify the query structure before execution.

4. Real-Time Web Data Integration & HITL Dashboard

The validated query is executed against the Screener.in live market data API. The system features an interactive dashboard (Flask/SSE) that visualizes all pipeline stages:

Transparency: Users view extracted constraints, normalized metrics, and the generated SQL.
Verification: Users can confirm or correct intermediate representations (specifically constraints) before the query is executed, implementing a human-in-the-loop verification step.

Key Contributions

The paper outlines three primary contributions:

A Modular Pipeline: A voice-driven financial query induction system offering full intermediate transparency, integrating streaming speech, RAG, constrained SQL generation, and rule-based validation.
FinScreenBench: A manually curated benchmark of 150 spoken financial screening prompts. The dataset spans three investment strategies (growth, dividend, value) and two input noise conditions (clean and noisy), with ground-truth SQL queries and high inter-annotator agreement ( $\kappa = 0.87$ ).
Empirical Evaluation: An analysis demonstrating that constrained decoding, RAG grounding, and interactive verification address distinct, non-overlapping failure modes in voice-driven text-to-SQL systems.

Results

The system was evaluated on the FinScreenBench dataset against three baselines: plain GPT-4o, GPT-4o + RAG, and GPT-4o + RAG + Validation (without HITL).

Performance Gains: The full StocksTalk pipeline achieved a 91.2% Constraint Extraction Accuracy (CEA) and 97.5% SQL Executability (EX) on clean inputs.
Logical Consistency: The addition of the validation layer provided the largest single gain in Logical Consistency Rate (+18.3 percentage points over the RAG-only baseline), reaching 93.8% for the full system.
Multi-Turn Stability: StocksTalk achieved 88.6% multi-turn stability, significantly outperforming the plain GPT-4o baseline (51.2%).
Noise Sensitivity: Performance degraded under noisy conditions, with CEA dropping by 12.8 percentage points and Multi-Turn Stability by 14.3 percentage points, highlighting ASR transcription errors as the dominant bottleneck.
Ablation Study: The study confirmed that no single component subsumes the others. RAG was critical for constraint extraction, constrained decoding for executability, and the validation layer for logical consistency. Removing HITL had a modest effect on single-turn executability but a significant negative impact on multi-turn stability.

Significance and Claims

The paper claims that StocksTalk demonstrates that reliable voice-driven financial query induction is achievable through a combination of RAG grounding, constrained SQL generation, rule-based validation, and human-in-the-loop verification.

The authors emphasize that the system's reliability stems not from a specific model choice (using GPT-4o as a backbone for all comparisons) but from the pipeline architecture. By exposing intermediate representations, the system allows users to intervene before errors propagate, effectively bridging the gap between unconstrained human intent and structured data systems. The authors position StocksTalk as a step toward general-purpose conversational financial analysis, noting that future work could extend the system to include news/earnings retrieval, portfolio-aware screening, and autonomous agentic workflows.