💬 NLP

Thinking Is Not Telling: Information Disclosure in User-Service LLM Agents

This paper identifies a critical communication failure in user-service LLM agents where internal reasoning does not automatically translate into timely user-facing information disclosure, demonstrating that explicitly externalizing decision-relevant knowledge is more effective for task success than simply increasing internal "think" efforts.

Original authors: Jiatong Li, Changdae Oh, Hyeong Kyu Choi, Jindong Wang, Sharon Li

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiatong Li, Changdae Oh, Hyeong Kyu Choi, Jindong Wang, Sharon Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a high-stakes video game with a teammate who can see the entire map, the hidden traps, and the secret rules, but you can only see the tiny square of the screen right in front of you. You are the player holding the controller, and your teammate is the "AI guide." In the world of computer science, these guides are called Large Language Model (LLM) agents. They are super-smart programs designed to help us do things like book flights, return items, or manage our schedules. Usually, we assume that if the AI is "thinking" really hard in its head, it will give us the best advice. But here's the catch: just because the AI is having a brilliant internal monologue doesn't mean it's actually telling you what it knows. If the AI sees a trap on the map but forgets to warn you before you jump, you're still going to fall. This paper asks a simple but crucial question: Is the AI's secret thinking actually helping us, or is it just keeping all the good info to itself?

The researchers behind this study decided to investigate a specific type of AI failure where the agent "thinks" but doesn't "tell." They looked at scenarios where an AI agent, a human user, and a digital environment (like a store's database or an airline's system) have to work together. The problem is that the AI often knows things the user doesn't, like "you can't cancel just one item" or "this flight is already full." The team wanted to see if forcing the AI to use a "think" tool—where it writes down its reasoning steps before speaking—actually made it a better partner. They tested this across five different AI models and three real-world service scenarios, like shopping and booking flights.

What they found was a bit of a surprise. They discovered that forcing the AI to "think" more didn't automatically make it better at the job. In fact, in many cases, the AI got worse! It turns out that the AI was using its "think" time to organize its own thoughts, but it wasn't sharing the most important parts of those thoughts with the user. It was like a chef who spends hours perfecting a secret recipe in the kitchen but serves the customer a dish without telling them it's spicy, even though the customer has a nut allergy. The AI was "thinking" about the allergy, but it failed to "tell" the customer.

To prove this, the researchers played a clever game of "what if." They took failed conversations where the AI messed up and secretly injected the missing information that the AI should have said. For example, if the AI forgot to mention a rule about cancellation fees, they added that rule back into the conversation. When they did this, the AI's next move became much more accurate. This showed that the real problem wasn't that the AI wasn't smart enough; it was that it wasn't disclosing the right information at the right time. The study suggests that the key to better AI isn't just making it think harder, but making sure it speaks up about the things the user needs to know to make a good decision.

The team also tested a simple fix: they gave the AI a tiny nudge in its instructions, telling it, "Hey, before you ask the user to decide, make sure you tell them the rules, the options, and what might happen." This small change worked wonders. It helped the AI recover from its "thinking" mistakes and actually improved its performance, even when it was forced to think internally. The bottom line is that for AI to be a true helper, it needs to stop keeping secrets and start sharing the facts that matter. It's not enough to be smart; you have to be a good communicator, too.

Technical Summary: Thinking Is Not Telling: Information Disclosure in User-Service LLM Agents

1. Problem Statement

Large Language Model (LLM) agents are increasingly deployed in user-service scenarios where task success relies on coordination between the agent, the user, and a stateful environment. Unlike static reasoning benchmarks, these interactions involve asymmetric information: the agent often possesses access to task policies, tool results, and environment states that the user cannot observe.

The paper identifies a critical communication-level failure mode: internal reasoning does not necessarily translate into timely, user-facing information disclosure. While agents are often equipped with a user-invisible "think" tool to perform internal reasoning before acting, this private processing frequently fails to externalize decision-relevant information to the user. Consequently, users may make consequential decisions (e.g., confirming an action, selecting an option) under incomplete context, leading to task failure. The authors argue that the gap lies not in the agent's ability to reason, but in its failure to convert that reasoning into Information Disclosure (InfoDis)—the proactive communication of state corrections, constraints, options, consequences, and retrieved environment data.

2. Methodology

2.1. Empirical Validation of the "Think" Tool

The authors first evaluated the utility of enforced internal thinking across five models (GPT-4o, GPT-4.1, GPT-5, DeepSeek-V3.1, gpt-oss-120b) and three user-service scenarios (Retail, Airline from $\tau$ -Bench, and TS-Phone from ToolSandbox).

Setup: They compared agents with and without an enforced "think" tool before each step.
Metric: Task reliability measured by $\text{Pass}^k$ (the probability that $k$ independent trials succeed).
Finding: Enforced thinking did not consistently improve performance; in several cases (e.g., GPT-4.1 on Retail), it significantly degraded success rates. Furthermore, increasing "think effort" (low, medium, high) did not reliably scale the length or informativeness of user-facing responses.

2.2. Defining and Probing Information Disclosure

To quantify the missing communication, the authors defined Information Disclosure (InfoDis) as the agent's proactive communication of decision-relevant information the user likely lacks. They operationalized this into five binary categories (Table 2):

State Correction (SC): Correcting user misconceptions about the current state.
Option (OP): Presenting available choices.
Constraint (CN): Disclosing rules, policies, or eligibility limits.
Consequence (CQ): Explaining outcomes (financial, timing, irreversible effects).
Proactive Retrieval (PR): Sharing environment data fetched by the agent but not explicitly requested.

2.3. Evidence-Grounded Probing and Counterfactual Intervention

The study employed a multi-stage pipeline to analyze and intervene in failed trajectories:

Evidence-Grounded Annotation: Annotators labeled InfoDis categories based on exact text spans in user-visible responses, masking internal "think" traces to ensure objectivity.
Counterfactual Intervention: For failed trajectories, the authors identified the last user decision point before the first environment-edit action. They compared these with matched successful ("golden") trajectories.
Injection Strategy: Missing InfoDis evidence from the golden trajectory was injected into the failed trajectory. This was compared against two controls:
- +Verbosity: Increasing response length without adding specific missing evidence.
- +Think Content: Injecting raw text from the agent's internal "think" trace.
Metrics:
- First Edit Correction Rate (FECR): The fraction of interventions where the first environment-edit action was corrected to match the golden trajectory.
- Recovery Rate (RR): The fraction of interventions where the final task outcome became successful.

2.4. Prompt-Level Intervention

Finally, the authors tested a lightweight InfoDis-aware prompt added to the system prompt, instructing agents to disclose decision-relevant states, constraints, and consequences before user decision points.

3. Key Results

3.1. Internal Thinking vs. User Communication

Enforced thinking showed unstable utility, often failing to improve task success or reliability.
Increased internal thinking effort did not correlate with richer user-facing responses; in some cases, response length decreased despite enforced thinking.

3.2. Causal Impact of Information Disclosure (RQ1)

Golden Info Injection produced the clearest improvement in FECR (micro-averaged 11.74%), significantly outperforming both +Verbosity (6.08%) and +Think Content (6.50%).
This indicates that missing decision-relevant disclosure causally affects the agent's immediate downstream environment-edit behavior.
Injecting raw "think" content was less effective than injecting targeted golden information, suggesting that internal reasoning traces do not automatically contain or structure the information needed for user disclosure.

3.3. Missing Dimensions in Failures (RQ2)

Analysis of intervention frequency revealed that State Correction (SC) and Constraint (CN) were the most frequently missing categories in failed trajectories.
Option (OP) was the least frequently missing.
This suggests failures often stem from users being unaware of current state realities or operational constraints (e.g., "you cannot cancel just one item").

3.4. Prompt-Level Improvements (RQ3)

Adding an InfoDis-aware prompt improved overall performance for agents without the "think" tool.
Crucially, it mitigated the performance degradation observed in enforced-think agents. For example, GPT-4o's micro $\text{Pass}^1$ on enforced-think tasks improved from 52.32% to 57.58% with the InfoDis prompt.
The gains were particularly pronounced in the Retail scenario.

4. Contributions and Significance

The paper makes three primary contributions:

Reframing the Problem: It identifies missing information disclosure as a distinct reasoning–communication problem, arguing that internal reasoning is insufficient when user decisions depend on information held by the agent.
Methodological Framework: It introduces an evidence-grounded framework for probing InfoDis in natural dialogues and uses counterfactual intervention to establish the causal link between disclosure and downstream task dynamics.
Practical Intervention: It demonstrates that disclosure-aware prompting is a lightweight, training-free method to improve agent performance and mitigate the negative effects of enforced thinking.

Significance:
The authors conclude that the central challenge for user-service agents is not merely enhancing internal reasoning capabilities, but ensuring that private reasoning and environment knowledge are externalized as timely, user-facing information disclosure. The study suggests that future agent designs must optimize not just how agents think, but what they disclose before user decision points to establish sufficient common ground for collaborative task completion.