💻 computer science

Tunable Tool-Call Rates in LLM Agents via Representation Steering

This paper demonstrates that tool-call rates in LLM agents can be precisely controlled at inference time without retraining by applying a single, training-free linear steering direction extracted from the model's residual stream, which monotonically adjusts call propensity to optimize the accuracy-cost trade-off across diverse architectures and tool types.

Original authors: Yuqi Chen, Vincent Siu, Yang Liu, Dawn Song, Chenguang Wang

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuqi Chen, Vincent Siu, Yang Liu, Dawn Song, Chenguang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Large language models are powerful engines of text, capable of answering questions, writing stories, and solving problems. Yet, these systems operate with a fundamental limitation: they rely entirely on the information stored in their internal memory. When a question requires facts that are too obscure, too recent, or too complex to be memorized, the model often guesses with dangerous confidence, producing answers that sound plausible but are entirely wrong. To fix this, developers have taught these models to use external tools, such as web search engines or calculators, to look up information before answering. However, teaching a model when to stop and reach for a tool is surprisingly difficult. If the model calls a tool too often, it wastes time and money, and risks triggering unwanted actions. If it calls a tool too rarely, it misses the chance to find the correct answer and remains confidently mistaken.

Researchers have long sought ways to make these decisions more reliable, usually by retraining the model with new examples or by carefully rewriting the instructions given to it. These methods are expensive and rigid; once a model is trained or prompted a certain way, changing its behavior requires starting the process over. A new study from researchers at UC Santa Cruz and UC Berkeley suggests a different approach. They discovered that the decision to use a tool is not a complex, hidden process that requires retraining to change. Instead, it is controlled by a single, simple signal within the model's internal workings that can be adjusted in real time, like turning a dial, without altering the model's training or its instructions.

The researchers began by observing how a language model behaves when it is presented with a list of available tools, such as a search engine, a calculator, and a coding interpreter. They noticed that before the model actually speaks or writes a response, it generates a specific internal signal that indicates whether it intends to use any tool at all. By comparing the internal state of the model when it decides to use a tool versus when it decides to answer from memory, the team identified a specific direction in the model's mathematical space that represents the "urge" to call a tool. This direction is not tied to a specific tool like a calculator or a search engine; rather, it is a general signal that simply means "use a tool."

To test this discovery, the researchers did not retrain the model or change its prompts. Instead, they took this identified direction and added it to the model's internal processing during the generation of an answer. By adding a small amount of this signal, they could encourage the model to use tools more often. By subtracting it, they could suppress the urge to use tools entirely. They found that this adjustment worked with remarkable precision. As they increased the strength of the signal, the rate at which the model called tools rose smoothly from nearly zero to almost one hundred percent. Crucially, the model did not start calling tools randomly. When the researchers nudged the model to call tools more often, it specifically targeted the questions it could not answer on its own, such as obscure historical facts or complex calculations, while ignoring questions it already knew the answer to.

The study also revealed that this control mechanism is highly adaptable. The researchers extracted the "tool-use" signal from a setup containing three specific tools and then applied it to a completely different set of tools, including weather forecasts, stock market lookups, and email sending. The same signal successfully induced the model to use these new tools with the same effectiveness as if it had been trained specifically for them. Furthermore, this method worked across a wide variety of different model architectures, from smaller, standard models to massive, complex systems used in industry. In every case, the single signal allowed the researchers to dial the tool-calling behavior up or down without breaking the model's ability to form correct sentences or select the appropriate tool once a call was made.

When the researchers tested this method in a live environment where the model could actually perform web searches, the results were striking. On a set of difficult factual questions, the unadjusted model answered correctly only about 29 percent of the time, often guessing when it should have searched. By applying the steering signal to encourage tool use, the accuracy jumped to 56 percent, nearly doubling the model's performance. This improvement came with a manageable cost, as the model made roughly one search call per question. The researchers were able to trace a clear path between the cost of using tools and the accuracy of the answers, showing that operators could choose a specific balance that suited their needs.

This work challenges the assumption that controlling an AI's behavior requires heavy retraining or complex prompt engineering. It demonstrates that a critical decision-making process—knowing when to reach for external help—is encoded in a simple, linear direction within the model's internal state. By finding and adjusting this direction, researchers can make AI agents more reliable and efficient without changing their fundamental training. The method offers a lightweight way to tune the cost and accuracy of AI systems, ensuring they use their tools only when necessary and avoid the pitfalls of both over-reliance and dangerous guessing. While the technique does not fix errors in how a tool is executed, it provides a powerful new lever for managing the most basic and costly choice an AI agent makes: whether to act on its own or to seek help.

Technical Summary: Tunable Tool-Call Rates in LLM Agents via Representation Steering

Problem Statement

Large Language Model (LLM) agents face a critical trade-off in deciding whether to invoke external tools. Incorrect decisions are costly:

Under-use: Agents fail to call tools when necessary, leading to confident but incorrect answers on questions requiring external knowledge (e.g., long-tail facts).
Over-use: Agents call tools unnecessarily (e.g., for simple arithmetic), incurring latency, financial costs, and potential irreversible side effects (e.g., sending emails).

Existing mitigation strategies, such as fine-tuning or prompt engineering, are computationally expensive and difficult to modify dynamically at inference time. Furthermore, while prior work has shown that which tool to select can be represented linearly, it remains unclear whether the binary decision of whether to call a tool at all possesses a similarly readable and steerable internal representation.

Methodology

The authors propose a training-free, inference-time intervention method based on representation steering. The pipeline consists of three main components:

1. Propensity Proxy

Instead of relying on full generation, the method extracts a "tool-use propensity" signal from a single forward pass. In a multi-tool harness, the model's tool-calling format begins with a special token ( $t^\star$ , e.g., <tool_call>) shared by all available tools. The propensity score $s(q)$ for a query $q$ is defined as the log-probability of emitting this token at the first generation position:
$s(q) = \log p_\theta(t^\star | x(q))$
This metric serves as a faithful ranking signal for real tool use without requiring actual generation.

2. Direction Extraction (Difference-of-Means)

Using a multi-tool environment (neutral system prompt, list of available tools), the authors:

Rank queries by $s(q)$ .
Form a high-propensity set ( $H$ ) and a low-propensity set ( $L$ ).
Compute the steering vector $v_\ell$ at each residual stream layer $\ell$ as the difference of means between the two sets:
$v_\ell = \frac{1}{|H|} \sum_{q \in H} h_\ell(q) - \frac{1}{|L|} \sum_{q \in L} h_\ell(q)$
This vector captures a general "call-propensity" signal rather than a tool-specific one, as the high-propensity set contains a mix of question types.

3. Inference-Time Steering

During inference, the extracted vector is added to the residual stream at a specific decoder layer $\ell$ :
$h'_\ell = h_\ell + \alpha v_\ell$
where $\alpha$ is a scalar steering coefficient.

$\alpha > 0$ : Encourages tool calls.
$\alpha < 0$ : Suppresses tool calls.
$\alpha = 0$ : Recovers the unmodified model behavior.

The authors also test projection interventions (clamping and ablation) to verify that the projection onto this direction causally controls the decision.

Key Contributions

Identification of a Linear Control Direction: The paper identifies a single linear direction in the residual stream that controls the binary decision of whether to call a tool. This allows for continuous, inference-time control without retraining or prompt modification.
Knowledge-Selectivity and Tool-Generality:
- Knowledge-Selective: Induced calls concentrate on questions the model cannot answer from parametric knowledge (e.g., low-popularity entities), rather than on questions it already knows.
- Tool-General: A direction extracted from a multi-tool harness effectively steers tools not present in the extraction set (e.g., translation, weather, SQL) with strength comparable to vectors extracted specifically for those tools.
Disentanglement of Decisions: The steering direction controls whether to call a tool without substantially altering which tool is selected. The routing distribution across tools remains model-dependent and largely preserved.
Cost-Accuracy Pareto Frontier: With live tool execution, varying the steering strength traces a smooth Pareto frontier, allowing operators to tune the trade-off between cost (number of calls) and accuracy.

Experimental Results

The method was evaluated across five models (Qwen3 variants, Gemma-4, gpt-oss) spanning dense, Mixture-of-Experts (MoE), and multimodal architectures, using datasets like PopQA (open-domain QA), GSM8K (math), and BIG-Bench Hard (string/date manipulation).

Monotonic Control: Varying $\alpha$ from $-2$ to $+3$ moves the tool-call rate monotonically from near 0% to over 90% (or 100% for some models) across different tool environments (search, calculator, Python).
Accuracy Improvement: On the PopQA dataset with live web search, steering increased open-domain QA accuracy from 0.29 (baseline) to 0.56 at approximately 1.1 searches per question.
Cross-Model Generalization: The same extraction and steering procedure worked across all tested models, regardless of their baseline tendency to under-use or over-use tools.
Layer Specificity: The steering effect is strongest and most monotone in a mid-to-late band of decoder layers (e.g., layers 21–24 for Qwen3-4B). Early layers are inert, and very late layers show erratic behavior.
Projection Interventions: Clamping the projection of the residual stream onto the steering vector also monotonically increased tool use, confirming the causal link, though additive steering provided a wider control range.

Significance and Claims

The paper claims to provide a lightweight, training-free mechanism for controlling agent behavior that addresses the "black box" nature of tool-use decisions. By demonstrating that the decision to call a tool is encoded in a single, steerable linear direction, the authors show that:

Agents can be tuned to avoid costly, unnecessary calls or to retrieve missing information without retraining.
This control is robust across diverse model architectures and tool sets.
The method enables the dynamic optimization of the cost-accuracy trade-off in real-world deployments.

The authors note limitations: the method controls the decision to call, not the quality of execution; for reasoning-first models, the reasoning span must be bypassed to access the decision layer; and excessive positive steering can corrupt the tool-call format. Nevertheless, the work establishes a new paradigm for inference-time control of agentic behaviors.