💬 NLP

Eta Given Delta: Defining LLM Tool Efficiency With Marginal Tool Utility

This paper introduces "tool efficiency" and "marginal tool utility" as new quantitative metrics to directly evaluate and optimize the rate of useful tool calls in LLM agent trajectories, aiming to guide the creation of leaner tool suites that complement traditional accuracy-based evaluations.

Original authors: Nyx Iskandar

Published 2026-07-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Nyx Iskandar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers don't just chat, but actually do things. This is the realm of AI Agents. Think of a standard Large Language Model (LLM) like a brilliant but bookish student who knows a lot of facts but can't leave the library. To make this student useful in the real world, we give them a set of "tools"—like a calculator, a search engine, or a code editor—so they can solve problems, fix bugs, or find information. This is the exciting frontier of modern AI: turning a passive brain into an active worker.

However, there's a catch. Just because you have a toolbox doesn't mean every tool is helpful. Sometimes, grabbing a hammer when you need a screwdriver just wastes time and confuses the worker. In the past, scientists mostly cared if the final job got done correctly (the "accuracy"). But they rarely asked: "How many wrong turns did the robot take to get there?" or "Did it waste time using tools that didn't help?" This paper steps into that gap, asking a simple but powerful question: How efficient is the AI's tool usage? The authors want to measure not just if the AI succeeded, but how well it used its tools to get there, hoping to build leaner, faster, and smarter AI workers for the future.

The "Useless Tool" Detective

In this paper, the authors introduce a new way to look at how AI agents use their tools. They call their new metric "Tool Efficiency." To understand this, imagine you are watching a detective solve a mystery. The detective has a bag of gadgets: a magnifying glass, a fingerprint kit, a walkie-talkie, and a crystal ball.

If the detective uses the magnifying glass to find a clue, that's a useful move. If they use the crystal ball to guess the answer and it turns out to be wrong, that's a waste. The authors realized that while we usually just check if the detective solved the case, we don't usually count how many times they grabbed the wrong gadget. They wanted a way to count the "good" tool uses versus the "bad" ones.

The Secret Ingredient: Marginal Tool Utility

To figure out if a tool use was good or bad, the authors invented a concept called Marginal Tool Utility. This sounds fancy, but it's actually quite simple. It asks: "Did this specific tool call help the AI get closer to the right answer, or did it just add noise?"

To test this, they used a clever trick called "LLM-as-a-Judge." Imagine a super-smart referee watching the detective's every move. After the detective uses a tool, the referee looks at the situation before the tool was used and after it was used.

If the situation looks clearer and the detective is more likely to solve the case, the referee gives a thumbs up (positive utility).
If the detective just got confused or went in circles, the referee gives a thumbs down (non-positive utility).

The authors found that they didn't need to know the exact math of how much better things got; they just needed to know the sign (positive or negative) of the change. This allowed them to label every single tool call in an AI's journey as either "useful" or "not useful."

The Experiment: Cleaning Out the Clutter

To prove their idea works, the authors set up a real-world test using a benchmark called APEX-SWE Observability. This is a tough challenge where AI agents have to fix bugs in computer code by looking at logs and talking to other systems.

The agents were given a "tool suite" containing:

Grafana/Loki: A tool for reading system logs (like reading the crime scene photos).
Mattermost: A tool for reading team chat messages (like reading the suspects' gossip).
Plane: A tool for reading project tickets (like reading the case file).

The authors suspected that reading the actual logs (Grafana/Loki) would be super helpful, but reading the chats and tickets might just distract the AI. To test this, they ran the same 25 difficult tasks three times with different tool sets:

The Full Kit: All three tools available.
The Log-Only Kit: Only the log tool (Grafana/Loki).
The No-Extra-Tools Kit: No chat or ticket tools at all.

What They Found

The results were a clear confirmation of their hypothesis.

1. Accuracy didn't drop when they removed the "chatter."
When they took away the chat (Mattermost) and ticket (Plane) tools, the AI's success rate stayed roughly the same. In fact, for one model (GPT-5.3-Codex), accuracy actually went up slightly from 0.32 to 0.36 when the extra tools were removed. This proved that those extra tools weren't helping; they were just clutter.

2. The "Log" tool was the real hero.
When they removed the log tool (Grafana/Loki), the AI's performance crashed. Accuracy dropped to 0.24 for the GPT model and 0.20 for the Gemini model. This confirmed that the log tool was essential, while the others were just "dead weight."

3. Efficiency skyrocketed when they cleaned up.
This is where the new metric shined. By removing the useless tools, the "Tool Efficiency" jumped dramatically.

For the GPT model, efficiency went from 0.359 (about 36% of tool calls were useful) in the full kit to 0.720 (72% useful) in the log-only kit.
For the Gemini model, it went from 0.367 to 0.593.

In plain English: When the AI stopped wasting time checking chat messages and project tickets, it spent almost twice as much of its time doing actually helpful things.

The "Middle Call" Mystery

The authors also noticed something funny about when the AI made mistakes. They found that the AI often started strong, using the right tools (like reading logs) at the beginning. But in the middle of its work, it would often get distracted, using tools that didn't help (like reading chats) and wasting steps. It was like a detective who starts by looking at the crime scene, then spends an hour reading the suspect's diary, and only then goes back to the scene. The authors suggest that future AI systems could be programmed to recognize this "middle slump" and stop using tools that aren't helping, essentially teaching the AI to "backtrack" before it wastes too much time.

Why This Matters

The paper concludes that we shouldn't just build AI agents with every tool under the sun. Instead, we should use these new metrics to build "lean" tool suites. Just like a carpenter doesn't carry a hammer, a saw, and a drill to fix a loose screw, an AI shouldn't carry tools it doesn't need.

The authors suggest that by measuring Marginal Tool Utility and Tool Efficiency, developers can create AI agents that are not only accurate but also faster and cheaper to run. They argue that this is a crucial step toward making AI agents that are truly efficient, rather than just "smart but clumsy." They don't claim to have solved everything, but they have handed the community a new ruler to measure how well our AI workers are actually working.

Technical Summary: Eta Given Delta – Defining LLM Tool Efficiency With Marginal Tool Utility

Problem Statement

While Large Language Models (LLMs) are increasingly equipped with tools to function as agents, current evaluation paradigms rely almost exclusively on outcome accuracy as a proxy for tool use efficiency. This approach fails to directly measure the rate of useful tool calls within an agent's trajectory. Consequently, there is a lack of quantitative metrics to determine whether specific tool calls contribute to task success or if they are redundant, potentially leading to bloated tool suites that do not improve accuracy but increase inference costs and latency. The paper identifies a gap in the literature: the absence of a direct, quantitative method to measure tool efficiency (the rate of useful tool calls) and to identify which specific tool calls can be safely removed without degrading performance.

Methodology

1. Defining Marginal Tool Utility ( $\Delta\alpha$ )

The authors introduce Marginal Tool Utility as a metric defined per individual tool call ( $\alpha_i$ ) within an agent trajectory.

Definition: $\Delta\alpha(\alpha_i)$ represents the difference in the likelihood of the task being solved correctly given tool calls up to and including $\alpha_i$ versus those excluding $\alpha_i$ .
Sign Determination: Instead of computationally expensive policy rollouts, the paper utilizes an LLM-as-a-Judge approach. A judge LLM ( $\pi^{(J)}$ $π^{(J)}$ ) analyzes the trajectory before and after a specific tool call to determine the sign of $\Delta\alpha(\alpha_i)$ $Δ α (α_{i})$ :
- $\Delta\alpha(\alpha_i) > 0$ : The tool call is useful.
- $\Delta\alpha(\alpha_i) \leq 0$ : The tool call is non-useful (safe to remove).
Aggregate Utility: By summing $\Delta\alpha$ across all calls of a specific tool, the paper determines if the tool itself is necessary for the agent's accuracy.

2. Defining Tool Efficiency ( $\eta_\alpha$ )

Based on the signs of marginal tool utility, Tool Efficiency is defined as the ratio of useful tool calls to the total number of tool calls in a trajectory:
$\eta_\alpha(\tau) = \frac{|\{\alpha_i | \Delta\alpha(\alpha_i) > 0\}|}{N}$
where $N$ is the total number of tool calls. The paper hypothesizes that removing tools with non-positive aggregate utility will increase $\eta_\alpha$ without negatively impacting accuracy.

3. Experimental Setup

Benchmark: The study uses the APEX-SWE Observability benchmark, which contains 25 public tasks where frontier models struggle to achieve >40% accuracy.
Agent Harness: The default harness includes native tools (bash, search, etc.) and three read-only MCP tools: Grafana/Loki (logs), Mattermost (discussions), and Plane (tickets/specs).
Ablation Study: Three variants were tested:
1. Default: All tools included.
2. Grafana: Only Grafana/Loki included.
3. No-MCP: No MCP tools included.
Models: Experiments were run using GPT-5.3-Codex and Gemini 3.1 Pro, generating 150 total trajectories.
Judge Model: GPT-5.4 was used as the LLM-as-a-Judge to classify the utility of each tool call.

Key Results

Task Accuracy

Removal of Redundant Tools: Removing Mattermost and Plane (comparing Default vs. Grafana) resulted in no significant decrease in task accuracy. In some cases, accuracy slightly improved or remained stable.
Removal of Necessary Tools: Removing Grafana/Loki (comparing Grafana vs. No-MCP) caused a noticeable decrease in accuracy (e.g., dropping from 0.36 to 0.24 for GPT-5.3-Codex).
Conclusion: Grafana/Loki is a necessary tool for these observability tasks, while Mattermost and Plane are redundant.

Marginal Tool Utility

Aggregate Utility: The LLM-as-a-Judge confirmed the hypothesis. Grafana/Loki showed positive aggregate utility ( $\sum \Delta\alpha > 0$ ), while Mattermost and Plane showed non-positive aggregate utility ( $\sum \Delta\alpha \leq 0$ ).
Call Patterns: Analysis of the judge's rationales revealed that:
- Useful calls ( $\Delta\alpha > 0$ ) often occurred early in the trajectory, providing concrete error logs.
- Non-useful calls ( $\Delta\alpha \leq 0$ ) frequently occurred in the middle of trajectories, often involving generic queries, noisy context, or retries on malformed queries.

Tool Efficiency

Efficiency Gains: Removing non-useful tools (Mattermost and Plane) significantly increased tool efficiency.
- GPT-5.3-Codex: Efficiency rose from 0.359 (Default) to 0.720 (Grafana).
- Gemini 3.1 Pro: Efficiency rose from 0.367 (Default) to 0.593 (Grafana).
This demonstrates that a leaner tool suite can maintain accuracy while drastically improving the ratio of useful actions.

Key Contributions

Marginal Tool Utility ( $\Delta\alpha$ ): A new metric defined per tool call to determine if a specific invocation increases the likelihood of task success.
Aggregate Tool Utility: A method to predict whether a specific tool affects agent accuracy when ablated, based on the sum of its call-level utilities.
Tool Efficiency ( $\eta_\alpha$ ): A quantitative metric representing the rate of useful tool calls in a trajectory.
LLM-as-a-Judge Implementation: A cost-efficient method to determine the sign of $\Delta\alpha$ without requiring repeated policy rollouts, validated through empirical ablation studies.

Significance and Claims

The paper claims that its primary contribution is establishing a direct, quantitative framework for measuring tool efficiency, distinct from outcome accuracy. By introducing marginal tool utility, the authors provide a mechanism to:

Design Leaner Tool Suites: Enable application-layer engineers to identify and remove redundant tools that do not contribute to accuracy, thereby reducing inference costs and complexity.
Optimize Agent Harnesses: Move beyond "one-size-fits-all" tool configurations by tailoring tool suites to specific backbone models and tasks based on empirical utility data.
Inform Future Research: Serve as a springboard for new benchmark designs and agent engineering that optimize for efficiency metrics complementary to accuracy.

The authors emphasize that while prior work has focused on improving tool use via reinforcement learning or retrieval, this work focuses on post-hoc trajectory analysis to derive actionable metrics for engineering leaner, more maintainable agent systems. The findings suggest that intuitive judgments about tool necessity align with the proposed quantitative metrics, validating the approach for practical deployment.