💻 computer science

NTILC: Neural Tool Invocation via Learned Compression

NTILC is a neural framework that replaces linear in-context tool registry lookups with learned latent retrieval and a signature-aware composite objective, achieving over 95% context token reduction and up to 74% lower latency while maintaining high selection accuracy for agentic tool-calling models.

Original authors: Andrew Krikorian, Yayuan Li, Jason J. Corso

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Andrew Krikorian, Yayuan Li, Jason J. Corso

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a highly skilled personal assistant (the AI) who needs to get things done for you. To do this, you have a massive phone book (the Tool Registry) containing thousands of different services: weather reports, calculators, file editors, web searchers, and more.

The Old Way: The "Read-Everything" Approach

Currently, when you ask your assistant a question like, "What's the weather in Ann Arbor?", the assistant has to read the entire phone book out loud before it can answer.

The Problem: If your phone book has 5,000 entries, that's a lot of reading! It takes a long time (latency), costs a lot of money (token usage), and the assistant gets confused by all the irrelevant information (like reading about "file editing" when you just want the weather).
The Result: As the phone book grows, the assistant gets slower and makes more mistakes because it's overwhelmed by the noise.

The New Way: NTILC (The "Smart Index" Approach)

The paper introduces NTILC, a new system that changes how the assistant finds the right tool. Instead of reading the whole phone book, NTILC uses a smart, pre-made index (like a digital library catalog).

Here is how it works, step-by-step:

The Request: You ask, "What's the weather?"
The Translation: Instead of reading the whole book, the system instantly translates your question into a "secret code" (an embedding) that represents the intent of your request.
The Match: It compares your secret code against a pre-computed list of secret codes for every tool in the registry. It finds the closest match (the weather tool) in a split second.
The Action: The assistant only looks at the specific page for the weather tool to get the details it needs to give you an answer.

The Magic Trick: The assistant never has to read the 5,000-page phone book. It only reads the one page it needs. This cuts the "reading time" (context usage) by over 95% and makes the process much faster.

The Tricky Part: "Semantic Blur"

There is a specific problem the paper solves. Imagine two tools in your phone book:

Tool A: "Get Weather" (requires a zip code).
Tool B: "Get Weather" (requires latitude and longitude).

They sound exactly the same! If the assistant just looks at the names, it might pick the wrong one. If you give it a zip code, but it picks the tool that needs latitude/longitude, the tool will fail. The paper calls this "Semantic Blur"—when tools look the same but work differently.

The Solution: The "Functional Margin"
NTILC uses a special training method called Functional Margin Loss. Think of this as a strict teacher who doesn't just teach the assistant to recognize the name of the tool, but also to feel the shape of the tool.

Even if two tools have the same name, if their "shapes" (the arguments they need) are different, the system pushes them far apart in its memory.
This ensures that when you ask for "Weather with a zip code," the system knows exactly which of the two "Weather" tools to pick, avoiding the wrong one.

The Results

The paper tested this on several public datasets (like ToolBench and API-Bank) and found:

Speed: It is up to 74% faster than the old method.
Efficiency: It reduces the amount of "reading" the AI has to do by over 95%.
Accuracy: It is just as accurate as the old method, even when the phone book gets huge (thousands of tools).

Summary

NTILC is like upgrading from a librarian who reads every single book in the library to find your answer, to a librarian who has a perfect, instant index. They know exactly which book you need without ever opening the others, saving time and preventing confusion, even when the library grows to millions of books.

Technical Summary: NTILC (Neural Tool Invocation via Learned Compression)

1. Problem Statement

Agentic systems relying on Large Language Models (LLMs) increasingly depend on extensive registries of callable APIs, functions, and local actions. The standard approach, In-Context Tooling (ICT), inserts the full text of every available tool definition into the model's prompt before processing a user query. While effective for small registries, this method incurs a token cost that scales linearly ( $O(N)$ ) with the number of tools.

As registries grow (e.g., to 5,000+ tools), ICT faces three critical limitations:

Context Exhaustion: Tool schemas (averaging ~120 tokens) rapidly consume the context window, leaving little room for task-specific information.
Latency and Cost: Processing massive context windows increases prefill time and inference costs.
Selection Accuracy Degradation: The presence of numerous irrelevant tools ("distractors") and semantic blur—where tools share nearly identical natural language descriptions but possess incompatible execution signatures (e.g., different argument types)—leads to higher error rates in tool selection.

2. Methodology: NTILC

The authors propose NTILC (Neural Tool Invocation via Learned Compression), a framework that replaces linear-scaling in-context registry scanning with an external, learned latent retrieval step.

2.1 Inference Pipeline

Instead of injecting the full registry into the prompt, NTILC operates as follows:

Intent Generation: A backbone LLM ( $M$ ) receives the user query ( $\chi$ ) and generates a plan block ( $P$ ) containing natural language tool intents.
Encoding: A dedicated encoder ( $E$ ) maps these intents into a trained embedding space.
Latent Retrieval: A nearest-neighbor search (NNS) is performed against a pre-built, offline tool index ( $V$ ) to retrieve the most relevant tool cluster identifier ( $\hat{t}$ ).
Constrained Generation: The schema of the retrieved tool is passed to the LLM. Using the Outlines library, a Finite State Machine (FSM) constrains the model's output vocabulary, ensuring generated arguments are syntactically valid and conform strictly to the selected tool's signature.
Execution: The dispatcher executes the tool and returns the result to the LLM for final response synthesis.

This approach reduces the registry prompt token cost from $O(N)$ to $O(1)$ , as the registry is externalized as an embedding index rather than text in the context window.

2.2 Training Objective

A core challenge is resolving semantic blur, where semantically similar tools are functionally incompatible. NTILC addresses this via a composite training objective for the encoder ( $E$ ):

Circle Loss: Used for general query-tool alignment, weighting pairwise similarities to concentrate gradient signals near the retrieval decision boundary.
Functional Margin (FM) Loss: A novel loss term designed to separate tools with similar descriptions but incompatible signatures.
- It calculates a compatibility score ($compat$) based on the intersection of argument names, types, and return schemas.
- It assigns a repulsive weight ( $w_{ij}$ ) to negative pairs based on their functional incompatibility.
- It applies a margin penalty specifically to "hard negatives" (semantically close but functionally distinct tools), pushing them apart in the embedding space.

The total objective is $L_{dispatch} = \lambda_1 L_{circle} + \lambda_2 L_{FM}$ .

3. Key Contributions

Framework Innovation: Introduction of NTILC, which decouples agent capability from prompt length by externalizing the tool registry into a learned embedding index, reducing registry prompt tokens from $O(N)$ to effectively zero.
Functional Margin Loss: Proposal of a signature-aware loss function that explicitly enforces separation between semantically similar but functionally incompatible tools, directly addressing the "semantic blur" problem.
Comprehensive Evaluation: Evaluation across five public datasets (ToolBench, BFCL, API-Bank, MetaTool, ToolEyes) demonstrating performance across varying registry sizes.

4. Experimental Results

The authors evaluated NTILC against ICT baselines (including Qwen3-27B, Ministral 3, Kimi Moonlight, and closed-source models like ChatGPT 5 and Claude Sonnet 4.6) on public datasets.

Context Reduction: NTILC reduces context window consumption by over 95%. For a registry of 5,000 tools, ICT requires ~600,000 input tokens, whereas NTILC requires only the intent and the selected schema (<5,000 tokens).
Latency: NTILC reduces inference latency by up to 74% compared to long-context ICT baselines (e.g., on ToolBench, latency dropped from 5,663ms to 1,452ms).
Accuracy: NTILC maintains parity with state-of-the-art ICT models in Top-1 and Top-5 accuracy.
- On ToolBench, NTILC achieved 98% Top-1 accuracy with zero registry prompt tokens.
- On semantic-blur subsets, the inclusion of Functional Margin Loss improved Semantic-Blur Accuracy from 70.8% (Circle Loss only) to 75.0%, while reducing the Functional Error rate to 8.7%.

5. Significance and Claims

The paper claims that NTILC demonstrates that tool registries need not reside in the context window. By moving the registry to a compact embedding index, the framework:

Decouples capability from context length: Agent performance no longer degrades as the tool registry scales.
Solves the "Semantic Blur" problem: Through the Functional Margin Loss, the model learns to distinguish between tools that are textually similar but functionally distinct, a failure mode common in standard semantic retrieval.
Enables Scalability: The approach allows for the deployment of agents with massive tool registries without incurring prohibitive token costs or latency penalties.

The authors note limitations, specifically that NTILC requires offline indexing and may struggle with highly dynamic or state-dependent tool documentation without frequent re-indexing. They also emphasize that while NTILC improves selection efficiency, safety and policy enforcement (e.g., allow-lists) remain separate concerns that must be paired with the dispatch mechanism.