💬 NLP

Internalizing Tool Knowledge in Small Language Models via QLoRA Fine-Tuning

This paper demonstrates that parameter-efficient QLoRA fine-tuning can internalize tool knowledge into small language models, enabling them to outperform larger, description-dependent baselines in tool planning while significantly reducing inference overhead and token usage.

Original authors: Yuval Shemla, Ayal Yakobe, Tanmay Agarwal

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Yuval Shemla, Ayal Yakobe, Tanmay Agarwal

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching a Robot to Memorize Its Toolbox

Imagine you are hiring a smart assistant to fix a complex machine. This machine has a huge toolbox with 23 different tools (like sensors, repair kits, and diagnostic scanners).

The Old Way (The Problem):
Every time you ask the assistant a question, you have to hand them a massive, 2,200-word instruction manual that lists every single tool, how it works, and what buttons to press.

The Issue: The assistant gets overwhelmed by the manual. They spend so much time reading the list of tools that they forget to actually answer your question. Also, if you want to use a "smaller" (cheaper) assistant, they can't handle such a huge manual at all. It's like trying to fit a whole library into a backpack just to find a single screwdriver.

The New Way (The Solution):
Instead of handing the manual over every time, you give the assistant a crash course. You train them until they have memorized the entire toolbox and how to use it.

The Result: Now, when you ask a question, you don't need the manual. The assistant already knows which tool to grab and how to use it. They can answer much faster, use less energy, and the "smaller" assistants can do the job just as well as the big ones.

How They Did It (The "QLoRA" Method)

The researchers used a technique called QLoRA fine-tuning. Think of this like giving the assistant a set of sticky notes (adapters) to stick on their brain, rather than rewriting their entire brain.

They took two small, open-source AI models (named Gemma and Qwen, both about the size of a standard smartphone app).
They fed them about 1,700 examples of questions and the correct "tool plans" to solve them.
The models learned to internalize the tool knowledge, moving it from the "external manual" into their own "internal memory."

The Results: Speed vs. Memory

The researchers tested these trained models against the "Old Way" (where the manual was still included).

Massive Savings: By removing the manual, they cut the amount of information the computer had to process by 82.6%. It's like switching from reading a novel to reading a text message.
Better Performance: Surprisingly, the models that didn't have the manual actually did a better job at planning than the ones that did.
- Why? When the manual was there, it crowded out the actual question. Without the manual, the model could focus 100% of its attention on your specific problem.
The Two Models:
- Gemma: Did the best job at planning (getting the highest scores), but it was a bit slower and used more computer memory.
- Qwen: Was 2.5 times faster and used 62% less memory than Gemma. It was almost as good at planning, making it a very efficient choice.

The Catch: The "Forgetting" Trade-off

There is a downside to this intense training. When you force a model to memorize a specific set of tools so well, it sometimes forgets other things it used to know.

The Analogy: Imagine a student who studies so hard for a specific math exam that they forget how to speak their native language or solve basic logic puzzles.
The Findings:
- Gemma forgot a little bit of its general knowledge (it retained about 80% of its old smarts).
- Qwen forgot a lot more (it retained only about 61% of its old smarts).
The Fix: The researchers found a "knob" they could turn (called the LoRA rank).
- If you turn the knob to high, the model becomes a master planner but forgets more general knowledge.
- If you turn the knob to low, the model stays a bit less perfect at planning but remembers much more of its general knowledge.

Summary of the Paper's Claims

You don't need the manual: Small AI models can be trained to "know" their tools without needing the tool descriptions written out in every prompt.
It's faster and cheaper: Removing the tool descriptions saves a huge amount of computer time and money.
It works better: In this specific test, the models that memorized the tools outperformed the models that were just reading the manual.
There is a trade-off: Making the model a better planner can make it forget some of its general knowledge, but you can adjust the training to balance this.

What the paper does NOT claim:

It does not say this works for every possible tool in the world (it only worked for a fixed set of 23 tools).
It does not claim the models are now perfect at doing the actual repairs (they are good at planning the repairs).
It does not suggest this is ready for medical or critical life-saving use yet; it is a proof-of-concept for industrial asset maintenance.

Technical Summary: Internalizing Tool Knowledge in Small Language Models via QLoRA Fine-Tuning

Problem Statement

Current agentic systems relying on Large Language Models (LLMs) for tool planning face significant inefficiencies. Standard pipelines require the inclusion of full tool schemas (descriptions, argument signatures, and return types) in every prompt. This creates substantial token overhead—accounting for up to 82.6% of input tokens in the authors' end-to-end setting—and forces reliance on large, expensive frontier models. Smaller models (2B–5B parameters) often fail to perform reliable tool planning without these extensive prompts, limiting their practicality in resource-constrained environments. The core challenge is whether a small language model can internalize a fixed tool catalog through parameter-efficient fine-tuning, enabling "description-free inference" where the model generates structured plans without explicit tool descriptions at runtime.

Methodology

The authors propose a supervised fine-tuning pipeline using QLoRA (Quantized Low-Rank Adaptation) to shift tool-use knowledge from prompt context into model weights.

Benchmark & Data: The study utilizes AssetOpsBench, an industrial asset operations benchmark containing 152 human-generated scenarios. The training corpus (~1,741 examples) was constructed using Gemini 2.5 Flash as a teacher model. The data includes three types of examples:
1. Tool/Agent Knowledge: Teaching agent-tool associations, argument schemas, and distinctions between similar tools.
2. Question-to-Plan: Mapping natural language queries to structured execution plans (Task/Agent/Tool/Args/Dependencies).
3. Execution-style: Combining planning steps with execution traces and placeholder resolution.
Models: Two open-source instruction-tuned models were evaluated: Gemma 4 E4B (4.5B active parameters) and Qwen3-4B (4.0B parameters).
Training Configuration: Both models were fine-tuned with 8-bit QLoRA on a single A100 GPU. The primary configuration used a LoRA rank ( $r$ ) of 32, $\alpha=64$ , and a learning rate of $2 \times 10^{-4}$ .
Evaluation Protocol:
- Inference Settings: Models were tested under three conditions: (1) Informed (full tool descriptions in prompt), (2) Description-free Baseline (no tool descriptions, unfine-tuned), and (3) Fine-tuned Description-free (no tool descriptions, fine-tuned).
- Metrics: Performance was measured via structural metrics (AT-F1 for agent/tool selection, ArgKey-F1) and semantic quality via an LLM-as-judge (Gemini 2.5 Flash) scoring six dimensions (correctness, routing, selection, arguments, efficiency, dependencies).
- Forgetting Analysis: Catastrophic forgetting was assessed using 100 multiple-choice questions from MMLU, ARC-Challenge, and HellaSwag.

Key Contributions

Formalization of Description-Free Planning: The paper defines a setting where a model must generate valid structured tool-use plans for a fixed catalog without receiving tool descriptions at inference time.
Demonstration of Internalization: The authors show that QLoRA fine-tuning on ~1,700 examples enables ~4B parameter models to surpass an "informed" baseline (which includes full tool schemas) in both structural accuracy and semantic planning scores, despite omitting tool descriptions entirely.
Architectural Comparison: A comparative analysis of Gemma 4 E4B and Qwen3-4B reveals that Qwen3 achieves comparable planning quality with 62% less memory and 2.5× faster inference, though it exhibits higher catastrophic forgetting.
Quality-Retention Trade-off: The study quantifies the trade-off between planning quality and general knowledge retention, showing that smaller LoRA ranks ( $r=8$ ) preserve more general knowledge with minimal quality loss compared to the quality-optimal rank ( $r=32$ ).
Pipeline Profiling: The authors profile the training and inference pipeline, identifying 8-bit matrix multiplication (MatMul8bitLt) as the dominant CUDA bottleneck (56.3% of time) and estimating the total experiment cost at approximately $62.

Experimental Results

Performance Gains: The best fine-tuned Gemma model achieved an AT-F1 of 0.65 and an LLM-judge score of 3.88, outperforming the informed baseline (AT-F1 0.47, Judge 2.88). The fine-tuned Qwen3 model achieved a judge score of 3.78.
Token Efficiency: Description-free inference reduced input length from ~2,400 tokens to ~128 tokens, an 82.6% reduction.
Efficiency: Qwen3-4B ran 2.5× faster at inference and used 62% less base memory than Gemma 4 E4B, while reaching comparable final evaluation losses.
LoRA Rank Impact: Planning quality peaked at $r=32$ (Judge 3.88, AT-F1 0.65) but declined slightly at $r=64$ . Smaller ranks ( $r=8$ ) preserved more general knowledge but yielded slightly lower planning scores.
Catastrophic Forgetting:
- Gemma: Retained 79.8% of its base performance on multiple-choice benchmarks at $r=32$ .
- Qwen3: Retained only 61.3% of its base performance at $r=32$ , with severe degradation in logical reasoning and commonsense completion.
Data Composition: Training on "Plan-only" data (where tool knowledge is implicit in the plan) yielded higher scores (3.90) than training on "Tool+Plan" data (3.60), suggesting demonstration-based learning is more effective than declarative knowledge injection for this task.

Significance and Claims

The paper claims that for fixed tool catalogs, QLoRA fine-tuning can successfully shift tool knowledge from prompt context into model weights. This approach offers a viable path to deploying smaller, more efficient models in agentic systems by eliminating the token overhead of repeated schema descriptions.

The authors position this work as a proof of concept demonstrating that:

Small models can internalize complex tool schemas to outperform larger, schema-informed baselines.
There is a tunable trade-off between planning quality and general knowledge retention via LoRA rank selection.
The approach is computationally accessible (low cost, single GPU training).

The paper remains modest regarding generalization, noting that results are specific to the AssetOpsBench domain and a fixed set of tools. It acknowledges limitations in statistical power (single seed runs, small held-out test set) and does not claim the method generalizes to unseen tools or dynamic catalogs without further training. The primary significance lies in the potential to reduce inference latency and operational costs in industrial agentic workflows by internalizing tool knowledge.