Training on Broken Contexts: Source Loss and Repair in Tool-Augmented LLM Post-Training
This paper identifies and quantifies a critical failure in tool-augmented LLM training where standard sequence construction policies sever the link between assistant targets and their source tool observations, and proposes reconstruction strategies that significantly improve supervision coverage and context integrity while reducing token redundancy.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Modern artificial intelligence assistants are increasingly designed to act as agents that can reach out to external tools to solve problems. When a user asks a question, the model might decide to call a database, check a weather service, or retrieve an order status. To teach a model how to do this, researchers provide it with thousands of examples of these interactions. These examples are not just simple questions and answers; they are complex conversations containing the user's request, the model's decision to use a tool, the raw data returned by that tool, and finally, the model's response that uses that data to answer the user. The goal of training is to show the model the entire story so it learns to connect the dots between a tool's output and the final answer.
However, there is a hidden step in this process that happens after the conversations are written down but before the model actually learns from them. To fit these long stories into the computer's memory, the training software must chop them up, rearrange them, or cut them to fit a specific size limit. This paper investigates what happens when that chopping and rearranging goes wrong. The researchers asked a simple but critical question: when the computer cuts up a conversation to fit it into a training slot, does it sometimes keep the final answer while accidentally throwing away the tool data that answer was based on? If the model is being graded on an answer but the evidence for that answer is missing from its view, the model is being asked to learn a magic trick rather than a logical process.
The researchers examined three large collections of tool-using conversations that are commonly used to train these systems. They built a tracking system that followed every single piece of text from the original conversation through the chopping and rearranging process. They looked specifically for moments where the model was supposed to learn from a value returned by a tool, such as a specific order number or an error code, and checked if that value was still visible to the model when it was learning to write the answer. They tested this across different sizes of training chunks, ranging from very short to quite long, and used several different methods for how the software decides to cut and pack the text.
The results revealed a significant problem with how some training data is currently prepared. When the software uses a method that splits long conversations into smaller pieces to fit them into a fixed size, it frequently separates the answer from the tool data it relies on. In one of the datasets they studied, nearly forty percent of the retained answers were being taught to the model without the tool observation that justified them. In another dataset, about twenty percent of the answers were similarly orphaned. This means the model is being rewarded for generating text that it cannot possibly know, because the evidence it needs has been cut away and placed in a different part of the training sequence that the model cannot see at that moment.
The study also found that the way the data is cut changes what the model actually learns. One common method for packing data, which tries to fit as many examples as possible into a single chunk without splitting them, avoids this separation problem entirely. However, this method is so strict that it throws away most of the training data, keeping only a tiny fraction of the original conversations. The researchers showed that there is a trade-off: methods that keep almost all the data often break the connection between the tool and the answer, while methods that keep the connection intact often discard too much of the data to be useful.
To fix this, the researchers proposed a new way to reconstruct the training examples. Instead of blindly cutting the text, they suggested grouping related parts of the same conversation together so that the tool data and the answer always stay in the same viewing window. They found that by carefully grouping these pieces, they could keep the same number of correct answers while using less computer memory and eliminating thousands of duplicated pieces of text. They also discovered that for some datasets, the list of rules describing how to use a tool takes up so much space that it pushes the actual conversation out of the window. By simplifying these rule lists to show only the specific parts needed for a single call, they were able to fit many more useful examples into the training process.
Finally, the researchers checked the software code used by many developers to prepare this data and found a hidden error in a widely used version. This error caused the software to accidentally repeat parts of the text or mix up the order of words when processing large batches of data, creating a stream of information that did not match the original conversation at all. This discovery highlights that the problem is not just in the data itself, but in the tools used to prepare it. The paper concludes that the way training data is constructed is just as important as the data itself. If the connection between a tool's output and the model's answer is broken during preparation, the model cannot learn to use the tool correctly, no matter how much data it sees. The researchers urge that future training systems must verify that the context required for an answer is actually present when the model is learning from it.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.