PERK: Long-Context Reasoning as Test-Time Learning

Large language models are powerful tools that have learned to read and write by studying vast amounts of text. They excel at finding patterns and answering questions when the information is right in front of them. However, these models face a significant hurdle when the information they need is buried inside a massive wall of text, such as a book-length document or a long transcript. As the text grows longer, the models often struggle to ignore the irrelevant noise and find the specific fact required to solve a problem. This difficulty is compounded by a tendency in many models to focus heavily on the very beginning or the very end of a text, while losing track of details hidden in the middle. Researchers have long sought ways to help these models handle such "long-context" tasks without requiring them to be retrained from scratch, a process that is often prohibitively expensive and time-consuming.

In a new study presented at the ICLR 2026 conference, researchers from EPFL propose a solution called PERK, which stands for Parameter Efficient Reasoning over Knowledge. Instead of trying to force the model to read a massive document all at once, PERK treats the act of reading as a learning process that happens right at the moment the question is asked. Imagine a student who, upon receiving a thick textbook and a specific question, quickly scans the pages to write a concise set of notes on a notepad. Once those notes are written, the student can put the textbook away and answer the question using only the notes. PERK works in a similar fashion. When a long document is presented, the model does not keep the entire text in its active memory. Instead, it uses a brief, internal learning phase to compress the most important parts of that text into a small, efficient set of adjustments to its own internal settings. These adjustments act like a specialized memory module, storing the essence of the long document. Once this "notepad" is created, the original text is discarded, and the model uses its new, updated settings to answer questions about the content.

The researchers developed a training method that teaches the model how to perform this compression and retrieval effectively. They used a technique involving two layers of learning. In the first layer, the model learns to quickly encode a chunk of text into its memory settings. In the second layer, the model learns how to use those settings to answer questions accurately. To keep this process efficient and prevent the computer from running out of memory, the model only updates a tiny fraction of its total parameters—specifically, a lightweight add-on component known as a low-rank adapter. This allows the model to learn from the context without altering its core knowledge base. The researchers tested this approach on various challenging tasks, including finding a single specific fact hidden within thousands of pages of text, answering complex questions that require connecting multiple pieces of information, and retrieving data from long lists of similar-looking records.

The results showed that PERK significantly outperformed standard methods where models are simply trained on long texts to answer questions later. On tests involving the Qwen-2.5 model, PERK improved accuracy by up to 20 percent compared to these standard approaches. The method proved robust even when the model was asked to handle texts much longer than it had seen during training, successfully generalizing to contexts of 64,000 and even 128,000 tokens. Furthermore, PERK demonstrated a unique ability to ignore the position of information. While other models often fail when the relevant fact is moved from the beginning or end of a text to the middle, PERK maintained high accuracy regardless of where the information appeared. This suggests that by encoding the text into a flexible memory structure rather than relying on fixed positions, the model can reason more reliably. The approach worked consistently across different model sizes, from very small models with 0.5 billion parameters to larger ones with 8 billion, and even matched or surpassed the performance of specialized, massive models designed specifically for long contexts.

Beyond accuracy, the study highlighted the efficiency of this method. Because the model processes the text in smaller, manageable chunks and discards the original text after encoding, it requires significantly less computer memory to handle extremely long documents. In tests where standard methods crashed due to memory limits at 128,000 tokens, PERK continued to function, processing the information in a fraction of the time and using a fraction of the memory. The researchers concluded that by reframing long-context reasoning as a test-time learning task, where the model adapts its own memory on the fly, it is possible to achieve superior performance without the heavy computational cost of traditional training. This approach offers a promising path forward for making large language models more capable of handling the vast and complex information found in the real world.