💬 NLP

Long-Context Encoder Models for Polish Language Understanding

This paper introduces a high-quality, long-context Polish encoder model capable of processing up to 8192 tokens through a two-stage training procedure and knowledge distillation, which achieves state-of-the-art performance on 25 tasks including long-document understanding while maintaining efficiency on short texts.

Original authors: Sławomir Dadas, Rafał Poświata, Marek Kozłowski, Małgorzata Grębowiec, Michał Perełkiewicz, Paweł Klimiuk, Przemysław Boruta

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sławomir Dadas, Rafał Poświata, Marek Kozłowski, Małgorzata Grębowiec, Michał Perełkiewicz, Paweł Klimiuk, Przemysław Boruta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant librarian named Polish-RoBERTa. For years, this librarian has been the best at organizing books, answering questions, and summarizing stories in the Polish language. But there's a catch: this librarian can only read a single page of text at a time. If you hand them a whole novel or a thick legal contract, they can only read the first page and have to guess what the rest is about. This is a problem for modern tasks like analyzing long bank reports or reading entire news articles.

The authors of this paper decided to give this librarian a superpower: the ability to read and understand a whole book (up to 8,192 pages, or "tokens") in one go, without losing their sharpness on short stories.

Here is the story of how they did it, explained simply:

1. The Problem: The "One-Page" Limit

Older AI models are like librarians with a very short attention span. They are great at quick tasks (like checking if a tweet is happy or sad), but they choke when faced with long documents. In the real world, especially in banking and finance, documents are often long and complex. You can't just read the first page of a mortgage agreement and expect to understand the whole deal.

2. The Solution: Training a "Long-Read" Librarian

The team took their existing, high-quality Polish librarian and gave them a crash course in reading long books. They did this in two clever steps:

Step 1: Learning the Map (Positional Embeddings): Imagine the librarian has a map of a small town (512 pages). Suddenly, they need to navigate a whole country (8,192 pages). If you just give them the new map, they get lost. So, first, they froze the librarian's brain and only trained them on the new map (the positions of the words). This helped them understand where they are in a long text without forgetting how to read short texts.
Step 2: Reading the Whole Library (Full Training): Once the librarian knew the map, they let them read the entire library again. They used a massive collection of 150 billion Polish words to practice. To make this efficient, they used special "Flash Attention" (a high-speed reading technique) and made sure the librarian didn't accidentally mix up the plot of one book with another while reading a stack of them.

3. The "Pocket" Versions: Knowledge Distillation

Not everyone needs a librarian who can read a whole library at once. Sometimes, you just need a quick assistant on a small device, like a smartphone or a tablet. These devices have limited memory and battery.

To solve this, the team created smaller, compressed versions of their super-librarian:

They took the big model and removed half (or even three-quarters) of its "brain" (layers).
Then, they used a technique called Knowledge Distillation. Think of this as a master chef (the big model) teaching a junior chef (the small model) how to cook. The junior chef watches the master and tries to mimic the results, learning to be almost as good as the master but much faster and lighter.
They even tried a "progressive" method: teaching the junior chef to be a medium-sized chef first, and then shrinking them down to a tiny size. This worked best, preserving the quality even in the smallest model.

4. The Test Drive: 25 Different Challenges

To see if their new librarian was actually good, they put it through a grueling test drive with 25 different tasks.

The Standard Tests: They checked if it could still do the old jobs well (like spotting names in text or guessing movie reviews).
The New "FinBench" Test: Since the model was built for a bank, they created a special test suite called FinBench. This included tasks like reading long mortgage documents, classifying customer emails, and analyzing stock market news.
The Results: The new model (Polish-RoBERTa-8k) was the clear winner. It was the best at reading long documents (like mortgage contracts) while staying just as good as the old models at short tasks. Even the tiny, compressed versions performed surprisingly well.

5. The Real-World Win: Inside the Bank

Finally, they tested the model on real, secret data from a major Polish bank (PKO Bank Polski).

The Task: Classifying thousands of customer emails and legal mortgage documents.
The Outcome: The new model, especially the one that got extra training on banking-specific text, significantly outperformed the old models. It correctly categorized customer emails 8% better than the previous best model.

The Big Takeaway

This paper is about building a Polish AI that doesn't have to choose between being smart and being fast.

The Big Model: Can read a whole novel (8,192 tokens) and understand the context perfectly.
The Small Models: Are like pocket-sized versions that fit on your phone but still know how to read long texts better than anyone else.

It proves that you don't need a massive, expensive "decoder" AI (like the ones that write stories) to understand long documents. A specialized "encoder" AI, trained correctly, can do the job cheaper, faster, and just as effectively.

1. Problem Statement

While decoder-only Large Language Models (LLMs) dominate the current NLP landscape, encoder-only models remain the standard for discriminative tasks (e.g., classification, NER) due to their cost-effectiveness and parameter efficiency. However, classic Polish encoder models (e.g., HerBERT, Polish RoBERTa) are limited by a short context window of 512 tokens, which is insufficient for processing long documents common in banking, legal, and financial sectors. Existing long-context solutions are primarily English-centric, leaving a gap for high-quality, long-context Polish models.

2. Methodology

A. Model Architecture and Training Strategy

The authors developed polish-roberta-8k, an encoder-only model based on the existing polish-roberta-large-v2 (443M parameters), extended to support 8,192 tokens.

Two-Stage Training Procedure:
1. Positional Embedding Adaptation: The positional embedding layer was extended to 8,192 positions and initialized randomly. In the first stage, all model weights were frozen, and only the new embeddings were trained for one epoch to align them with the existing model weights, preventing gradient fluctuations.
2. Full Parameter Fine-tuning: In the second stage (4 epochs), all model parameters were updated on a Polish corpus of 150 billion tokens.
Architectural Optimizations:
- Flash Attention: Implemented to reduce memory consumption and computational overhead.
- Contamination-Free Packing: A technique where multiple documents are concatenated into a single sequence, but the attention mechanism is constrained to prevent cross-contamination between unrelated documents within the same sequence.
- Masking Strategy: Used a 20% masking rate (higher than standard BERT/RoBERTa) with whole-word masking.
Checkpoint Merging: The final model was created by merging checkpoints from epochs 2, 3, and 4 of the second stage to ensure balanced performance on both short and long texts.

B. Knowledge Distillation (Compression)

To address edge deployment constraints, the authors created compressed variants using Knowledge Distillation:

Target Models: Reduced layer counts by 50% (12 layers) and 75% (6 layers).
Distillation Method: Used Mean Squared Error (MSE) between token representations of the teacher (8k model) and student. Attention matrix distillation was deemed infeasible due to the quadratic complexity of 8k sequences.
Progressive Distillation: For the 6-layer model, a "progressive" approach was tested (distilling 24L $\to$ 12L $\to$ 6L) versus "simple" distillation (24L $\to$ 6L). The progressive method yielded better results.

3. Key Contributions

polish-roberta-8k: The first high-quality Polish encoder-only model capable of processing sequences up to 8,192 tokens.
Compressed Variants: Release of distilled models (polish-roberta-8k-12L and polish-roberta-8k-6L) optimized for efficiency-critical applications.
FinBench Benchmark: Introduction of a new financial benchmark suite containing 7 Polish-language tasks specific to banking and finance (e.g., intent classification, sentiment analysis of press notes, multi-label topic classification).
Comprehensive Evaluation: A rigorous evaluation across 25 tasks, including the KLEJ benchmark, FinBench, and various other classification/regression tasks.

4. Results

A. Performance on Standard and Long-Context Tasks

Overall Performance: polish-roberta-8k achieved the highest average score (85.93%) across all 25 tasks, outperforming both older Polish models (HerBERT, Polish RoBERTa-v2) and multilingual models (XLM-R, EuroBERT, mmBERT).
Long-Context Superiority: The model significantly outperformed competitors on the four tasks involving texts exceeding 512 tokens (Banking-Long, MIPD, EURLEX, and a long IMDB subset).
Short-Text Parity: On short-text tasks (the majority of the KLEJ benchmark), the model maintained performance comparable to the best existing Polish encoders, proving that context extension did not degrade short-text capabilities.
Multilingual Comparison: New multilingual models (EuroBERT, mmBERT) performed poorly on general Polish tasks compared to models trained exclusively on Polish data, though they showed some resilience on long-text tasks.

B. Distilled Model Performance

The 12-layer distilled model achieved the best balance, outperforming base-sized models in the small category.
The 6-layer progressive model outperformed the simple 6-layer model and the standard polish-roberta-base-v2, demonstrating that progressive distillation is effective for extreme compression.

C. Internal Banking Evaluation

Evaluated on proprietary bank data (emails and mortgage agreements).
Domain Adaptation: A version of polish-roberta-8k further trained on 8 billion tokens of banking-specific data showed significant improvements:
- Banking Emails: +8 percentage points over the baseline HerBERT.
- Mortgage Documents: Significantly outperformed HerBERT, matching the performance of the standard polish-roberta-8k while leveraging domain-specific knowledge.

5. Significance

This paper addresses a critical gap in the Polish NLP ecosystem by providing a cost-effective, long-context encoder that bridges the performance gap between discriminative tasks and long-document understanding.

Practical Impact: The model is particularly valuable for the financial and legal sectors in Poland, where documents often exceed standard token limits.
Efficiency: The distilled variants allow for deployment on resource-constrained devices without sacrificing the ability to process long contexts.
Benchmarking: The introduction of FinBench sets a new standard for evaluating Polish models in the financial domain, moving beyond generic benchmarks like KLEJ.

In conclusion, the authors demonstrate that with careful architectural adaptation and training strategies, encoder-only models can effectively scale to long contexts, offering a superior alternative to LLMs for specific, high-precision discriminative tasks in low-resource languages like Polish.