Making Bielik LLM Reason (Better): A Field Report

This paper outlines a research program focused on evaluating and enhancing the reasoning capabilities of the Polish large language model Bielik through benchmarking, comparative analysis, and strategic planning to ensure its competitiveness in the evolving AI landscape.

The Gist

Here is an explanation of the paper "Making Bielik LLM Reason (Better): A Field Report," translated into simple language with creative analogies.

🇵🇱 The Big Picture: Poland's AI Underdog

Imagine the world of Artificial Intelligence as a high-stakes Olympic Games. For the last few years, countries like the US and China have been winning almost every gold medal in the "Thinking" and "Math" events. Poland, according to the authors, has been sitting in the back of the stadium, watching the race and realizing they missed the last seven years of training.

To fix this, a team of Polish researchers created Bielik, a large language model (an AI brain) designed to speak Polish and compete on the global stage. This paper is their "training log." It details how they tried to teach Bielik not just to chat, but to think, reason, and solve puzzles like a human.

🧠 Phase 1: The "Einstein Riddle" Test (Finding the Weakness)

At first, the team treated Bielik like a student taking a test. They asked it logic puzzles, similar to the famous "Einstein's Riddle" (where you have to figure out who owns the fish based on a list of clues).

• The Problem: Bielik was struggling. It was like a student who could memorize a poem but couldn't solve a math word problem. If the puzzle was short, Bielik got it right. But if the puzzle got longer or had a twist, Bielik started to "hallucinate" (make things up) or get confused.
• The "Lost in the Middle" Effect: Imagine reading a long story and forgetting the beginning by the time you reach the end. Bielik had this problem; it forgot its own rules halfway through a complex task.
• The Fix: The team realized they couldn't just ask questions manually. They built an automated "Judge" (another AI) to grade Bielik's answers. They also realized that Bielik needed to learn how to think step-by-step, not just guess the answer.

🏗️ Phase 2: Building the "Reasoning Engine" (Bielik-R)

Once they knew what was broken, they started the heavy construction work. They didn't just tweak the model; they rebuilt its brain to handle "reasoning."

Think of this like upgrading a car from a standard sedan to a Formula 1 race car. They did three main things:

1. Supervised Fine-Tuning (SFT): They fed Bielik 1.3 million examples of "good thinking" (step-by-step solutions) from other smart AI models. It was like giving the student a library of solved homework problems to study.
2. Preference Optimization (DPO): They taught Bielik to prefer the better answer over the worse one, even if both were technically possible.
3. Reinforcement Learning (RL): This was the most crucial part. They created a "gym" with 143,000 Polish math and logic problems. Every time Bielik got an answer right, it got a "treat" (a reward signal). Every time it failed, it had to try again.

The Result: They created Bielik-R, the first Polish AI specifically designed to "think" before it speaks.

📊 The Scoreboard: How Did It Do?

They put Bielik-R up against the world's best AIs (like Google's Gemini, OpenAI's o1, and DeepSeek).

• The Verdict: Bielik-R is still in the "middle of the pack." It's not winning gold medals yet (it scored 56% compared to the leaders' 87%), but it's a massive improvement over its older versions.
• The "Token" Surprise: Here is a funny twist. To solve problems, Bielik-R "thought" for a long time. It used more "thinking tokens" (mental energy) than almost any other model.
  • Analogy: Imagine a race where everyone runs a 100-meter dash. The winners sprint fast. Bielik-R is like a runner who stops, takes a deep breath, checks their map, draws a diagram, and then runs. It's slower, but it's trying to be very careful. Sometimes it runs out of breath (hits the token limit) before finishing, but when it does finish, the logic is often solid.

🚀 Phase 3: The Future (Beyond Just Math)

The team isn't stopping at logic puzzles. They have a new game plan:

1. The "AI Tutor" (Math): They built a team of AI agents to solve Polish high school math exams. One agent finds the method, another writes the code, and a third explains it to the student. They found that even a smaller AI can be a genius if it has the right tools and a good team.
2. The "Lawyer" (Legal): They want to teach Bielik to read legal texts and spot contradictions without making things up.
3. The "Debater" (Argumentation): They plan to train it to analyze arguments, spot fallacies (like "ad hominem" attacks), and understand the difference between a good argument and a bad one.
4. The "Gamer" (Strategy): They tested Bielik in video games. At first, it played badly. But after losing a few battles, it learned to change its strategy and eventually won. This proves it can learn from failure.

🏁 The Bottom Line

This paper is a story of honesty and hard work. The Polish team admits they are behind the global leaders, but they are catching up fast.

• The Metaphor: If the global AI race is a marathon, Poland was just tying its shoes. Now, Bielik is running. It's not the fastest runner yet, and it sometimes trips over its own shoelaces (hallucinations), but it has a solid training regimen and a clear map to the finish line.

The ultimate goal isn't just to have a Polish AI that can chat; it's to build a Polish AI that can solve real-world problems, from fixing math homework to analyzing legal contracts, acting as a reliable partner in the future of science and technology.

Technical Summary

` delimiters to separate the reasoning trace from the final answer.

• Hybrid Mode: The model supports toggling between reasoning and non-reasoning modes via system message changes.

C. Multi-Agent Systems

Beyond single-model training, the paper explores orchestrated pipelines where LLMs act as intermediaries. A key example is Bielik-M, a multi-agent system for solving Polish matura (high school exit exam) math problems, featuring:

• Analytical Agent: Identifies methods.
• Executor Agent: Generates and self-repairs SymPy code.
• Summary Agent: Provides pedagogical explanations.
• Formalizer Agent: Optional Lean 4 verification.
• RAG Service: Local retrieval augmented with Polish synonyms.

3. Key Results

The paper presents comparative rankings and specific performance metrics:

• Benchmark Performance (Bielik-R 11B):
  • Ranked 18th out of 27 models with a score of 56%.
  • Comparison: Significantly behind leaders like gemini-3-pro-preview and o3 (87%) but outperformed mistral-small3.1 (45%) and earlier Bielik versions (29%).
  • Formal Logic Specifics: When tested specifically on formal logic (excluding the general benchmark noise), Bielik-R achieved 89% in first-order logic and 80% in propositional calculus.
• Token Efficiency:
  • Bielik-R (11B) used an average of 3,152 reasoning tokens, which is higher than some commercial models (e.g., o4-mini at 1,800) but comparable to SOTA reasoning models like DeepSeek-R1 (3,318).
  • Issue: The model occasionally exceeded token limits mid-solution, leading to point losses despite being on the correct path.
• Multi-Agent Success:
  • The Bielik-M system demonstrated that a relatively small 11B model could effectively solve exam-level mathematics when supported by task decomposition, symbolic verification (SymPy), and retrieval.
• Game Strategy:
  • Experiments in turn-based games showed models (e.g., Claude Opus 4.5) could iteratively improve strategies over multiple battles (7 battles to reach victory), validating the multi-turn interaction framework.

4. Key Contributions

1. First Polish Reasoning Benchmark: Established a rigorous, automated evaluation framework (Bielik-R Benchmark) with a specific taxonomy for Polish LLMs.
2. Bielik-R Model: Released the first Polish LLM explicitly trained for reasoning, utilizing a hybrid SFT + DPO + RL pipeline with Polish-specific data.
3. Methodological Insights:
   • Identified that deterministic puzzles (logic/math) are more effective for evaluation than non-deterministic ones due to objective scoring.
   • Highlighted the "lost-in-the-middle" effect in long reasoning tasks and the model's rigidity in belief revision.
   • Demonstrated that multi-agent orchestration (decomposing tasks) can compensate for the limitations of smaller parameter models (11B) in complex domains like mathematics.
4. Open Data & Tools: Created large-scale Polish datasets for reasoning (490k tasks) and developed tools for synthetic reasoning data generation and verifiability proofing.

5. Significance and Future Directions

• Strategic Importance: The work aims to restore Poland's visibility in the global AI research map by producing a competitive, locally-relevant reasoning model.
• Beyond Single Models: The paper argues that the future of AI reasoning lies not just in larger models, but in orchestrated pipelines where LLMs act as controllers for tools (formal analyzers, code executors, RAG systems) that provide strict formal analysis and repeatability.
• Future Research Tracks:
  • Domain Specialization: Developing specialized agents for Law (legal text interpretation) and Mathematics (AI tutors).
  • Advanced Cognitive Tasks: Moving into discourse analysis, Winograd schemas, and moral dilemma resolution.
  • Theoretical Inquiry: Investigating the boundary between memorization and genuine inference, and characterizing heuristic shortcuts models use to simplify problems.
  • Hallucination Reduction: Developing benchmarks specifically for inducing and measuring hallucinations in reasoning contexts.

In conclusion, the paper provides a comprehensive "field report" on the iterative process of transforming a general-purpose LLM into a reasoning-capable agent, emphasizing that while Bielik currently lags behind global giants, targeted training, multi-agent architectures, and rigorous benchmarking offer a viable path to competitiveness.