Annotation-Efficient Vision-Language Model Adaptation to the Polish Language Using the LLaVA Framework

1. Problem Statement

Vision-Language Models (VLMs) have achieved state-of-the-art performance, but they remain predominantly English-centric. This limitation stems from the composition of standard training datasets, which restricts the usability of these models for non-English speakers and hinders the development of multimodal systems that reflect diverse linguistic and cultural realities.

Specifically, Polish presents unique challenges for VLMs due to its:

• Rich morphological complexity: Including seven grammatical cases, complex agreement patterns, and free word order.
• Lack of dedicated resources: Unlike text-only Large Language Models (LLMs), there were no existing high-quality Polish VLMs or specialized multimodal instruction datasets prior to this work.

The authors investigate whether large-scale automated translation, combined with minimal manual intervention, can effectively bootstrap high-quality VLMs for low-resource languages like Polish.

2. Methodology

The authors propose a fully automated pipeline to adapt the LLaVA-NeXT architecture for Polish, relying on translation and synthetic data generation rather than expensive manual annotation.

A. Model Architecture

• Base Framework: LLaVA-NeXT (supports higher resolutions and dynamic tiling).
• Language Backbone: Three Polish-native, instruction-tuned LLMs (11B–12B parameters):
  • PLLuM-12B (two variants)
  • BIELIK-11B
• Vision Tower: Replaced the standard CLIP encoder with SigLIP2 (SO400M/14, 384px), chosen for its superior multilingual image-text alignment and robust text-localization signals.
• Training Strategy: A two-stage process using LoRA (Low-Rank Adaptation) for parameter efficiency:
  1. Stage 1 (Pre-training): Aligns the visual encoder with the LLM using image-caption pairs (updating only the projector).
  2. Stage 2 (Instruction Tuning): Jointly fine-tunes the projector and vision encoder while applying LoRA to the LLM on a diverse set of multimodal instructions.

B. Data Construction Pipeline

The training dataset (approx. 2.0M samples) was constructed using a scalable, automated pipeline:

1. Translation: English multimodal datasets were translated into Polish using Tower+ 72B, a state-of-the-art multilingual model.
2. Quality Filtering: A reference-less COMET metric assessed translation quality. Samples falling below a dynamic threshold (0.4–0.8) were discarded.
3. Synthetic Data Generation:
   • OCR: To address reading capabilities, the authors generated synthetic OCR data (SynthDoG-PL) by rendering Polish text snippets from Wikipedia onto natural image backgrounds (MS COCO) with randomized fonts and layouts.
   • Knowledge: Adapted the WIT (Wikipedia-based Image-Text) dataset, retaining only human-written Polish captions.
4. Instruction Mixture: The final dataset maintains an 85:15 ratio of Polish-adapted to original English samples to preserve English capabilities while prioritizing Polish.

C. Evaluation Benchmark Adaptation

The authors adapted the MMBench-dev dataset (1,292 questions) for Polish:

• Machine-translated using Tower+ 72B.
• Human Revision: Native Polish linguists performed a thorough review, identifying and correcting 3.56% of questions for linguistic/content inaccuracies and 3.02% for foreign cultural contexts.

3. Key Contributions

1. Automated Pipeline: A scalable, fully automated pipeline for preparing multimodal datasets for low-resource languages, combining large-scale translation, metric-based filtering, and synthetic data generation.
2. Polish VLM Family: Introduction of the LLaVA-PLLuM and LLaVA-Bielik model families (11B–12B parameters), the first Polish-native VLMs trained on this methodology.
3. Benchmark Adaptation: A comprehensive adaptation of MMBench-dev for Polish, including linguistic re-annotation that identified issues in nearly 4% of samples.
4. Open Release: Public release of the models and the Polish version of MMBench-dev to facilitate future research.

4. Results

A. MMBench Performance

• Polish Adaptation: The proposed models significantly outperformed English-centric baselines on the Polish-adapted MMBench.
  • LLaVA-PLLuM-12B-nc achieved 79.35% accuracy on the Polish split, a +9.55 percentage point improvement over the best English baseline (LLaVA-1.6-Vicuna-13B at 69.80%).
  • The models maintained competitive performance on the English split, demonstrating that Polish adaptation did not degrade English capabilities.
• Comparison: The models surpassed open-source competitors like Qwen2.5-VL-7B and PaliGemma2-10B on the Polish benchmark, though they trailed behind Pixtral-12B (whose training data is undisclosed).

B. Image Captioning (XM3600)

• Linguistic Correctness: Human evaluation by professional linguists showed a strong advantage for the proposed models over baselines.
  • The best model (LLaVA-PLLuM-12b-nc) achieved an 84% win rate against Pixtral-12B in linguistic correctness.
  • Automatic judges (LLMs/VLMs) often underestimated the linguistic superiority of the Polish models compared to human judges, highlighting the difficulty of automated evaluation for morphologically rich languages.
• Content Quality: The models performed comparably to top-tier baselines in describing image content, though they slightly lagged behind Qwen2.5-VL-7B in specific content fidelity metrics.

C. Qualitative Analysis

Case studies demonstrated that the proposed models correctly identified cultural contexts (e.g., Polish geography, local landmarks) and avoided the "translationese" or hallucinated terminology often seen in baselines when answering in Polish.

5. Significance and Limitations

Significance:

• Cost-Effectiveness: The study proves that high-quality VLMs for morphologically complex, low-resource languages can be built primarily through automated translation and synthetic data, drastically reducing the need for expensive manual annotation.
• Cultural Relevance: It addresses the "English-centric" bias in AI, providing a foundation for multimodal systems that understand local cultural and linguistic nuances.
• Resource Availability: The release of models and benchmarks fills a critical gap in the Polish NLP ecosystem.

Limitations:

• Translation Artifacts: The reliance on automated translation may introduce subtle semantic drift or unnatural phrasing ("translationese") that was not systematically ablated.
• Cultural Coverage: While improved, the visual tasks still largely originate from English-centric benchmarks, meaning the models may not fully capture authentic Polish scenarios.
• Evaluation Discrepancies: There is a notable gap between automatic judge scores and human evaluations, particularly regarding linguistic correctness, suggesting current automated metrics are insufficient for Polish.
• Ablation Gaps: The study did not comprehensively ablate specific design choices (e.g., SigLIP vs. CLIP, LoRA vs. Full Fine-tuning).

In conclusion, this work demonstrates that large-scale automated translation, when paired with lightweight filtering and native LLM backbones, is a viable and effective strategy for bootstrapping multimodal models for languages with limited native resources.