propella-1: Multi-Property Document Annotation for LLM Data Curation at Scale

1. Problem Statement

Current Large Language Model (LLM) pretraining data curation relies heavily on single-scalar quality scores (e.g., "educational value") generated by small classifiers distilled from large LLMs. The authors identify two fundamental limitations in this dominant paradigm:

• Conflation of Dimensions: A single score merges distinct quality dimensions (e.g., reasoning depth, safety, commercial bias, and pedagogical structure). This prevents flexible filtering; for instance, high-reasoning legal documents may be discarded for low "educational" scores, while high-scoring educational texts might contain heavy commercial bias or PII.
• Lack of Interpretability and Flexibility: Single scores offer no insight into why a document was filtered, making it impossible to compose complex filtering strategies (e.g., "high reasoning AND low commercial bias AND no PII").
• Language Bias: Most existing pipelines are English-centric or rely on per-language classifiers that still output single scores, failing to capture the nuance of multilingual data curation.

2. Methodology

The authors introduce Propella-1, a framework and model family designed to move from scalar scoring to structured multi-property annotation.

A. Annotation Framework

The system defines 18 distinct properties organized into six categories, allowing for granular, composable filtering:

1. Core Content: Content Integrity, Content Ratio, Content Length.
2. Classification: One-Sentence Description, Content Type (18 classes), Business Sector (37 sectors), Technical Content (7 types).
3. Quality & Value: Content Quality, Information Density, Educational Value, Reasoning Indicators.
4. Audience & Purpose: Audience Level, Commercial Bias, Time-Sensitivity.
5. Safety: Content Safety, PII Presence.
6. Geographic: Regional Relevance, Country Relevance.

The annotation rubric spans ~8,000 words, providing precise operational definitions, decision trees, and language-specific guidelines (e.g., handling agglutinative languages or non-Latin scripts).

B. Model Architecture & Training

• Models: Three decoder-only models based on the Qwen-3 architecture with parameter sizes of 0.6B, 1.7B, and 4B.
• Training Data: Annotated using frontier LLMs (prompted with the full rubric) on a diverse dataset covering 57 languages (approx. 35% English, remainder European, Asian, and Middle Eastern languages). The data includes web crawls, PDFs, code, math, and synthetic instruction-following data.
• Output Format: The models produce structured JSON conforming to a predefined schema with enumerated values, eliminating the need for separate classifiers per property.
• Inference: Optimized for speed using fp8 mixed-precision and SGLang with llguidance for strict JSON schema enforcement, ensuring valid outputs without post-processing retries.

3. Key Contributions

1. Propella-1 Models: Small, specialized multilingual LLMs that achieve high agreement with frontier models at a fraction of the cost and latency.
2. Propella-Annotations Dataset: A release of over 3 billion document annotations covering major pretraining corpora (FineWeb-2, FinePDFs, HPLT 3.0, Nemotron-CC, etc.). This is the largest open release of multi-property annotations to date.
3. Composable Filtering Framework: A shift from rigid single-score thresholds to flexible, multi-dimensional filtering logic (e.g., selecting documents based on specific combinations of reasoning depth and safety).
4. Multilingual Support: Native support for 57 languages with specific rubric adaptations for linguistic nuances.

4. Results

Evaluation Setup

• Reference: Annotations were compared against Gemini-3-Pro (set to "high" reasoning effort) acting as a high-quality proxy.
• Metrics: Quadratic Weighted Kappa (QWK) for ordinal properties, F1 for binary, and Intersection-over-Union (IoU) for multi-select properties.

Performance

• Accuracy: The Propella-1-4B model achieved an overall agreement score of 0.779, outperforming Gemini-3-Flash (0.778) and significantly larger open models like Qwen3-30B (0.556) and Gemma-3-27B (0.604).
• Efficiency: The 4B model processes 27 documents/second on a single H100 GPU (fp8), requiring only ~10.3 GPU-hours to annotate 1 million documents.
• Scaling: The authors successfully annotated ~500 million documents from FineWeb-2 in ~3.5 hours using 3,936 A100 GPUs.

Dataset Insights (Case Studies)

1. Source Profiling: Analysis of German-language corpora revealed that FinePDFs contains significantly more "excellent" quality content and analytical reasoning than FineWeb-2, despite both being German. Single-score filters would likely miss these compositional differences.
2. Quality Tier Audit: In Nemotron-CC, documents labeled "high quality" by a single-score classifier still contained significant issues in specific dimensions (e.g., heavy commercial bias or thin information density), proving that aggregate scores mask heterogeneity.
3. Cross-Language Variation: The same processing pipeline applied to FineWeb-2 produced vastly different quality profiles across languages (e.g., German vs. Finnish), suggesting that uniform filtering thresholds are suboptimal for multilingual curation.

5. Significance

• Paradigm Shift: Propella-1 challenges the "single-score" dogma in LLM data curation, demonstrating that structured, multi-dimensional annotation is feasible at scale and provides superior granularity.
• Cost-Effectiveness: It proves that small, specialized models (4B parameters) can match or exceed the annotation quality of much larger general-purpose models (30B+), drastically reducing the compute cost of data curation.
• Reproducibility and Openness: By releasing the model weights (Apache 2.0), the full rubric, and the 3B+ annotation dataset (CC-BY-4.0), the authors enable the community to perform data-centric research, design better data mixtures, and conduct compliance-aware governance without incurring annotation costs.
• Future Impact: The framework enables advanced techniques like property-aware curriculum learning (evolving data mixtures based on multiple quality dimensions) and targeted data transformation strategies.

In conclusion, Propella-1 provides the infrastructure and data necessary to move LLM pretraining from coarse, heuristic filtering to precise, interpretable, and multi-dimensional data curation.