compar:IA: The French Government's LLM arena to collect French-language human prompts and preference data

Technical Summary of compar:IA: The French Government's LLM Arena

Problem Statement
Large Language Models (LLMs) exhibit reduced performance, cultural misalignment, and weaker safety robustness in non-English languages. This disparity stems from the dominance of English in both pre-training corpora and human preference alignment datasets. While techniques like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) rely heavily on human preference data, such datasets remain scarce, largely proprietary, and non-public for languages other than English. Existing public "arena" platforms for collecting preference data (e.g., LMSYS Chatbot Arena) are heavily concentrated in English, limiting the ability to train and evaluate models for underrepresented languages.

Methodology and Platform Design
To address this gap, the authors introduce compar:IA, an open-source digital public service developed within the French government (jointly by the Ministry of Culture and DINUM). The platform functions as a blind pairwise comparison arena designed to collect large-scale human preference data from a predominantly French-speaking audience.

• Interaction Flow: Users enter free-form, unconstrained prompts. Two models generate responses side-by-side without revealing their identities. Users can engage in multi-turn conversations and provide feedback at two levels: fine-grained reactions to individual messages and overall conversation-level votes. After submission, model identities and metadata (including estimated environmental impact via the Ecologits library) are revealed.
• Design Principles: The platform prioritizes low friction and accessibility. No account creation is required, minimizing metadata collection and lowering barriers for non-expert users. The interface avoids technical jargon to encourage participation from the general public, including students and educational institutions.
• Infrastructure: Initially built on the LMSYS Chatbot Arena codebase using Gradio, the system was refactored to a FastAPI backend with a SvelteKit frontend for stability and performance. Inference is managed via a self-financed model using providers like OpenRouter and Hugging Face Inference Providers, ensuring independence from ad-hoc credit allocations.
• Data Privacy and Filtering: To comply with GDPR and national regulations, the platform employs a conservative post-collection filtering pipeline. An LLM-based detector scans conversations for personal or sensitive information. If detected, the entire conversation, along with its associated votes and reactions, is excluded. Approximately 5% of conversations are filtered out. No span-level anonymization is performed to avoid semantic distortion or residual identifiability risks.

Key Contributions and Results
As of February 7, 2026, compar:IA has collected over 600,000 free-form prompts and 250,000 preference votes, with approximately 89% of the data in French. The platform has attracted over 300,000 unique visitors since its public launch in October 2024.

The authors release three complementary datasets under the Etalab 2.0 open license:

1. comparia-conversations: Multi-turn prompts and model responses.
2. comparia-votes: Conversation-level pairwise preference annotations.
3. comparia-reactions: Message-level feedback signals.

Key Findings and Analyses:

• Usage Patterns: Analysis indicates that French users primarily engage with AI for learning, advice-seeking, content creation, and information retrieval. Interactions are predominantly "augmentative" (assisting human effort) rather than fully automative.
• Topic Distribution: Technical and educational prompts dominate (approx. 32% combined), followed by business, society, and politics.
• Model Leaderboard: A French-language model leaderboard was released, utilizing Bradley–Terry models to rank models based on aggregated pairwise preferences. This serves as an educational and exploratory tool rather than a definitive benchmark.
• Dataset Reuse: Early survey data suggests active interest from academics (68%) and private companies (20%), primarily for model training, research, and evaluation.

Significance and Claims
The paper positions compar:IA as a significant contribution to the limited pool of open, non-English preference data for conversational AI. Its primary significance lies in demonstrating that:

1. Scalable Public Participation: Large-scale human evaluation can be successfully conducted with a general public when interfaces are accessible and participation friction is minimized, without relying on expert annotators.
2. Public Infrastructure: Public institutions can effectively build and operate open AI evaluation infrastructure that balances participation, transparency, and privacy, operating outside commercial incentives.
3. Replicability: The platform offers a replicable model for language-specific, human-centered evaluation. The underlying architecture is language-agnostic, and the project is currently expanding to other European languages (starting with Denmark) to create a digital common for multilingual model training and evaluation.

Limitations
The authors explicitly acknowledge several limitations:

• User Representativeness: The lack of socio-demographic data prevents weighting for population imbalances, and the user base skews toward those with existing interest in AI or digital tools.
• Task Coverage: Professional and sensitive use cases (e.g., law, healthcare) are underrepresented due to privacy concerns regarding open publication.
• Evaluation Bias: The arena format may encourage "testing" prompts rather than naturalistic usage. Furthermore, standard biases in pairwise comparison (e.g., verbosity, length, and style biases) are present, though the open release of data allows the research community to study and mitigate these effects.
• Infrastructure Variability: Differences in system prompts, model quantization, and inference providers may introduce confounds in model comparability.

In conclusion, compar:IA serves as both an awareness-raising platform and a data collection infrastructure, providing empirical evidence for the role of public institutions in building open AI ecosystems and addressing the linguistic imbalance in LLM development.