FrenchNews-7: Benchmarking Cross-Publisher French News Editorial Desk Classification
This paper introduces FrenchNews-7, a new benchmark for classifying French news articles into seven editorial categories across multiple publishers, which demonstrates that a fine-tuned CamemBERT model outperforms zero-shot large language models and reveals that classification performance varies significantly by category, with ambiguous domains like "Economie" and "Societe" approaching human-level agreement limits.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast, noisy landscape of modern news, every article belongs to a specific department within a newspaper or website. Just as a library sorts books into distinct sections like history, science, or fiction, news organizations route stories to specialized editorial desks: one team handles sports, another covers politics, and a third manages international affairs. For researchers studying how different media outlets shape public opinion, the ability to automatically sort these stories is essential. Without it, comparing how two newspapers cover the same event becomes a manual, impossible task. The challenge, however, is that every publisher uses its own unique system for labeling stories. One site might call a story "society," while another calls it "domestic affairs," even if the content is identical. To build a tool that works across the entire French media landscape, scientists needed a way to translate these different internal languages into a single, shared understanding.
A researcher has now created a new standard for this task, called FrenchNews-7. They gathered nearly 88,000 news articles from 13 different French publishers, ranging from major national dailies to digital-only outlets and regional papers. Their goal was to teach a computer to recognize which editorial desk a story belongs to, regardless of which website published it. To do this, they first mapped out the seven most common categories used across these outlets: society, culture and leisure, international news, domestic politics, sports, economy, and science and technology. They then built a hybrid labeling system. For about 72 percent of the articles, the label was obvious because the website's address itself contained the category name, such as a web link ending in "/sport" or "/politics." For the remaining 28 percent, where the address was vague or missing, the researcher used a large language model to read the headline and full text and assign the correct category. This process was carefully checked by both human experts and artificial intelligence to ensure the labels were accurate, resulting in a dataset where the agreement between human raters was very high.
Using this new dataset, the researcher trained several computer models to perform the sorting task. They tested models that looked only at the headline against models that read the entire article. The results showed that reading the full text provided a clear advantage, allowing the computer to understand the context of a story rather than just guessing from a title. The best-performing system, a model specifically trained on the French language, achieved a high level of accuracy, correctly identifying the editorial desk for roughly 80 percent of the stories it had never seen before. This performance held true even when the model was tested on four completely new publishers that were not part of the original training group. The study found that while sports and culture stories were easy to sort, stories about the economy and general society were much harder to categorize consistently. In fact, when the researcher asked human experts to sort these difficult economic stories from the new publishers, the humans agreed with each other only about 55 percent of the time. This suggests that the difficulty was not a flaw in the computer's intelligence, but rather a genuine ambiguity in how newsrooms themselves decide to file these complex stories.
The researcher also compared their specialized model against several powerful, general-purpose artificial intelligence systems that had not been trained on this specific news data. Even with the most advanced general models, the specialized French model performed better, particularly in the tricky categories of economy and society. This indicates that for specific, high-volume tasks like news sorting, a model fine-tuned on the actual language and structure of the target domain outperforms a generalist tool. The researcher made their dataset and their best-performing model available to the public, allowing other researchers to build upon this work. By providing a reliable way to sort news across different publishers, this work offers a new foundation for studying how French media covers the world, ensuring that future analyses are based on a consistent and accurate understanding of where every story belongs.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.