BenCSSmark: Making the Social Sciences Count in LLM Research

Technical Summary: BenCSSmark: Making the Social Sciences Count in LLM Research

1. Problem Statement

The paper identifies a critical gap in the current landscape of Large Language Model (LLM) evaluation: the systematic under-representation of social science tasks in mainstream benchmarks. While benchmarks are pivotal in structuring AI research agendas, reputations, and commercial outcomes, they predominantly focus on traditional Natural Language Processing (NLP) tasks (e.g., paraphrasing, named entity recognition) or specialized domains like medicine and law. Social sciences (sociology, political science, history, economics, etc.) are largely absent.

This absence creates a dual problem:

1. Evaluation Blind Spots: Models are rarely tested on the nuanced, context-sensitive, and culturally situated tasks that define social scientific inquiry. Consequently, state-of-the-art models often struggle with tasks that require interpretive robustness rather than just factual recall or precision.
2. Resource Invisibility: High-quality, rigorously annotated datasets generated by social scientists remain "invisible" to the NLP community. These datasets are often dispersed, uncatalogued, or created as ad-hoc collections for specific studies, lacking the standardization required for inclusion in benchmark suites. Furthermore, social science tasks often resist standardization due to their reliance on context-specific categories and the preservation of multiple, sometimes conflicting, expert perspectives (perspectivism), which contrasts with the "single ground truth" logic of many existing benchmarks.

2. Methodology: The BenCSSmark Initiative

To address this gap, the authors introduce BenCSSmark, a benchmark composed of datasets derived from actual computational social science projects. The initiative is part of the "Pantagruel" project, aimed at developing French-language LLMs. The methodology involves a collaborative effort between computer scientists and social scientists to curate and structure these resources.

Data Collection Strategies:
The authors employed two distinct strategies to gather data:

• New Data Production: Assembling teams of annotators to generate new datasets. Crucially, unlike standard NLP practices that often discard disagreements, this approach preserved all individual annotations alongside the adjudicated decision to reflect the diversity of viewpoints.
• Curating Existing Expert Annotations: Utilizing datasets annotated by single domain experts (often the researchers themselves). While these cannot be evaluated using standard inter-annotator agreement metrics, they represent high-quality, domain-specific knowledge spanning various topics and historical periods.

Guiding Principles:
The benchmark was constructed based on three core principles:

1. Representation of Diversity: Drawing tasks from underrepresented fields (e.g., sociology, political science) and respecting disciplinary approaches to tasks like frame detection (e.g., distinguishing between social media and 20th-century newspaper frames).
2. Contextual and Temporal Variation: Assembling data from diverse genres (social media, news, political speeches, radio) and historical periods (late 20th century to present) to test model adaptability to shifting social categories and discourse conventions.
3. Preservation of Perspectivism: For tasks annotated by the team, the benchmark provides two versions of the data: one with a single ground truth and one retaining all individual annotations. This allows for the evaluation of task difficulty based on annotator agreement and supports the "perspectivist" call for disaggregated data.

Data Structure and Taxonomy:
BenCSSmark currently contains 27 datasets (all in French). The authors introduced a novel dual-taxonomy system:

• Task Category: Maps to standard NLP terminology (e.g., binary classification, span detection, coreference resolution).
• Concept: Explicitly identifies the specific social scientific concept being captured (e.g., "appeal to authority," "gender," "reform pledges," "political forecasting").
• Metadata: Includes detailed information on data genre, temporal scope, annotator background, and disciplinary context.

The tasks cover a wide range of types, including binary/multiclass classification, multi-label classification, span detection, and coreference resolution. The data spans text genres such as national press, political manifestos, radio transcripts, and academic abstracts.

3. Key Contributions

• BenCSSmark Benchmark: The primary contribution is the creation of a benchmark specifically designed for social science tasks, bridging the gap between NLP evaluation and social scientific inquiry.
• Taxonomic Framework: The introduction of a dual categorization system (NLP task type vs. Social Science concept) to facilitate communication and exchange between computer scientists and social scientists.
• Perspectivist Data Handling: The inclusion of datasets that preserve annotator disagreement and multiple viewpoints, challenging the "single ground truth" paradigm and offering a stress test for model interpretive robustness.
• Interdisciplinary Collaboration: The paper demonstrates a practical framework for integrating social scientists directly into the design of LLM evaluation, moving beyond using them merely as auditors of bias.

4. Results and Current Status

The paper does not present experimental results evaluating specific LLMs on BenCSSmark. Instead, it presents the infrastructure and the dataset collection itself as the result.

• Dataset Availability: The benchmark comprises 27 datasets with varying sizes (from ~100 to ~28,000 units) and granularities (sentence, paragraph, full text).
• Task Diversity: The collection includes both well-studied NLP categories (e.g., hate speech detection) applied to novel social science concepts (e.g., distinguishing "critical" from "hateful" comments in political contexts) and unique tasks (e.g., detecting "reform pledges" in manifestos or "prescriptive" music news).
• Limitations Acknowledged: The authors explicitly state that the benchmark is currently closed (due to proprietary data and privacy concerns) and limited to the French language and specific media formats. They acknowledge that the scale is currently limited and that the data is not yet fully open for public download.

5. Significance and Claims

The paper positions BenCSSmark not as a definitive solution, but as a provocative first step and an invitation for collaboration. Its significance is framed around several key arguments:

• Stress Testing AI: The authors argue that social science tasks serve as a "stress test" for AI, evaluating a model's ability to navigate ambiguity, conflicting perspectives, and context-dependent meaning—dimensions often peripheral to standard NLP evaluations.
• Improving Generalization: By integrating social scientific perspectives, the authors claim that models can achieve better generalization and robustness, not only for social science tasks but potentially for broader applications requiring cultural and contextual nuance.
• Reciprocal Benefit: The initiative aims to benefit both communities: NLP researchers gain access to rich, fine-grained, expert-annotated data, while social scientists gain access to models trained on tasks aligned with their research needs.
• Caution Against Benchmarkization: The paper modestly warns against the risks of "benchmarkization," where excessive standardization could stifle the pluralism and interpretive nuance essential to social sciences. They advocate for benchmarks that allow for filtered evaluation (by discipline, time, culture) rather than collapsing results into a single aggregate score.

In conclusion, the paper argues that making the social sciences "count" in LLM research is essential for developing more robust, transparent, and socially relevant AI systems, and that BenCSSmark provides the necessary infrastructure to begin this dialogue.