The GDN-CC Dataset: Automatic Corpus Clarification for AI-enhanced Democratic Citizen Consultations

1. Problem Statement

The analysis of large-scale democratic citizen consultations (e.g., France's Grand Débat National) faces significant challenges due to the noisy, unstructured, and multi-topic nature of raw citizen contributions.

• Data Complexity: Contributions often mix facts, criticisms, proposals, and demands within a single text, making standard NLP tasks like topic modeling, summarization, and clustering difficult.
• Ethical & Transparency Concerns: Current solutions often rely on proprietary Large Language Models (LLMs) for end-to-end analysis. This raises ethical issues regarding opacity, bias, and dependence on private entities, which contradicts the transparency required for democratic processes.
• Scalability vs. Quality: Existing methods either focus on small, curated datasets (limiting scalability) or rely on powerful, closed-weight LLMs that obscure the decision-making process.

The authors propose a solution that standardizes these contributions to make them amenable to transparent, explainable, and locally-runnable NLP methods, while minimizing reliance on opaque, large-scale models.

2. Methodology: Corpus Clarification

The paper introduces a novel preprocessing framework called Corpus Clarification, which transforms raw, noisy contributions into structured, self-contained argumentative units. The process consists of three sequential steps:

1. Argumentative Unit (AU) Extraction:

   • Segments the raw text into distinct, topic-coherent units.
   • Handles non-contiguous segments (e.g., a text might discuss fuel taxes in two separate paragraphs; these are grouped into one AU).
   • Filters out non-argumentative noise.

2. Argumentative Structure (AS) Detection:

   • Classifies segments within each AU into three specific roles:
     • Statement: Expressions of sentiment or observation.
     • Solution: Proposals for action or policy change.
     • Premise: Justifications, evidence, or examples supporting a statement/solution.
   • Note: The authors distinguish "Solutions" from traditional "Claims" to better align with policy analysis needs.

3. Argumentative Unit Clarification:

   • Rewrites each AU to be context-independent (self-sufficient) and stylistically normalized.
   • Removes linguistic markers specific to the author (e.g., slang, incomplete sentences) and adds necessary context from the original text if the AU was fragmented.
   • Ensures all contributions are treated equally regardless of the writer's socioeconomic status or writing proficiency.

3. Key Contributions

A. The GDN-CC Dataset (Manually Curated)

• Source: 1,231 contributions from the French Grand Débat National (2019).
• Annotation: Contains 2,285 argumentative units manually annotated for structure (Statement, Solution, Premise) and clarified by human experts.
• Quality Control: A hybrid human-LLM annotation process was used. Annotators could regenerate clarifications using different LLMs (GPT-4.1, Qwen-32b, Llama-70b, Llama-8b) to select the best version, ensuring high-quality ground truth.
• Inter-Annotator Agreement: High agreement on AU extraction (WindowDiff ~0.08) and moderate-to-high agreement on structure detection (F1 ~0.72 for Solutions).

B. The GDN-CC-Large Dataset (Automatically Annotated)

• Scale: An automatically annotated corpus of 240,000 contributions (300,748 argumentative units).
• Method: Generated using the best-performing Small Language Models (SLMs) fine-tuned on the GDN-CC dataset.
• Significance: This is currently the largest annotated democratic consultation dataset available to the community and the first of its kind in French.

C. Methodological Innovation

• Corpus Clarification Task: Formalizes the transformation of raw text into structured data as a distinct NLP task.
• SLM Viability: Demonstrates that fine-tuned Small Language Models (SLMs) (e.g., Gemma-2-9b, Qwen2.5-7b) can match or outperform large proprietary models (like GPT-4.1) on these specific tasks when locally deployed.

4. Experimental Results

Model Performance (SLMs vs. LLMs)

The authors evaluated various models (Llama-3, Mistral, Qwen, Gemma) on the three tasks:

• AU Extraction: Fine-tuned SLMs (e.g., Qwen2.5-7b) achieved Micro-F1 scores of ~0.75, matching or exceeding GPT-4.1 (0.75).
• AS Detection: Fine-tuned Gemma-2-9b achieved a Macro-F1 of 0.79, outperforming GPT-4.1 (0.76).
• Clarification: Fine-tuned Gemma-2-9b achieved a BERTScore of 0.86 and ROUGE-L of 0.60, outperforming GPT-4.1.
• Error Analysis: Fine-tuning significantly reduced "Over-analysis" errors (adding unjustified justifications), a common failure mode in raw LLM outputs. The error rate for over-analysis dropped from 19% (GPT-4.1) to 2% (Fine-tuned Gemma).

Impact on Downstream Tasks (Clustering)

The authors tested the impact of clarification on opinion clustering (using UMAP + HDBSCAN):

• Metric Improvement: Using clarified AUs instead of raw contributions improved the Silhouette score from 0.46 to 0.59 and reduced the Davies-Bouldin score from 0.48 to 0.46 (lower is better).
• LLM-as-a-Judge: In pairwise comparisons, an LLM judge preferred clusters formed from clarified texts over raw contributions or unclarified AUs in 91% of cases, citing better thematic coherence.

5. Significance and Implications

• Democratizing AI: The work proves that transparent, local, and open-weight models are sufficient for high-quality democratic analysis, reducing reliance on opaque, proprietary APIs.
• Scalability: By releasing GDN-CC-large, the authors provide a resource that enables political scientists and NLP researchers to perform rigorous, large-scale analysis of public opinion without needing massive computational resources.
• Ethical Framework: The "Corpus Clarification" approach prioritizes legibility and standardization while attempting to preserve the original intent. It addresses the "black box" problem by making the preprocessing steps explicit and auditable, rather than relying on end-to-end black-box generation.
• Future Directions: The paper highlights that while SLMs are effective, careful prompt engineering and fine-tuning are crucial to prevent models from hallucinating justifications (over-analysis), which is critical for maintaining the integrity of democratic data.

In conclusion, the paper establishes a new standard for processing democratic data, showing that structured, fine-tuned small models can effectively clean and organize noisy citizen input, thereby enabling more reliable, transparent, and scalable political analysis.