💬 NLP

CLaC at SemEval-2026 Task 6: Response Clarity Detection in Political Discourse

The CLaC team's system for SemEval-2026 Task 6 demonstrates that prompt-based large language models outperform fine-tuned encoders in detecting response clarity and evasion in political discourse, achieving top-tier results through an ensemble strategy and enriched input contexts while highlighting the persistent challenge of distinguishing between clear and ambivalent replies.

Original authors: Nawar Turk, Lucas Miquet-Westphal, Leila Kosseim

Published 2026-05-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nawar Turk, Lucas Miquet-Westphal, Leila Kosseim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a live TV debate where a politician is being grilled by a tough journalist. Sometimes the politician answers the question directly. Sometimes they give a vague, "maybe-so" answer that sounds like an answer but isn't. And sometimes, they completely dodge the question, pretending they didn't hear it or saying they don't know.

The paper you're reading is about a team of computer scientists (from Concordia University) who built a "smart referee" to automatically grade these answers. They entered a competition called SemEval-2026 Task 6 to see who could build the best referee.

Here is the story of how they did it, explained simply:

The Two Levels of the Game

The competition had two levels, like a video game with an "Easy Mode" and a "Hard Mode":

Level 1 (The Big Picture): Is the answer Clear, Vague, or a Total Dodge? (3 categories).
Level 2 (The Fine Print): If the answer is vague or a dodge, exactly how did they do it? (9 specific categories, like "deflecting," "being too general," or "claiming ignorance").

The Three Strategies They Tried

The team tried three different ways to build their referee:

1. The "Specialized Trainers" (Encoder Models)
Think of these as students who have memorized thousands of examples of political answers. They are very good at spotting patterns, but they are rigid.

The Experiment: They tried 8 different "students" (computer models) and taught them using a four-step training process.
The Trick: They found that if they let the students memorize everything (full training), the students got confused and made mistakes. But if they told the students, "You already know the basics, just tweak the top 25% of your brain," the students performed much better.
The Result: Even when they combined all 8 students into a "study group" (an ensemble), they were good, but not the best.

2. The "Long-Context Reader" (Longformer)
Some interviews are very long. The team tried a special model designed to remember long stories.

The Result: It didn't work as well as expected. Even though it could "read" longer texts, it didn't understand the nuance of the answers better than the standard models. It's like having a library card for a huge library but still not knowing how to find the right book.

3. The "Super-Intelligent Tutors" (Large Language Models / LLMs)
These are the "genius" models (like GPT-5, Gemini, and Qwen) that haven't been specifically trained on this dataset. Instead, the team gave them a very detailed instruction manual (a "prompt") and showed them 27 examples of good answers before asking them to judge.

The Secret Sauce: The team realized that the way they asked the question mattered more than the size of the model.
- They gave the models the full context (the whole conversation, not just the question).
- They gave the models a "cheat sheet" that explained the tricky categories in detail.
- They told the models to "think about it internally" before giving an answer.
The Result: These "Tutors" crushed the competition. They didn't need to be retrained; they just needed the right instructions.

The Big Wins and Surprises

The "Study Group" Effect: The team combined their three best "Tutors" (GPT-5, Gemini, and Qwen) into a final team. If two tutors agreed on an answer, that was the final verdict. This team got 80 out of 100 on the easy level and 59 out of 100 on the hard level.
Size Doesn't Matter: You might think a bigger, more powerful model would always win. But the team found that a massive 253-billion-parameter model actually performed worse than a smaller, smarter 70-billion-parameter model. It's not about how big the brain is; it's about how you talk to it.
The "Vague" Problem: The hardest part for both the computers and the human judges was telling the difference between a "Clear Answer" and a "Vague Answer." Even the humans disagreed on this often. The computers made the same mistake, showing that some political answers are just naturally confusing.

The Bottom Line

The team's "Super-Intelligent Tutor" system was the 9th best out of 41 teams for the easy level and the 3rd best out of 33 for the hard level.

The main lesson? When trying to understand tricky political answers, you don't necessarily need to build a new, super-expensive computer brain from scratch. Sometimes, you just need to take a smart, existing brain, give it a really good instruction manual, show it a few examples, and let it do the work.

They also noted that their code and instructions are free for anyone to use, so other researchers can try to beat their score next time!

Technical Summary: CLaC at SemEval-2026 Task 6 (CLARITY)

Problem Definition

The paper addresses SemEval-2026 Task 6 (CLARITY), a shared task focused on detecting response clarity and evasion in political discourse. The system processes English question-answer pairs extracted from U.S. presidential interviews. The task is structured hierarchically:

Task 1: Classify responses into one of three Clarity categories: Clear Reply, Ambivalent Reply, or Clear Non-Reply.
Task 2: Predict one of nine fine-grained Evasion labels that subcategorize the Clarity classes (e.g., Dodging, Deflection, Explicit, Partial).

The dataset comprises 3,993 instances with significant class imbalance (dominated by Ambivalent in Level 1 and Explicit/Dodging in Level 2) and inherent annotation difficulty, evidenced by moderate inter-annotator agreement ( $\kappa = 0.65$ ) on the boundary between Clear Reply and Ambivalent.

Methodology

The CLaC team evaluated three distinct architectural approaches, optimizing them through a multi-stage pipeline:

1. Encoder-Based Models

The team evaluated 8 transformer encoders (including English variants like BERT, RoBERTa, DeBERTa and multilingual variants like mBERT, XLM-RoBERTa, mDeBERTa) across four optimization stages:

Stage 1 (Encoder Adaptation): Compared full fine-tuning against partial layer unfreezing (top 25%, 50%, 75%) and LoRA. Partial unfreezing (specifically the top 25%) significantly outperformed full fine-tuning.
Stage 2 (Classification & Representation): Evaluated different classification heads (Linear over CLS, MLP, Mean Pooling) and truncation strategies (standard vs. head-tail).
Stage 3 (Loss & Regularization): Tested Focal Loss and varying dropout rates against a baseline of Weighted Cross-Entropy (WCE) with 0.1 dropout. WCE proved sufficient.
Stage 4 (Data & Input): Tested data augmentation (LLM-generated paraphrases for the minority Clear Non-Reply class) and enriched input (concatenating the full interviewer turn with the target question).
Ensemble: The best configuration for each of the 8 encoders was combined via majority voting.

2. Long-Context Architecture

To address the ~30% of instances exceeding the 512-token limit, the team experimented with Longformer-base-4096. They conducted an ablation study on input ordering, global attention patterns (CLS-only vs. extended to question tokens), and classification heads. Despite the extended context window, this approach did not outperform the optimized encoder ensemble.

3. Prompt-Based LLMs

The team evaluated open-weight (LLaMA, Qwen, Mixtral) and proprietary (GPT-5, Claude-Opus-4.5, Gemini-3-Flash-Preview) Large Language Models without task-specific parameter updates.

Prompting Strategies: Evaluated Zero-Shot, Zero-Shot with repeated instructions, Few-Shot (3, 9, 27 shots), and Chain-of-Thought (internal reasoning only).
Optimization: The best-performing strategy (FS27 with enriched input) was refined by augmenting label definitions with Level-2 Evasion subcategories and, for some models, prepending the president's name.
Ensemble: The top 3 performing LLM configurations (GPT-5, Gemini-3-Flash-Preview, Qwen3-235B) were combined via majority voting.

Key Results

Task 1 Performance: The LLM ensemble achieved the highest performance with 80 macro-F1 on the test set (ranking 9th/41). This significantly outperformed the encoder ensemble (70.5 macro-F1 on dev, lower on test) and the Longformer model (64.3 macro-F1).
Task 2 Performance: Applying the Task 1 LLM ensemble configuration to Task 2 yielded 59 macro-F1 (ranking 3rd/33).
Encoder Findings:
- Partial Unfreezing: Unfreezing the top 25% of layers improved macro-F1 by +17.9 over full fine-tuning (59.1 vs. 41.2), suggesting full fine-tuning overfits the limited data.
- Ensemble Diversity: Combining English and multilingual encoders (70.5) outperformed ensembles of either family alone, despite multilingual models individually underperforming English-only models.
- Input Enrichment: Adding the full interviewer turn degraded encoder performance but improved LLM performance.
LLM Findings:
- Prompt Engineering: Prompt refinement (adding subcategory definitions) drove a +7.1 macro-F1 gain for GPT-5.
- Model Scale vs. Performance: Parameter count did not predict performance; smaller models (e.g., Qwen3-30B) sometimes outperformed larger ones (e.g., LLaMA-3.1-Nemotron-Ultra-253B).
- Minority Classes: The LLM ensemble showed superior robustness on minority classes (Clear Reply and Clear Non-Reply), improving F1 by nearly 10 points over the encoder ensemble.

Significance and Claims

The paper claims that for low-resource, hierarchically labeled tasks like CLARITY, inference-time design choices (prompt structure, label hierarchy integration, and ensemble composition) are more impactful than parameter-level adaptation (fine-tuning).

Key observations include:

Prompting Superiority: Prompt-based LLMs, without any fine-tuning, outperformed fine-tuned encoders, particularly on minority classes.
Context Utilization: The divergence in performance regarding "enriched input" (helping LLMs but hurting encoders) suggests that LLMs leverage additional context more effectively than encoder architectures, a factor not solely attributable to sequence-length capacity.
Inherent Ambiguity: The dominant failure mode for both systems was the confusion between Clear Reply and Ambivalent, mirroring human annotator disagreement. This indicates the boundary is inherently subjective and difficult for both humans and models.
Ensemble Strategy: Architectural diversity (combining English and multilingual models) provided complementary signals that improved overall ensemble performance more than increasing model scale alone.

The authors note limitations regarding the dataset's restriction to U.S. presidential interviews, the reliance on proprietary APIs for reproducibility, and the lack of a dedicated prompt sweep for Task 2, which was addressed by transferring the Task 1 configuration.