💬 NLP

Duluth at SemEval-2026 Task 6: DeBERTa with LLM-Augmented Data for Unmasking Political Question Evasions

The Duluth team's SemEval-2026 Task 6 entry, which combines a DeBERTa-V3-base model with LLM-generated synthetic data to address class imbalance, achieved an 8th-place ranking with a Macro F1 of 0.76 in classifying political question evasions, demonstrating that such augmentation effectively improves minority-class recall despite persistent challenges in distinguishing ambivalent from clear responses.

Original authors: Shujauddin Syed, Ted Pedersen

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shujauddin Syed, Ted Pedersen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a live TV interview with a President. The reporter asks a direct question: "Will you raise taxes?"

The President might answer in three ways:

The Direct Answer: "Yes, we will." (Clear Reply)
The "Maybe" Answer: "We are looking into the numbers, but it depends on the economy..." (Ambivalent)
The "No" Answer: "I'm not going to talk about that right now." (Clear Non-Reply)

The challenge is that politicians are experts at Answer #2. They use fancy words, vague promises, and deflections to avoid giving a straight "yes" or "no."

This paper describes how a team of computer scientists from the University of Minnesota Duluth built a robot brain (an AI) to spot these evasive answers during the SemEval-2026 competition. Here is how they did it, explained simply.

1. The Problem: The AI Was Biased

The team started with a dataset of thousands of real interview clips. But there was a big problem: The data was unbalanced.

Most answers were "Ambivalent" (the tricky, evasive ones).
Very few were "Clear Non-Replies" (the outright refusals).

It was like trying to teach a student to recognize rare animals (like a snow leopard) when 90% of the pictures in their textbook were just dogs. The AI kept guessing "Dog" (Ambivalent) because that's what it saw most often, and it failed to spot the rare "Snow Leopards."

2. The Solution: The "Fake Data" Factory

To fix this, the team didn't just collect more real interviews (which takes forever). Instead, they hired two super-smart AI assistants, Gemini 3 and Claude Sonnet 4.5, to act as creative writers.

They told these AI writers: "Write 2,000 new fake interview answers that sound exactly like the rare 'Clear Non-Reply' type, but make them up."

Gemini was the "Context Master": It wrote new scenarios where a politician politely refused to answer, mixing in realistic political jargon.
Claude was the "Paraphrase King": It took existing answers and rewrote them in different ways to create more variety.

By adding these synthetic (fake but realistic) examples, they balanced the class sizes. Now, the AI student had a textbook with an equal number of "Dogs" and "Snow Leopards."

3. The Engine: A Specialized Brain

The team used a powerful AI model called DeBERTa. Think of DeBERTa as a very advanced reading comprehension engine. But they didn't just use it "out of the box." They gave it three special upgrades:

Focal Loss (The "Focus Filter"): Imagine a student who gets bored with easy questions and only studies the hard ones. This technique told the AI: "Don't waste time on the easy 'Ambivalent' answers you already know. Spend all your energy figuring out the tricky ones you keep getting wrong."
Layer-wise Learning (The "Gradual Update"): The AI has many layers of "thinking." The team told the bottom layers (which know general language) to stay calm and not change much, while letting the top layers (which need to learn political tricks) change rapidly.
Boolean Features (The "Cheat Sheet"): They gave the AI two simple yes/no flags from the data, like "Is this a question with multiple parts?" or "Is it a yes/no question?" This helped the AI pay attention to the structure of the sentence, not just the words.

4. The Results: Top 20%

When they tested their system:

They finished 8th out of 40 teams.
They scored a 0.76 (on a scale where 1.0 is perfect).
The winning team scored 0.89, but the average team only scored 0.70.

This means their "Fake Data Factory" strategy worked. They significantly improved the AI's ability to spot the rare, evasive answers.

5. The Remaining Mystery: The "Gray Zone"

Despite their success, the AI still struggled with one specific thing: Distinguishing between a "Maybe" and a "Yes."

The Confusion: If a politician says, "We will consider raising funds, but we need to study the impact," is that a Clear Reply (they said they will consider it) or Ambivalent (they are hedging)?
The Human Factor: The paper notes that even humans argue about this. Annotators (the people who labeled the data) often disagreed on these "gray zone" answers.

The AI made the same mistakes humans did. It got confused by "hedging" words (like "maybe," "consider," "study").

The Takeaway

The Duluth team showed that you can teach an AI to spot political dodging by generating fake examples to fill in the gaps where real data is missing. While the AI still gets tripped up by the most subtle, tricky answers (the ones where even humans can't agree), this approach proved that synthetic data is a powerful tool for making AI smarter about complex human behavior.

In short: They taught a robot to spot a politician's "maybe" by feeding it a diet of AI-generated "maybe" examples, helping it see through the fog of political spin.

1. Problem Definition

The paper addresses SemEval-2026 Task 6 (CLARITY), which focuses on detecting and classifying evasive communication strategies in political discourse. The task is grounded in political science theories regarding equivocation and involves analyzing Question-Answer (QA) pairs from U.S. presidential interviews.

The task is divided into two subtasks based on a hierarchical taxonomy:

Task 1 (Clarity-level Classification): A 3-class classification problem where the model must categorize a response as:
- Clear Reply: Explicitly provides requested information.
- Ambivalent: Provides a response that allows multiple interpretations (e.g., implicit, general, partial, or deflective).
- Clear Non-Reply: Open refusal, claim of ignorance, or request for clarification.
Task 2 (Evasion-level Classification): A 9-class fine-grained classification problem mapping responses to specific evasion techniques (e.g., Dodging, Deflection, Declining to answer).

Key Challenge: The dataset suffers from severe class imbalance (Ambivalent: ~59%, Clear Reply: ~31%, Clear Non-Reply: ~10%), leading to poor recall for minority classes in standard models. Additionally, the distinction between "Ambivalent" and "Clear Reply" is linguistically subtle and often results in inter-annotator disagreement.

2. Methodology

The authors propose a system based on the DeBERTa-V3-base transformer architecture, enhanced with specific training strategies and data augmentation techniques.

A. Data Augmentation (Addressing Imbalance)

To mitigate the class imbalance, the team employed Large Language Model (LLM)-based synthetic data generation to augment the minority classes (Clear Reply and Clear Non-Reply). Two distinct strategies were used:

Context-Aware Synthetic Generation (CASA): Using Gemini 3, the system extracted rhetorical frames from minority classes and combined them with randomized political contexts. This generated 2,672 synthetic examples, perfectly balancing the dataset to 2,040 samples per class.
Lexical Paraphrasing (EDA-inspired): Using Claude Sonnet 4.5, the system applied synonym replacement, random insertion, swap, and deletion to answer texts. This generated 1,086 examples, creating a partially balanced distribution.

Quality Control: 50 random samples were manually inspected.
Training Strategy: A confidence-weighted loss function was applied during training (0.7× weight for Gemini data, 0.5× for Claude data) to reduce overfitting on synthetic data.

B. Model Architecture & Enhancements

The core model is Microsoft DeBERTa-V3-base (184M parameters), chosen for its superior performance on nuanced political language compared to BERT and DistilBERT. The system was extended with:

Boolean Discourse Features: Two statistically significant binary features from the original data (affirmative_questions and multiple_questions) were extracted, processed through a small neural network (Linear→ReLU→Dropout), and concatenated with the pooled transformer output.
Focal Loss: Implemented to focus training on hard examples and re-weight the signal toward minority classes. The formula used is $FL(p_t) = -\alpha_t(1 - p_t)^\gamma \log(p_t)$ with $\gamma = 2.0$ .
Layer-wise Learning Rate Decay (LLRD): Lower layers received progressively smaller learning rates ( $\alpha = 0.9$ ) to preserve pre-trained knowledge while allowing task-specific adaptation in higher layers.
Training Configuration: Cosine annealing scheduler with 15% warmup, gradient accumulation (effective batch size 32), and early stopping based on validation Macro F1.

3. Key Contributions

LLM-Augmented Data Strategy: Demonstrated that synthetic data generated by advanced LLMs (Gemini 3 and Claude Sonnet 4.5) can effectively balance training sets and improve minority-class recall in political discourse tasks.
Hybrid Feature Integration: Successfully integrated discrete boolean discourse features with continuous transformer embeddings to capture specific linguistic cues (e.g., question types) relevant to evasion.
Error Analysis & Human Alignment: Identified that the primary source of error (confusion between Ambivalent and Clear Reply) mirrors human annotator disagreements, suggesting the model has reached a ceiling similar to human performance on these ambiguous cases.

4. Results

The system was evaluated on the official test and evaluation sets using Macro F1 as the primary metric.

Task 1 Performance:
- The Gemini-augmented DeBERTa-V3-base achieved a Macro F1 of 0.76 on the evaluation set.
- This placed the team 8th out of 40 participating teams.
- Comparison: The top-ranked system (TeleAI) scored 0.89, while the mean score was 0.70. The Duluth system placed in the top 20%.
- Impact of Augmentation: The augmentation strategy improved the evaluation Macro F1 from 0.69 (original data only) to 0.76.
- Per-Class Performance: The model achieved an F1 of 0.73 for Ambivalent, 0.56 for Clear Reply, and 0.62 for Clear Non-Reply. Notably, it achieved zero false positives for Clear Non-Reply when the true label was Clear Reply.
Task 2 Performance:
- The system achieved a Macro F1 of 0.28 on the evaluation set (Rank 30/33).
- The authors attribute this sharp drop from the development phase (0.45) to a distribution shift between the phases, highlighting the difficulty of fine-grained classification without robust validation.

5. Significance and Future Directions

Significance: The paper validates that LLM-based data augmentation is a viable and effective strategy for handling class imbalance in specialized NLP tasks like political discourse analysis. It also highlights that the "gray zone" between evasive and clear answers is the most difficult aspect of the problem, often indistinguishable even by humans.
Future Work: The authors propose three directions for improvement:
1. Multi-task Learning: Jointly predicting clarity and evasion-level labels to leverage the hierarchical taxonomy.
2. Hedging-Aware Features: Explicitly detecting qualifiers and conditionals to better distinguish partial answers from direct ones.
3. Robust Validation: Developing better validation strategies for fine-grained tasks (Task 2) to handle distribution shifts.

In conclusion, the Duluth system demonstrates that combining state-of-the-art transformers with targeted synthetic data augmentation and specialized loss functions can significantly advance the state-of-the-art in detecting political question evasions.