💬 NLP

UCSC NLP at SemEval-2026 Task 10: Boundary-Aware Span Extraction and RoBERTa Classification for Conspiracy Detection

The UCSC NLP team's systems for SemEval-2026 Task 10 (PsyCoMark) employ boundary-aware span extraction with hard-negative sampling and RoBERTa-based classification to detect conspiracy markers and documents, achieving 7th and 11th place rankings respectively while highlighting robust detection of entity-like roles versus sensitivity in abstract role boundaries.

Original authors: Dom Marhoefer, Milos Suvakovic, Glenn Grant-Richards, Aidan Pinero, Ryan King

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dom Marhoefer, Milos Suvakovic, Glenn Grant-Richards, Aidan Pinero, Ryan King

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking for fingerprints, you are looking for the specific "ingredients" that make a story feel like a conspiracy theory. This paper describes how a team from UC Santa Cruz built a computer program to do exactly that for a competition called SemEval-2026.

Here is the breakdown of their work, using simple analogies:

The Two Main Jobs

The competition had two distinct challenges, which the team tackled with two separate "detective agents" (AI models).

1. The Ingredient Hunter (Subtask 1)

The Goal: Find specific parts of a sentence that play a role in a conspiracy story. The roles are:
- The Actor: Who is doing the bad thing?
- The Victim: Who is getting hurt?
- The Action: What is the bad thing?
- The Evidence: What "proof" is being offered?
- The Effect: What is the scary result?
The Challenge: It's not enough to just find the word "Jeffrey." The computer has to know exactly where the sentence starts and ends. For example, is the "Action" just "spilled the beans," or is it the whole phrase "spilled the beans on the Mossad operation"?
The Trick: The team taught the computer to be a "boundary perfectionist." They told it: "If you guess the start or end of the phrase even slightly wrong, it doesn't count." They also used a technique called Hard-Negative Mining. Imagine a teacher showing a student a picture of a cat and saying, "That's a cat." Then, they show a picture of a very fluffy dog that looks like a cat and say, "That is not a cat." This helps the student learn the fine line between the two. The computer learned to spot the difference between a true conspiracy marker and a phrase that almost looks like one but isn't quite right.

2. The Story Judge (Subtask 2)

The Goal: Read the whole document and decide: Is this a conspiracy theory? Is it definitely not? Or is it impossible to tell?
The Trick: This model didn't look at the specific ingredients found by the first detective. Instead, it read the whole story like a human reader, looking for the overall "vibe" or tone. It learned to distinguish between a critical news report (which might mention an actor and a victim but isn't a conspiracy) and a true conspiracy narrative (which uses those same roles but with a suspicious, secretive tone).

How They Did It

The team used a powerful AI brain called RoBERTa. Think of this as a very well-read librarian who has read almost everything on the internet and understands how words fit together.

For the Ingredient Hunter: They didn't just ask the librarian to label words one by one. Instead, they asked the librarian to look at every possible chunk of text (up to 32 words long) and ask, "Does this chunk fit the definition of 'Evidence'?" They trained the librarian to be extremely picky about the start and end dates of these chunks.
For the Story Judge: They fed the whole story to the librarian and asked for a simple verdict: Yes, No, or Maybe.

The Results

When the team entered their detectives into the official competition:

The Ingredient Hunter ranked 7th out of all teams. It was very good at finding the "Actor" and "Victim" (like people's names), but it sometimes struggled with the more abstract parts like "Action" or "Effect," where the boundaries of the phrase are fuzzier.
The Story Judge ranked 12th. It was quite good at telling the difference between a conspiracy story and a regular news story.

What They Learned (The "Aha!" Moments)

Precision is Hard: The biggest mistake the computer made was getting the start and end of a phrase slightly wrong. If the "Evidence" was the word "report," the computer sometimes guessed "a report" or "according to a report." The competition rules were strict: if you didn't get the exact words, you got zero points.
Two Heads are Better Than One (Separately): The team decided not to let the two detectives talk to each other while they were working. They thought, "If the first detective makes a mistake, we don't want the second detective to get confused by that mistake." They kept them separate and only combined their answers at the very end.
The "Can't Tell" Problem: The competition had a third option called "Can't tell." However, the final scoring rules ignored this category. This meant the computer's performance looked different on the test day compared to how it performed during practice, simply because the rules changed slightly at the finish line.

In Summary

The UC Santa Cruz team built a system that acts like a very precise editor and a keen observer. They showed that while computers are getting better at spotting the ingredients of conspiracy theories, they still struggle with the exact boundaries of those ingredients. However, they are quite good at sensing the overall mood of a story to decide if it's a conspiracy or not.

Technical Summary: UCSC NLP at SemEval-2026 Task 10 (PsyCoMark)

Problem Definition

The paper addresses SemEval-2026 Task 10 (PsyCoMark), which focuses on the detection of conspiracy narratives within English text. The task is divided into two distinct subtasks:

Subtask 1 (Conspiracy Marker Extraction): Identifying and extracting spans of text corresponding to five specific semantic roles: Actor, Action, Effect, Evidence, and Victim. The challenge lies in the fact that these role structures appear in both conspiratorial and non-conspiratorial (critical) discourse; the distinction relies on framing and epistemic stance rather than the mere presence of these roles.
Subtask 2 (Document-Level Conspiracy Detection): Classifying entire documents into one of three categories: Yes (conspiracy), No (non-conspiracy), or Can't tell.

The dataset consists of over 4,100 Reddit submissions. A key difficulty noted is the dense annotation of markers and the moderate class imbalance, particularly in the document-level labels.

Methodology

The authors propose two independent systems based on RoBERTa-large, trained separately without parameter sharing or cross-task feature transfer.

Subtask 1: Boundary-Aware Span Extraction

Rather than token-level tagging, the authors formulate the task as multi-label span classification over enumerated candidate spans.

Architecture: The model enumerates all candidate spans up to a maximum length of $L=32$ $L = 32$ tokens. For each span $(i, j)$ $(i, j)$ , a contextual representation is constructed by concatenating:
- Start and end token embeddings ( $h_i, h_j$ ).
- Mean-pooled embeddings of tokens within the span ( $\bar{h}_{i:j}$ ).
- A learned embedding encoding the span width ( $w_{emb}$ ).
- Context embeddings of the immediate left and right tokens ( $h_{i-1}, h_{j+1}$ ).
  This 6-component vector (dimension 6144) is passed through a two-layer MLP to predict the five roles.
Training Strategy:
- Labeling: To enforce precise boundary learning, a candidate span is labeled positive only if its Intersection-over-Union (IoU) with a gold span is $\ge 0.95$ .
- Sampling: A three-tier sampling strategy is used to handle the overwhelming number of negative spans: (1) all gold-aligned spans, (2) hard negatives (spans with partial overlap, IoU $\in [0.50, 0.75)$ ), and (3) random non-overlapping negatives.
- Loss: Binary Cross-Entropy with logits, weighted per role to address class imbalance.
Decoding: Predictions undergo post-processing involving per-role probability thresholding (tuned on validation) and containment-based Non-Maximum Suppression (NMS). NMS prunes overlapping spans based on containment ratios and IoU thresholds, followed by merging adjacent spans of the same role separated by small character gaps.

Subtask 2: Document Classification

Architecture: A standard sequence classification approach using the [CLS] token representation from RoBERTa-large.
Training: The model is fine-tuned with Cross-Entropy loss and label smoothing ( $\alpha=0.05$ ) to reduce overconfidence. A stratified 90/10 train-validation split is used.
Independence: This model operates solely on raw text, ignoring the span predictions from Subtask 1 during training and inference.

Key Results

Subtask 1: The system ranked 7th out of participating teams with a macro F1 of 0.2251 on the official test set (evaluated with token-level IoU $\ge 0.5$ $\geq 0.5$ ).
- Internal Validation: The model achieved a decoded micro F1 of 0.6501 on the validation set (using character-level IoU $\ge 0.3$ ).
- Performance Disparity: The drop in performance is attributed to the stricter token-level evaluation metric and the sensitivity of abstract roles (Action, Effect, Evidence) to boundary criteria. Entity-like roles (Actor, Victim) showed higher robustness.
Subtask 2: The system ranked 12th with a weighted F1 of 0.7694 on the official test set.
- Note: The official evaluation excluded "Can't tell" instances, computing metrics over 608 samples, which differs from the three-class validation setup.

Significance and Claims

The paper claims that its primary contribution lies in the specific formulation of span extraction for conspiracy markers and the analysis of boundary sensitivity.

Hard Negative Mining: The authors identify the use of hard negatives (partial overlap spans) as critical for preventing the model from predicting overly broad spans, thereby sharpening boundary localization.
Independence vs. Coupling: The system demonstrates that while marker extraction and document classification are intrinsically related, they can be modeled independently. The document classifier achieved competitive performance without explicit role features, suggesting that conspiracy signals can be learned directly from raw text.
Evaluation Alignment: The paper emphasizes that performance is highly sensitive to the alignment between development metrics and the official evaluation protocol. The discrepancy between validation and test scores is largely attributed to the shift from relaxed character-level IoU criteria during development to strict token-level IoU criteria in the final evaluation.
Limitations: The authors modestly acknowledge that abstract roles remain difficult due to semantic abstraction and variable span lengths. They note that the fixed maximum span length ( $L=32$ ) may impose a recall ceiling for longer, multi-clause markers.

The work concludes that while current systems can learn conspiracy signals, future improvements may require joint modeling of roles and classification, structured span decoding, and the integration of psycholinguistic features to better distinguish conspiratorial framing from critical narrative.