💬 NLP

CascadeMind at SemEval-2026 Task 4: A Hybrid Neuro-Symbolic Cascade for Narrative Similarity

The CascadeMind system for SemEval-2026 Task 4 achieves competitive performance by employing a hybrid neuro-symbolic cascade that routes instances based on LLM vote consensus, demonstrating that confidence-aware compute allocation is more effective than adding auxiliary symbolic representations for narrative similarity.

Original authors: Sebastien Kawada, Dylan Holyoak

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sebastien Kawada, Dylan Holyoak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge trying to decide which of two short stories is more similar to a third "anchor" story. This is the challenge the CascadeMind system tackled in a competition called SemEval-2026.

Here is how the system works, explained through simple analogies:

1. The Problem: When to Trust Your Gut

Large Language Models (AI) are usually great at comparing stories. However, sometimes the stories are tricky, and the AI gets confused. If you ask an AI the same question once, it might give a wrong answer. But if you ask it eight times, the answers might vary.

The team behind CascadeMind discovered a clever trick: The more the AI agrees with itself, the more likely it is to be right.

If the AI votes "Story A" 7 or 8 times out of 8, it's almost certainly right (85% accuracy).
If the votes are split (like 4 vs. 4), the AI is confused, and its accuracy drops significantly.

2. The Solution: A Three-Step "Decision Cascade"

Instead of treating every story comparison the same way, CascadeMind uses a smart, tiered approach, like a customer service hotline that routes calls based on urgency.

Step 1: The Quick Vote (The "Supermajority" Rule)
The system asks the AI (specifically a model called Gemini 2.5 Flash) to vote on the answer eight times at once.

If 7 or 8 votes agree: The system says, "Great, we're confident!" and immediately picks that answer. This solves 74% of all problems instantly.
The Metaphor: It's like asking a room of eight experts. If seven of them say "Yes," you don't need to call a meeting; you just go with the consensus.

Step 2: The Escalation (The "Tie-Breaker" Round)
If the first eight votes are split (e.g., 4 vs. 4, or 5 vs. 3), the system knows the AI is unsure.

It doesn't give up; it asks the AI to vote 24 more times (for a total of 32 votes).
It then takes the majority of all 32 votes.
The Metaphor: This is like calling in a larger panel of experts to break a deadlock. It costs more time and money (compute power), but it helps solve the tricky cases that the first round missed.

Step 3: The "Symbolic" Fallback (The "Rulebook" Check)
If, after 32 votes, the AI is still perfectly tied (16 votes for A, 16 for B), the system gives up on the AI and switches to a human-designed rulebook.

This rulebook uses five specific "signals" based on old storytelling theories:
1. Word Overlap: Do they use the same vocabulary?
2. Story Structure: Do they have the same "beginning, middle, climax, end"?
3. Overall Meaning: Do the sentences mean the same thing?
4. Emotional Curve: Do the stories get exciting and sad at the same times?
5. Action Chains: Do the characters do the same types of actions (fighting, escaping, discovering)?
The Metaphor: When the experts are completely deadlocked, the system pulls out a physical manual written by literary scholars to make the final call.

3. The Surprising Results

The team entered this system into a competition with 44 other teams and finished 10th. But the most interesting part wasn't the ranking; it was why it worked.

The "Magic" was in the Routing: The system's success came almost entirely from knowing when to stop and when to ask more questions (the voting and escalation steps).
The Rulebook was Rarely Used: The "Symbolic Fallback" (the rulebook) was only triggered in 5% of cases.
The Big Reveal: When the researchers tested the system without the rulebook, the performance barely changed. They realized that the "Rulebook" wasn't the hero; the confidence check was. The system didn't need a fancy rulebook to solve hard problems; it just needed to know when to spend extra effort to get a clearer answer from the AI.

Summary

CascadeMind is like a smart manager who knows that:

Most problems are easy and can be solved by a quick group vote.
Hard problems need more time and a bigger group vote.
Impossible problems (perfect ties) are so rare that having a backup plan for them doesn't actually help the overall score much.

The main lesson from this paper is that for comparing stories, knowing when to spend more effort is more important than having a complex, fancy backup plan.

Technical Summary: CascadeMind at SemEval-2026 Task 4

Problem Definition
The paper addresses SemEval-2026 Task 4, which focuses on Narrative Story Similarity. The task requires systems to compare two candidate stories against an anchor story and determine which candidate is more similar based on three specific axes: abstract theme, course of action, and outcomes. The core challenge lies in the fact that narrative similarity extends beyond surface wording to include shared themes, event sequences, and resolutions, often leading to ambiguous comparisons where Large Language Models (LLMs) may degrade in performance.

Methodology: The CascadeMind Architecture
The authors propose CascadeMind, a hybrid neuro-symbolic system that utilizes a confidence-aware routing mechanism. The system treats the distribution of votes across self-consistency samples as an uncertainty signal, dynamically selecting a decision pathway based on the level of consensus.

The pipeline operates in four stages:

Neural Self-Consistency Voting: The system queries Gemini 2.5 Flash eight times ( $k=8$ ) with a temperature of 1.0 to generate eight independent A/B votes.
Confidence Check (Supermajority): If at least 7 out of 8 votes agree (supermajority), the system immediately commits to that decision. This path resolves the majority of cases (74% in development).
Escalation: If the initial votes are split (e.g., 4-4, 5-3, 6-2), the system escalates by issuing three additional API calls (totaling 32 votes) and takes the majority vote of the expanded set.
Symbolic Fallback (Tiebreaker): Only in the rare event of a perfect tie after escalation (5% of cases), the system falls back to a Multi-Scale Narrative Analysis Ensemble. This symbolic component computes a weighted sum of five similarity signals:
- Lexical Similarity: TF-IDF cosine similarity (Weight: 49%).
- Story Grammar Similarity: Average cosine similarity of sentence-transformer embeddings across five narrative phases (setting, conflict, rising action, climax, resolution) (Weight: 40%).
- Semantic Similarity: Whole-story embedding cosine similarity (Weight: 8%).
- Narrative Tension Curve: Pearson correlation of sentiment-derived tension curves (Weight: 2%).
- Event Chain Similarity: Longest Common Subsequence (LCS) of ordered action verbs (Weight: 1%).

The weights for the symbolic ensemble were optimized via differential evolution on a synthetic training set of 1,900 LLM-generated triplets.

Key Contributions
The paper outlines three primary contributions:

Confidence-Aware Routing: A demonstration that LLM vote agreement serves as a reliable monotone gradient for instance difficulty. On the development set, supermajority cases (≥7/8 votes) achieve 85% accuracy, split votes 67%, and perfect ties 61%.
Hybrid Fallback Mechanism: The implementation of a symbolic ensemble grounded in narrative theory (Propp, Freytag, Todorov) that is triggered exclusively on perfect ties, rather than as a primary classifier.
Ablation Analysis: Evidence showing that the performance gains are almost entirely driven by the confidence-aware routing (voting and escalation), while the symbolic component contributes negligibly to the end-to-end score because it is invoked in only 5% of cases.

Results

Official Performance: On the Track A test set (400 triplets), CascadeMind achieved 72.75% accuracy, ranking 10th out of 44 teams.
Development Diagnostics: On a development subset, the system achieved 81.0% accuracy.
Pathway Analysis:
- Supermajority cases (74% of the subset) resolved at 85% accuracy.
- Escalated cases resolved at 67% accuracy.
- The symbolic tiebreaker achieved 61.1% accuracy on a separate diagnostic set of perfect ties.
Symbolic Component Isolation: When applied as a standalone classifier across all cases (ignoring the neural cascade), the symbolic ensemble dropped to 53% accuracy, confirming its suitability only as a high-uncertainty fallback.
Discrepancy: The authors note a significant gap between the development cascade subset performance (81.0%) and the official test performance (72.75%), attributing this to distribution mismatch and potential overfitting to the synthetic calibration data.

Significance and Claims
The paper's primary methodological takeaway is that for narrative similarity tasks, calibrating when to spend additional compute on hard instances is more impactful than adding auxiliary representations. The authors argue that the system's success stems from the routing strategy—using vote consensus to distinguish easy, hard, and ambiguous cases—rather than the sophistication of the symbolic fallback. The symbolic component, while theoretically grounded, played a negligible role in the final score due to its low invocation rate. The work highlights the utility of self-consistency voting as a black-box uncertainty signal for selective prediction without requiring the model to abstain.

1. The Problem: When to Trust Your Gut

2. The Solution: A Three-Step "Decision Cascade"

3. The Surprising Results

Summary

Technical Summary: CascadeMind at SemEval-2026 Task 4

More like this