💬 NLP

Counter with Evidence! A Multi-Agent Memory Efficient Reasoning Framework for Hate Category Informed Counterspeech Generation

The paper introduces FIRE, a multi-agent framework that improves hate speech counterspeech generation by categorizing abuse into five distinct types and mapping them to targeted strategies, supported by a new dataset (FactualCS) and demonstrating superior factual accuracy and reduced toxicity compared to existing baselines.

Original authors: Sujoy Nath, Aswini Kumar, Tanmoy Chakraborty

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sujoy Nath, Aswini Kumar, Tanmoy Chakraborty

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Online hate speech is a pervasive problem that poisons digital conversation, causing real psychological harm to vulnerable communities. While social media platforms connect the world, they are also frequently weaponized to spread abuse. To combat this, researchers have long sought a way to generate "counterspeech"—polite, fact-based replies that neutralize hostility without simply deleting the offending content. The challenge has been that hate speech is not a single, uniform thing. It comes in many forms: some posts spread verifiable lies, others rely on harmful stereotypes, while some use conspiracy theories or dehumanizing language to strip people of their dignity. Treating all of these as the same problem leads to weak responses. A reply that works to debunk a fake statistic will fail to address a cruel insult, just as a moral argument cannot fix a factual error. The core question for scientists in this field is how to build a system that can distinguish between these different types of abuse and respond with the specific strategy each one requires.

A team of researchers at the Indian Institute of Technology Delhi has developed a new approach to solve this problem, moving away from the idea of a single, all-knowing computer program. Instead, they created a system called FIRE, which stands for Factuality Informed Multi-Agent REasoning Framework. Rather than trying to force one large computer model to do everything at once, FIRE breaks the task down into smaller, specialized steps. It operates like a small team of experts working together. First, a "Hate Speech Analyst" examines the abusive message to determine exactly what kind of hate it is. Is it a stereotype? A conspiracy theory? A lie? Once the type is identified, the system decides on the best way to fight back. If the hate speech contains a false claim, the system automatically searches the internet for real evidence to prove it wrong. Finally, a "Counterspeech Generator" uses this analysis and the gathered evidence to write a response that is tailored to the specific situation.

To teach this system how to work, the researchers had to create a new kind of training material, because existing data sets were not detailed enough. They curated a collection called FactualCS, which contains nearly 4,800 examples of hate speech paired with high-quality counterspeech. What makes this dataset unique is that every example is carefully labeled with the specific category of hate, the reasoning behind the chosen response, and the actual evidence used to support the rebuttal. This allows the system to learn not just what to say, but why a particular response is effective for a particular type of abuse. The researchers trained their system using very small computer models, each with fewer than two billion parameters, which is tiny compared to the massive models often used in artificial intelligence today. Despite their small size, these models were able to perform complex reasoning because they were guided by the structured steps of the FIRE framework and the rich information in the new dataset.

When the researchers tested their system against other leading methods, the results were striking. The FIRE system significantly outperformed existing tools in generating responses that were both factually accurate and appropriate for the specific type of hate speech it was addressing. It improved the accuracy of identifying the right response strategy by about 11 percent and increased the factual correctness of the replies by roughly 12 percent. Perhaps most importantly, the responses generated by FIRE were less toxic and more respectful than those from other systems. In tests where human experts compared the outputs, they consistently preferred the responses from FIRE over those from much larger, more powerful models. The system managed to achieve these results while using far less computing power, proving that a smart, organized approach can be more effective than simply using a bigger, more expensive computer.

The researchers also tested what would happen if they removed parts of the system to see how important each piece was. When they took away the tool that searches the web for evidence, the system's ability to tell the truth dropped significantly, showing that having access to real facts is crucial for debunking lies. When they removed the memory that helps the system remember past examples, the responses became less coherent and more repetitive. This confirmed that the success of the project relies on the combination of all its parts: the ability to analyze the problem, the ability to find evidence, and the ability to remember how to speak effectively. The study suggests that by breaking down the complex task of fighting hate into smaller, manageable steps, and by grounding every response in evidence, we can create tools that are safer and more effective for real-world use. While the system is not perfect and still faces challenges with very ambiguous language or cultural nuances, it represents a significant step forward in creating automated tools that can engage with online hate in a way that is intelligent, factual, and human-centered.

Technical Summary: FIRE – A Multi-Agent Memory Efficient Reasoning Framework for Hate Category Informed Counterspeech Generation

1. Problem Statement

Current automated counterspeech (CS) generation systems face a critical limitation: they largely treat hate speech (HS) as a homogeneous entity. While prior work has advanced stylistic control (e.g., politeness, emotion), it often fails to distinguish between the distinct logical structures of abusive content. Online hate manifests through diverse categories—such as misinformation (verifiable false claims), stereotypes (oversimplified generalizations), conspiracy theories, dehumanization, and non-factual assertions—each requiring fundamentally different refutation strategies.

Monolithic Large Language Models (LLMs) struggle to simultaneously perform deep semantic reasoning (distinguishing hate types), factual verification, and stylistic coherence. Without explicit task decomposition, these models frequently hallucinate facts or generate context-agnostic responses that fail to address the core premise of the abuse. Furthermore, existing datasets lack the intermediate reasoning traces and evidence mappings necessary to supervise an agent's logic, relying instead on simple input-output pairs.

2. Methodology: The FIRE Framework

To address these gaps, the authors introduce FIRE (Factuality Informed Multi-Agent REasoning Framework). FIRE employs a hierarchical, multi-agent paradigm using compact models (<2B parameters) to decouple reasoning from generation, avoiding the need for fine-tuning large models.

Core Components:

FactualCS Dataset: A novel dataset of 4,784 hate-counterspeech pairs annotated across 14 target communities. Unlike prior resources, it includes dense annotations for hate categories, reasoning traces, search queries, and evidence mappings.
Memory Module: A lightweight, trained contrasting encoder (22M parameters) that maps hate speech inputs to embeddings. It clusters instances by hate type, enabling the retrieval of top- $k$ exemplars (reasoning patterns and stylistic examples) from a fixed index without fine-tuning the language models.
Two-Phase Agentic Workflow:
1. Phase 1: Hate Explanation with Analysis and Lookup (HEAL):
  - Hate Speech Analyst (HSA): A specialized agent (<2B) that analyzes the input hate speech ( $h$ ) and retrieved exemplars ( $E$ ). It predicts the target group ( $g$ ), hate type ( $t$ ), reasoning trace ( $r$ ), and a search query ( $q$ ).
  - Purity Score ( $\rho$ ): The system calculates the homogeneity of the retrieved neighbors. If the score is low (indicating ambiguity), the HSA is prompted to choose between top candidate hate types with justification.
  - Web Search Tool: If the detected hate type falls into factual categories (misinformation, stereotype, conspiracy), the system triggers an external web search using the generated query ( $q$ ) to retrieve supporting evidence ( $e$ ). For non-factual hate, $e$ is set to empty.
2. Phase 2: Category-aware Agentic Response Engine (CARE):
  - Counterspeech Generator (CSG): A second agent (<2B) synthesizes the final response ( $c$ ). It conditions on the original input, the structured analysis from the HSA, the retrieved evidence, and stylistic patterns from the memory exemplars. This ensures the response is factually grounded and tailored to the specific hate category.

Key Design Choice: The framework relies on in-context learning via the memory module and structured prompts rather than supervised fine-tuning (SFT) of the language models. Only the small encoder for the memory module is trained using a supervised contrastive loss to separate hate types.

3. Key Contributions

FIRE Framework: A multi-agent reasoning system that decomposes counterspeech generation into intent analysis and grounded synthesis, utilizing specialized agents to handle semantic diversity and factual verification.
FactualCS Dataset: A curated resource of 4,784 instances providing the first comprehensive annotations for hate categories, reasoning traces, and evidence mappings, addressing the data scarcity for grounded generation.
Performance with Compact Models: Demonstrating that a system using two <2B parameter agents can outperform larger, monolithic models (e.g., LLaMA-3.1-8B) by leveraging task decomposition and external knowledge retrieval.

4. Experimental Results

The authors evaluated FIRE against 28 baseline configurations, including zero-shot/few-shot prompting, retrieval-based RAG, and supervised fine-tuning (SFT) of various architectures (BART, T5, LLaMA, Mistral, Qwen).

Quantitative Metrics: FIRE achieved state-of-the-art performance across 9 of 13 metrics.
- Factual Accuracy: Achieved a Factual Score (FSc) of 0.969, a ~12.2% improvement over the strongest baseline.
- Category Accuracy: Achieved 0.702, a ~11.1% improvement, indicating superior alignment between hate type and refutation strategy.
- Safety: Reduced toxicity by ~11% (score 0.016) compared to baselines.
- Semantic Alignment: Led in BERTScore (0.886) and METEOR (0.247).
Efficiency: Despite using smaller agents, FIRE reduced peak VRAM usage by 75% (~4GB vs. ~16GB for 8B models) while maintaining comparable inference latency.
Ablation Studies: Removing the Web Search Tool caused a 14.1% drop in Factual Score. Removing the Memory Module caused a 14.5% drop in Category Accuracy. Removing the HSA (simulating a single-pass model) resulted in a 36.0% drop in Factual Score, confirming the necessity of multi-agent decomposition.
Human Evaluation: In a study with 30 expert annotators, FIRE was preferred over LLaMA-3.1-8B-Instruct (SFT), HiPPrO, and CoARL in 86–98% of instances across metrics like Argumentative Effectiveness and Contextual Relevance.

5. Significance and Claims

The paper claims that decomposing the underlying intent of hate speech is essential for generating safe, effective, and contextually precise counterspeech. The significance of FIRE lies in demonstrating that:

Specialization over Scale: Collaborative, reasoning-centric agents using compact models can outperform larger, monolithic LLMs when the task requires specific logical strategies and factual grounding.
Grounded Generation: Explicitly mapping hate categories to evidence retrieval and reasoning traces is critical for reducing hallucinations and improving the logical validity of counterspeech.
Scalability: The memory-efficient, non-finetuned approach offers a viable path for deploying counterspeech systems that are both factually accurate and computationally feasible.

The authors maintain a modest stance, acknowledging limitations such as the dataset's coverage (limited to 14 communities and 5 categories), the potential for web search failures, and the reliance on English-language data. They emphasize that while FIRE improves immediate response quality, long-term conversational impact and conflict escalation remain areas for future investigation.