Technical Summary: VARM-Bench

Problem Statement

Current benchmarks for Chinese abusive-language detection primarily evaluate final-label accuracy (e.g., harmful vs. non-harmful). However, a correct final label can mask significant errors in the underlying reasoning. A model might correctly classify a text as "harmful" while misidentifying the target, misinterpreting the author's stance (e.g., confusing a quote or opposition with endorsement), or assigning an inconsistent harm category. This is particularly critical in Chinese social media, where implicit references, homophonic substitutions, negation, and sarcasm frequently alter the meaning of offensive wording. Existing benchmarks offer isolated components (labels, spans, or taxonomies) but lack a unified representation that allows for the deterministic verification of the basis of a moderation decision. Free-form rationales are difficult to evaluate consistently, and structured field tuples often lack input-specific justification.

Methodology

VARM-Bench Construction

The authors introduce VARM-Bench (Verifiable Abusive Reasoning Moderation Benchmark), a dataset of 8,000 Chinese social media comments (5,600 training, 800 development, 1,600 test). The dataset includes 1,440 challenging non-harmful examples involving quotations, opposition, neutral mentions, and behavioral criticism.

• Data Pipeline: Data was collected from Bilibili, Zhihu, Baidu Tieba, and Hupu. It underwent normalization, anonymization, and filtering to remove duplicates and low-quality text.
• Annotation: A hybrid process was used where LLMs performed initial pre-annotation, followed by rigorous human review by graduate students and a doctoral student against a shared codebook.
• Structure: Each instance consists of an input text, a reference moderation record, and a field-anchored Chain-of-Thought (CoT) rationale.

Task Definition

The core task requires a model to generate a single natural-language CoT explanation containing six explicit anchors that correspond to specific moderation decisions:

1. Target: The referent of the abusive claim.
2. Target Type: Scope (single object, group, or unclear).
3. Target Explicitness: Whether the target is explicit or implicit.
4. Author Stance: The author's relation to the claim (e.g., attack/endorse, oppose, quote, neutral).
5. Harmfulness Label: Harmful vs. non-harmful.
6. Fine-grained Category: The basis of harm (e.g., gender, region, occupation).

A deterministic parser extracts these anchored values from the generated CoT to reconstruct a complete moderation record. This allows for traceability between the natural language explanation and the structured decision fields.

Evaluation Protocol

The evaluation moves beyond simple label accuracy to a multi-level assessment:

• Field Correctness: Macro-F1 scores for discrete fields and character-overlap F1 (T-F1) for free-text targets.
• Structural Validity: Parse success rate (all six anchors present and ordered correctly).
• Complete-Record Agreement (JREM): A metric requiring the parsed record to match the reference across all fields simultaneously.
• Hidden Record Errors (HER-C/L): Measures the rate of errors in target, stance, or category conditional on the final label or category being correct. This specifically quantifies how often a model gets the "right answer for the wrong reason."
• No LLM Judge: The protocol relies on deterministic parsing and exact matching against the reference, avoiding the subjectivity of LLM-based judges.

Key Contributions

1. Unified Benchmark: A Chinese abusive-speech benchmark where six moderation decisions are embedded in a single natural-language CoT and deterministically reconstructed into a complete record.
2. Multi-Level Evaluation Protocol: A framework separating decision quality, field correctness, record validity, and complete-record agreement, including audits of reference grounding and rationale sufficiency.
3. Diagnostic Analysis: An evaluation of open and closed models across zero-shot, taxonomy-guided, and CoT-supervised settings, revealing hidden record errors and field-level bottlenecks.

Experimental Results

The paper evaluates models including Qwen2.5-7B, Llama-3.1-8B, InternLM3-8B, GPT-5.5, Qwen3.7-Max, and DeepSeek-V4-Pro.

• Label vs. Reasoning: Strong performance on final-label accuracy (often >90% for top models) conceals substantial errors in complete moderation records. For example, in zero-shot settings, while label accuracy is high, the Joint Record Evaluation Metric (JREM) is significantly lower (e.g., ~30-50% for smaller models).
• Impact of Supervision: CoT-Supervised Fine-Tuning (CoT-SFT) significantly improves performance across all metrics, including JREM and field-level accuracy, compared to zero-shot or taxonomy-guided prompting.
• Hidden Errors: A significant portion of models that predict the correct final label still exhibit "hidden errors" in target identification or stance interpretation.
• Bottlenecks: The analysis identifies referent localization (identifying the correct target) as the primary bottleneck under frozen-reference scoring, more so than label classification.

Significance and Claims

The paper claims that VARM-Bench provides an auditable and reproducible foundation for studying verifiable reasoning in Chinese abusive-speech moderation. By shifting the evaluation focus from isolated labels to complete, traceable moderation records, the benchmark exposes the limitations of current models that rely on spurious lexical cues rather than contextual understanding. The authors assert that their deterministic protocol allows for the detection of "correct labels with incorrect reasoning," a failure mode that traditional benchmarks miss. The work emphasizes that reliable moderation requires verifying that the target, stance, and harm category are jointly correct, not just that the final output matches a binary label.