Technical Summary: RACC: Representation-Aware Coverage Criteria for LLM Safety Testing

1. Problem Statement

Large Language Models (LLMs) face significant safety risks from jailbreak attacks, which exploit model vulnerabilities to generate harmful content. Current safety testing relies heavily on static, open-source datasets (e.g., SafetyBench, SorryBench) and lacks systematic criteria to evaluate the quality and adequacy of test suites. While coverage criteria have proven effective for smaller Deep Neural Networks (DNNs) by monitoring neuron activations, they are ill-suited for LLMs for two primary reasons:

1. Scalability: The computational overhead of profiling billions of neurons across all layers is prohibitive for LLMs.
2. Objective Misalignment: Traditional neuron-level criteria treat all activations equally. In LLMs, the vast majority of neuron activations are redundant or irrelevant to safety. Consequently, neuron-level metrics often fail to distinguish between meaningful safety-relevant prompts and semantically redundant or benign inputs, introducing noise into coverage assessments.

There is a critical need for coverage criteria that are scalable to LLMs and specifically grounded in safety-relevant signals rather than raw activation statistics.

2. Methodology: RACC Framework

The authors propose RACC (Representation-Aware Coverage Criteria), a framework that shifts coverage measurement from individual neuron activations to safety representations—low-dimensional directions in the model's hidden state space that encode specific safety concepts.

2.1 Core Design Principles

RACC is built upon three design principles tailored for LLM safety testing:

• Synonym-insensitive: Semantically similar prompts (e.g., paraphrased jailbreaks) should not artificially inflate coverage metrics.
• Invalid-insensitive: Benign or non-adversarial inputs should not contribute to coverage, as the goal is to test the model's ability to reject harmful content.
• Jailbreak-sensitive: Metrics must prioritize prompts that successfully trigger harmful behaviors or activate safety-critical concepts.

2.2 Technical Workflow

The RACC framework operates in three stages:

1. Safety Representation Extraction:

   • A small calibration set of harmful prompts is curated to represent specific safety categories.
   • Hidden state activations from these prompts are collected from the model's middle layers.
   • Principal Component Analysis (PCA) is applied to these activations to identify the top-$n$ principal components. These components form the safety representations, representing the primary directions of safety concepts in the hidden space.

2. Concept Activation Calculation:

   • For any input prompt, the model's hidden state is projected onto the identified safety representations.
   • The magnitude of this projection serves as the concept activation score, quantifying the intensity of specific safety concepts triggered by the input.

3. Coverage Computation:
   RACC defines six sub-criteria divided into two dimensions:

   • Dimension I: Individual Concept Coverage
     • Safety Feature Coverage (SFC): Measures the proportion of safety features activated above a threshold by at least one prompt.
     • Top-K Feature Coverage (TKFC): Measures the proportion of safety features that appear in the top-$k$ most active features for at least one prompt.
     • Feature Intensity Coverage (FIC): Discretizes activation magnitudes into bins to measure the diversity of activation intensities across features.
   • Dimension II: Compositional Concept Coverage
     • Semantic Cluster Coverage (SCC): Uses K-Means clustering on the projected safety vectors to measure the diversity of semantic modes visited by the test suite.
     • Pairwise Concept Coverage (PCC): Measures the co-occurrence of pairs of safety features, capturing interactions between concepts.
     • Cluster Boundary Coverage (CBC): Identifies prompts residing in low-density regions between semantic clusters, targeting ambiguous safety boundaries where models are most vulnerable.

3. Key Contributions

• RACC Framework: The proposal of a scalable, representation-aware coverage criterion specifically designed for LLM safety, addressing the limitations of neuron-level metrics.
• Six Sub-Criteria: A novel set of metrics covering both individual concept activation and compositional interactions, grounded in safety representations rather than raw neurons.
• Empirical Validation: Comprehensive experiments demonstrating that RACC satisfies the three design principles (synonym-insensitivity, invalid-insensitivity, jailbreak-sensitivity) and outperforms existing neuron-level baselines.
• Practical Applications: Demonstration of RACC's utility in test suite prioritization (filtering redundant/invalid prompts) and attack prompt sampling (selecting diverse and successful jailbreaks).
• Open Source: Release of the implementation to facilitate further research.

4. Experimental Results

The authors evaluated RACC on three open-source LLMs (Vicuna-7B, Llama-2-7B, Qwen-2.5-7B) using benchmarks like SorryBench and ORBench-Toxic.

• Effectiveness vs. Neuron-Level Criteria:
  • RACC metrics showed high sensitivity to jailbreak attacks (e.g., $S_{JA}$), yielding significant coverage gains.
  • Crucially, RACC remained insensitive to invalid inputs (benign prompts) and redundant semantics (synonyms). In contrast, neuron-level baselines (e.g., NC, TKNC) showed large, misleading coverage increases when benign or synonymous prompts were added, failing to distinguish between effective safety tests and noise.
• Risk-Type Diversity: RACC metrics increased monotonically with the diversity of safety violation categories in the test suite, whereas neuron-level metrics remained stagnant or fluctuated randomly.
• Conceptual Alignment: When test suites were constructed with varying semantic overlap to the calibration set, RACC scores decreased strictly as overlap diminished, confirming the metrics measure specific safety concepts rather than generic anomalies.
• Applications:
  • Prioritization: RACC-guided selection front-loaded valid candidates significantly better than neuron-level baselines (e.g., AUC of 0.667 vs. 0.340 on SorryBench).
  • Sampling: RACC accelerated the discovery of successful jailbreaks and diverse attack categories compared to baselines.
• Generalization: RACC proved robust across different model sizes (scaling to 13B), calibration set sizes (effective even with small sets), extraction layers, and hyperparameter configurations. An adaptive parameter selection strategy (AdaRACC) was also proposed to eliminate manual tuning.

5. Significance and Claims

The paper claims that RACC provides a scalable and principled foundation for coverage-guided LLM safety testing. By grounding coverage in safety concept directions rather than raw neuron activations, RACC resolves the scalability bottleneck and the objective-alignment gap inherent in traditional DNN coverage criteria.

The authors assert that RACC is not merely a generic anomaly detector but a tool that specifically measures the activation of safety-relevant semantics. This allows for the rigorous evaluation, prioritization, and optimization of safety test suites, moving the field beyond reliance on static datasets toward dynamic, coverage-aware safety engineering. The work establishes that focusing on low-rank safety representations is a viable and superior approach for assessing the adequacy of LLM safety testing.