🤖 machine learning

SwordBench: Evaluating Orthogonality of Steering Image Representations

This paper introduces SwordBench, a comprehensive benchmark for evaluating the orthogonality and effectiveness of steering image representations in vision models, revealing that while linear methods offer superior separability, they often fail to prevent collateral damage compared to sparse autoencoders.

Original authors: Vladimir Zaigrajew, Dawid Pludowski, Hubert Baniecki, Przemyslaw Biecek

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vladimir Zaigrajew, Dawid Pludowski, Hubert Baniecki, Przemyslaw Biecek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly stubborn, robot artist. This robot is great at drawing pictures, but sometimes it gets distracted by the background. For example, if you ask it to draw a "polar bear," it might only draw it correctly if the bear is on ice. If you put the bear on grass, the robot gets confused and thinks it's a different animal. This happens because the robot learned a "shortcut": it associates the animal with the background instead of looking at the animal itself.

To fix this, researchers want to "steer" the robot's brain while it's working, essentially telling it, "Ignore the grass, focus on the bear." They do this by finding a specific direction in the robot's brain that represents "grass" and pushing the robot's thoughts away from that direction.

The Problem: The "Swiss Army Knife" vs. The "Scalpel"
The paper introduces a new testing ground called SWORDBENCH. Think of this as a giant obstacle course designed to test different tools for steering these robots.

Previously, researchers mostly tested these steering tools on language models (chatbots), which are like trying to steer a ship made of fog—it's hard to know exactly where you are. But vision models (image AI) are more like a ship with a clear map. The authors realized that while we have many tools to steer these image robots, we didn't have a fair way to test if they actually work without breaking other things.

The Two Big Tests
The paper proposes two new ways to measure if a steering tool is good, using some creative analogies:

Cross-Concept Robustness (The "Steady Hand" Test):
Imagine you are trying to remove the "grass" direction from the robot's brain. But what if "grass" and "green" are tangled together? If you pull on "grass," do you accidentally pull on "green" too?
- The Analogy: It's like trying to untangle two knots of yarn. If you pull one knot to straighten it, does the other knot get tighter or messier? A good steering tool should be able to remove the "grass" idea without messing up the robot's ability to recognize "green" or other unrelated things.
Collateral Damage (The "Safe Zone" Test):
This is the most critical part. When you steer the robot to ignore "grass," you want to make sure you don't accidentally make it worse at recognizing bears that are already on grass (because maybe that's the only way it can see them).
- The Analogy: Imagine you are a surgeon removing a tumor (the bias). You want to make sure you don't accidentally cut the healthy heart (the robot's ability to do its job) in the process. If the robot gets worse at recognizing bears on grass after you "fix" it, that's Collateral Damage. The goal is to have zero damage to the healthy parts.

What They Found
The researchers tested many different "steering tools" (mathematical methods) on different robot brains (models like CLIP, DINOv2, and SigLIP). Here is what they discovered:

Simple is often better: You might think you need a super-complex, high-tech tool (like a "Sparse Autoencoder," which is like a fancy, multi-layered filter) to untangle these ideas. But the paper found that simple, old-school math tools (like a "Linear SVM," which is like a straight ruler) often work just as well, or even better, at finding the right direction.
The "Perfect" Tool Doesn't Exist: Some tools are great at finding the "grass" direction (high accuracy), but they are terrible at removing it without hurting the robot's other skills (high collateral damage). Conversely, some tools are very gentle but fail to actually remove the bias.
The "Hydra" Effect: In language models, if you try to fix one part of the brain, the model sometimes grows two new problems elsewhere (like a Hydra). The authors found that because image models have a clearer structure, they avoid this specific problem, making them easier to test.
The Best Tool Depends on the Job:
- If the robot's brain is very messy and complex (like real-world photos with many distractions), the "fancy" optimization tools work better.
- If the task is simpler (like a synthetic test with clear watermarks), the simple statistical tools work perfectly.

The Bottom Line
The paper argues that we can't just look at one number (like "how well did the robot find the bias?") to say a tool is good. We have to look at the whole picture: Did it fix the bias? Did it mess up other things? Did it stay steady when we tried to fix other biases?

SWORDBENCH is the new rulebook and obstacle course that forces researchers to answer all these questions at once, ensuring that when we try to make AI safer and fairer, we don't accidentally break the things that make it smart in the first place.

Technical Summary: SWORDBENCH

Problem Statement

While steering model representations at inference time has become a critical tool for AI safety and interpretability—particularly for correcting predictions without fine-tuning—existing evaluation protocols are heavily biased toward language models. These language-based benchmarks often rely on ambiguous, multi-token representations and imperfect "LLM-as-a-judge" metrics, which introduce noise and confounding factors. Furthermore, there is a lack of unified frameworks to evaluate steering in vision models, specifically regarding the removal of biases (concept erasure) and the assessment of second-order effects, such as whether steering inadvertently degrades performance on unrelated tasks or inputs.

Methodology

The authors introduce SWORDBENCH, a comprehensive benchmark designed to evaluate the orthogonality, robustness, and potential collateral damage of steering image representations across multiple vision backbones (CLIP, SigLIP, DINOv2) and concept removal tasks.

Benchmark Design

SWORDBENCH shifts the evaluation domain from the noisy, token-distributed setting of language models to the controlled, global representation space of vision transformers. The framework utilizes:

Datasets: Four real-world bias-removal datasets (CelebA, ISIC, Waterbirds, Counteranimal) and two synthetic control datasets (ImageNet-W with watermarks, ImageNet-C with corruptions) to provide ground-truth counterfactuals.
CAV Extraction Methods: A diverse suite of 16 methods categorized into:
- Standard Linear: Difference-in-Means, SVM, Logistic Regression, FastCAV, PatCAV, PCA, LAT, AurA.
- SAE-based: Methods applying statistical aggregations or specific algorithms (SAS, S&PTopK, SAE-AurA) on sparse autoencoder latent spaces.
- Nonlinear Probes: SVM-RBF and TabPFN to establish upper bounds.
Steering Intervention: The benchmark employs simple orthogonalization ( $\bar{h} = h - (v_c^T h)v_c$ ) to remove concept directions, avoiding the need for tuning scaling parameters ( $\alpha$ ) and ensuring fair comparison.

Novel Evaluation Metrics

Beyond standard detection accuracy (AUC, F1), SWORDBENCH introduces two critical metrics to uncover second-order effects of orthogonalization:

Cross-Concept Robustness (CCR): Measures the stability of a target concept's detection performance after the input is orthogonalized against alternative concepts. High CCR indicates the concept vector is functionally independent of others.
Collateral Damage (CD): Quantifies the drop in downstream task performance for inputs that lack the target concept. Ideally, steering should not alter performance on concept-absent samples; a non-zero CD indicates the CAV is entangled with other task-relevant features.
Steering Disparity (SD): For synthetic counterfactuals, this measures whether removing a concept from a biased sample restores the prediction to match its clean counterpart.

Key Results

The authors present five primary findings derived from extensive experiments:

Linear vs. Nonlinear Performance: Linear methods, specifically Support Vector Machines (SVM) and Logistic Regression (LR), consistently achieve concept detection accuracy (AUC) and downstream F1 scores comparable to complex nonlinear probes (e.g., TabPFN, SVM-RBF). Statistical tests confirm no significant difference between these linear approaches and their nonlinear counterparts on real-world datasets.
Trade-offs in Methodology:
- Optimization-based methods (e.g., SVM, LR) excel at disentanglement (low Maximum Similarity) and robustness in complex, real-world data (CelebA, ISIC) but risk overfitting in simpler regimes.
- Statistical methods (e.g., DiffMean, FastCAV) often capture global directions effectively in cleaner datasets (ImageNet-W/C) but may exhibit lower orthogonality in entangled scenarios.
- SAE-based methods generally require rigorous filtering (e.g., S&PTopK) to recover utility; raw SAE representations often degrade performance.
Imperfect Steering: While many methods achieve zero collateral damage, achieving "perfect steering" (zero Steering Disparity) is rare. Notably, Linear Artificial Tomography (LAT) and PCA achieved optimal steering disparity on synthetic tasks (ImageNet-W/C), whereas standard optimization methods often failed to achieve perfect steering despite high separability.
Metric Insufficiency: Standard classification metrics (AUC, F1) are insufficient proxies for safety and fidelity. High detection accuracy does not correlate with low collateral damage or high cross-concept robustness. Vector metrics (like MS) do not predict steering outcomes, necessitating the new evaluation protocols.
Sample Efficiency: Statistical methods converge rapidly (often with $N < 100$ samples), whereas optimization-based methods continue to improve with larger sample sizes ( $N > 500$ ) on complex datasets.

Significance and Claims

The paper claims that SWORDBENCH provides a necessary, unified framework for evaluating steering in vision models, moving beyond the limitations of language-centric benchmarks. Its primary contribution is the demonstration that steering is not merely a matter of finding a separating hyperplane; effective steering requires assessing functional independence (CCR) and safety (CD).

The authors modestly claim that their work:

Unifies the terminology and evaluation of steering and bias removal across vision and language domains.
Reveals that standard linear baselines are often sufficient for concept detection, challenging the necessity of complex nonlinear probes for this specific task.
Highlights that "perfect" steering is difficult to achieve and that high separability does not guarantee the absence of collateral damage.
Serves as a living benchmark to discourage selective reporting and encourage the development of methods that minimize unintended side effects in high-stakes applications like medical diagnosis and demographic fairness.

The paper does not claim to solve all steering challenges but rather establishes a rigorous standard for measuring the trade-offs inherent in current methods, emphasizing that safety in steering requires metrics beyond simple accuracy.