💬 NLP

GGSS: Geodesic-Gated Spherical Steering for Inference-Time Debiasing of Generative Vision-Language Models

The paper introduces GGSS, a norm-preserving inference-time intervention that steers visual tokens along geodesic arcs on a unit hypersphere to effectively reduce demographic bias in generative vision-language models while maintaining their core visual-language capabilities.

Original authors: Yiqun Sun, Junyu Chen, Pengfei Wei, Lawrence B. Hsieh

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiqun Sun, Junyu Chen, Pengfei Wei, Lawrence B. Hsieh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Artificial intelligence systems that can see and speak are becoming common assistants in our daily lives, helping to sort resumes, answer questions, and interpret medical images. These systems, known as vision-language models, learn by studying millions of pictures and the text descriptions that go with them. However, because they learn from human-created data, they often absorb the same unfair stereotypes we hold about race, gender, and occupation. A computer might look at a photo of a person and, based solely on their perceived race or gender, guess a different salary or job title than it would for someone else, even if the visual details are identical. This is a serious problem for technology used in high-stakes decisions, yet fixing it is difficult. Traditionally, to remove these biases, developers had to retrain the entire massive computer model from scratch, a process that requires enormous computing power and time. Furthermore, many existing methods designed to fix bias were built for older types of AI that process an image as a single summary, rather than the modern systems that break an image down into thousands of tiny pieces to understand it.

A team of researchers has now developed a new way to correct these biases without retraining the model. They call their method GGSS, which stands for Geodesic-Gated Spherical Steering. Instead of rebuilding the AI, they insert a small, lightweight adjustment that works while the model is thinking. Imagine the AI's internal understanding of an image as a collection of directions pointing to different concepts. When the AI sees a face, some of these directions point toward the person's identity, while others point toward their job or background. The researchers found that the AI's bias toward race or gender is concentrated in a specific set of these directions, while the rest of the information remains neutral. Their goal was to gently nudge the biased directions away from the stereotype without disturbing the useful information about the person's actual appearance or the context of the image.

The researchers tested their method on four different modern vision-language models. First, they showed the models a series of carefully controlled images where only the perceived race or gender of the person changed, while everything else like their clothing, pose, and background stayed exactly the same. By watching how the model's internal signals shifted when only the race or gender changed, they mapped out the specific "bias path" the model takes. They then created a two-step process to fix this during normal use. The first step involves identifying which specific parts of the image trigger the bias. Not every part of an image is equally biased; for instance, a face might carry a strong signal of race, while the background or the person's shoes carries almost none. The new method uses a smart filter to focus its corrections only on the parts of the image that actually carry the bias, leaving the neutral parts alone.

The second step is how the correction happens. Older methods tried to simply subtract the bias, like erasing a line on a drawing. The researchers found that this "hard" subtraction often distorted the image's meaning, causing the AI to lose its ability to understand the picture correctly. Instead, they used a technique that moves the information along a curved path, similar to how a plane flies the shortest route between two cities by following the curve of the Earth rather than a straight line through the ground. This allows the AI to rotate its understanding away from the stereotype while keeping the overall shape and strength of its perception intact. By combining this curved movement with the smart filter that targets only the biased parts, the system can reduce unfairness without breaking the model's general intelligence.

The results of their tests were clear and significant. When they applied this method to the four different AI models, it reduced the bias in the models' answers by a large margin. In some cases, the unfair gap in how the models judged salaries or occupations dropped by more than 90 percent. Crucially, the models did not lose their ability to perform other tasks. The researchers checked the models' general knowledge and reasoning skills using a standard test of 1,500 questions covering science, math, and logic. The models that received the bias correction performed just as well as the original, uncorrected models, with their accuracy changing by less than one percentage point. This proved that it is possible to make these systems fairer without making them dumber.

The study also showed that this approach is more reliable than previous attempts. When they tried to use older, simpler methods that did not use the curved path or the smart filter, the models often became confused or failed to answer questions correctly. The researchers found that the curved path was especially important for the largest and most complex models, where simple corrections caused the system to fail completely. They also demonstrated that the method is flexible; by adjusting how strongly the correction is applied, users can choose to reduce bias significantly while still allowing the model to recognize demographic details if a specific task requires it, such as in medical documentation.

This work offers a practical tool for making artificial intelligence more equitable. It shows that we do not need to scrap and rebuild our most advanced AI systems to fix their prejudices. Instead, a targeted, careful adjustment made while the system is running can remove unfair stereotypes while preserving the model's ability to see and understand the world. The researchers have made their code available so that others can test and use this approach, providing a new path toward deploying AI systems that are both powerful and fair.

Technical Summary: GGSS – Geodesic-Gated Spherical Steering for Inference-Time Debiasing

Problem Statement

Generative Vision-Language Models (VLMs) are increasingly deployed in human-centered and high-stakes decision-making contexts, yet they frequently produce systematically biased outputs regarding demographic attributes such as race and gender, even when visual content is controlled. While existing inference-time debiasing methods (e.g., INLP, R-LACE, LEACE) have been effective for static embeddings or CLIP-like dual-encoders, they are ill-suited for modern generative VLMs. These models encode images as sequences of visual tokens fused with a decoder-only language model. Applying traditional debiasing techniques to this architecture presents three specific challenges:

Geometry Preservation: Euclidean subtraction of bias directions distorts token norms and directions, leading to sharp drops in general quality and "over-steering" failure modes, particularly as model scale increases.
Token-Level Bias Heterogeneity: Demographic signals are concentrated in a small fraction of visual tokens (e.g., face or clothing), while others encode pose or background. Uniform subspace projection over-corrects non-biased tokens, degrading task performance.
Single-Direction Rigidity: Many methods assume a binary protected attribute and a single bias direction, whereas attributes like race possess multi-class structures requiring low-dimensional subspace handling.

Methodology: GGSS Framework

The authors propose GGSS (Geodesic-Gated Spherical Steering), a norm-preserving intervention framework that operates in two stages: offline discovery and online inference.

1. Offline Discovery (Counterfactual Bias Subspace Estimation)
Using a frozen VLM and a set of counterfactual images (varying demographic attributes while holding other factors fixed), the method estimates the bias geometry:

Spherical Representatives: Visual activations are pooled and normalized to the unit hypersphere ( $S^{D-1}$ ).
Subspace Estimation: For each context, the Fréchet mean is computed, and tangent-space shifts between group means are calculated. Singular Value Decomposition (SVD) is applied to these shifts to identify a multi-dimensional bias subspace ( $V_{bias}$ ).
Gate Calibration: The magnitude of the projection of these representatives onto the bias subspace is measured to establish a distribution of bias norms. This yields statistics (median $\tilde{b}$ and standard deviation $\sigma_b$ ) used to calibrate the adaptive gate.

2. Online Inference (Token-Level Geodesic Steering)
During inference, GGSS intervenes at the vision-to-language projection layer:

Projection: Each visual token is mapped to the tangent space at a global reference point ( $\mu$ ). The component of the token lying within the learned bias subspace is projected out, leaving a "clean" component.
Adaptive Gating: A per-token gate ( $g_i$ ) is computed based on the token's bias magnitude relative to the discovery statistics. Tokens with low bias signals receive little to no steering, while those with strong demographic signals are steered more aggressively.
Geodesic Update: Instead of Euclidean subtraction, the method uses Spherical Linear Interpolation (Slerp) to rotate the token direction along a geodesic arc from its original position toward the debiased target direction. This ensures the token's norm is preserved exactly.
Norm Restoration: The original token norm is restored before passing the steered activations to the language decoder.

Key Contributions

Identification of Limitations: The paper demonstrates that hard, token-uniform, single-direction debiasing operators underperform on generative VLMs due to geometric distortion and lack of selectivity.
Norm-Preserving Geodesic Alternative: GGSS introduces a framework that replaces Euclidean steering with spherical geometry, preserving token norms and avoiding the quality degradation associated with over-steering.
Adaptive Gating: An inference-time gate is introduced that assigns per-token steering strengths based on the distribution of bias norms observed during discovery, eliminating the need for additional training.
Unified Benchmarking: The authors ported ten representative CLIP- and LM-era debiasers to the same generative VLM intervention layer, creating a unified comparison suite.

Experimental Results

The authors evaluated GGSS on four generative VLMs (Pixtral-12B, LLaVA-1.6-Vicuna-7B, LLaVA-1.6-Mistral-7B, and Qwen3-VL-4B) against ten adapted baselines and prompt-based mitigation.

Bias Reduction: GGSS achieved the lowest average bias across all four models. It produced significant bias reductions, with per-task improvements up to 96% (Nurse/Doctor classification), 84% (MCQ salary), and 61% (2AFC income). Statistical significance was observed on three of the four backbones.
Capability Preservation: Crucially, GGSS preserved general multimodal reasoning capabilities. On the MMStar benchmark, GGSS maintained accuracy within $\pm 0.6$ percentage points of the unsteered baseline. In contrast, several baselines (e.g., spherical BendVLM) caused significant drops in general capability or collapsed generation entirely.
Ablation Studies: Component ablations confirmed that the combination of spherical steering (Slerp) and the adaptive gate is responsible for the method's robustness and selectivity. Specifically, spherical steering was found to be decisive for robustness at larger model scales (12B), while the gate provided necessary selectivity.

Significance and Claims

The paper claims that for generative VLMs, adaptive geodesic steering is a superior primitive for inference-time bias mitigation compared to hard, uniform projection. The significance lies in demonstrating that bias can be reduced effectively without retraining the model or sacrificing its general reasoning capabilities.

The authors maintain a modest stance regarding deployment:

Not a Panacea: GGSS reduces measured bias but does not remove biased knowledge from model parameters or guarantee fairness under distribution shifts.
Trade-offs: The steering strength ( $\alpha$ ) controls a trade-off between bias reduction and the retention of demographic information. At high strengths, the method effectively removes the attribute, which may be undesirable for tasks requiring demographic recognition (e.g., clinical documentation).
Dual-Use: The mechanism can be used to amplify bias as easily as reduce it.
Scope: The evaluation is limited to perceived race and gender in image-conditioned settings; generalization to other attributes, languages, or domains is not established.

The work concludes that matched-gate ablations attribute robustness at scale to on-sphere steering and controllability to the gate, supporting adaptive geodesic steering as a practical, lightweight intervention for inference-time bias mitigation in frozen generative VLMs.