Technical Summary: PermDoRA - Understanding Adapter Interference in Language Models

Problem Statement

The deployment of Large Language Models (LLMs) in enterprise and scientific settings often requires modular access control, where users are authorized to access specific subsets of domain-specific knowledge without retraining the entire model. While existing approaches handle access control at the prompt, routing, or retrieval levels, they lack modularity at the level of learned representations.

A central challenge in parameter-level access control is adapter interference: when multiple domain-specific adapters are composed at inference time, performance often degrades. A prevailing hypothesis suggests this interference stems from overlap in linear parameter updates. Consequently, it is hypothesized that enforcing orthogonality or directional independence in the parameter space (via geometry-aware merging) should mitigate this interference and improve multi-domain performance compared to standard Euclidean averaging.

Methodology: DoRA-RBAC

The authors propose DoRA-RBAC, a hierarchical adapter composition framework designed to test the geometric interference hypothesis.

1. Framework Architecture

• Base Model: A pretrained backbone (LLaMA-3.1-8B or Mistral-7B) is frozen.
• Shared Baseline: A shared baseline adapter is trained and frozen.
• Manager Adapters: For each domain $d$, a manager adapter stores an incremental update to the baseline.
• DoRA Decomposition: Unlike standard LoRA ($\Delta W = BA$), DoRA decomposes updates into a unit-direction vector $\hat{v}_i$ and a scalar magnitude $m_i$:
  $$\Delta W_{DoRA}^i = m_i \cdot \hat{v}_i$$
  where $\hat{v}_i = \frac{\Delta W_i}{\|\Delta W_i\|_F}$.
• Composition: At inference, a user's authorized domains are composed into a single flattened adapter.

2. Composition Strategies

The study compares two merging strategies for authorized domains $D$:

1. Euclidean Composition: The effective weight updates are averaged in Euclidean space, followed by truncated Singular Value Decomposition (SVD) to refactor the result into a low-rank adapter.
   $$\Delta W_D^{Euc} = \text{SVD}_r \left( \frac{1}{|D|} \sum_{i \in D} \Delta W_i \right)$$
2. Riemannian (Geometry-Aware) Composition: Leveraging DoRA's separation of direction and magnitude, this method treats normalized directions as points on a unit hypersphere. It computes the Fréchet mean via normalized directional averaging and aggregates magnitudes separately:
   $$\hat{v}_D^{Riem} = \frac{\sum_{i \in D} \hat{v}_i}{\|\sum_{i \in D} \hat{v}_i\|}, \quad m_D^{Riem} = \sum_{i \in D} m_i$$
   The final weight is reconstructed via SVD.

3. Evaluation Metrics

• Utility: Measured via Utility Gap Index (UGI), the accuracy gain of a domain-specific manager over the baseline.
• Interference/Security: Measured via Domain Distinguishability Index (DDI), an AUC score quantifying how well a model can be distinguished as "in-domain" vs. "out-of-domain" using membership inference attacks (e.g., Min-K++).
• Geometric Diagnostics: Cosine similarity between adapter updates and symmetric orthogonality losses.

Key Results

1. Single-Domain Performance

On single-domain tasks across GPQA, PubMedQA, SimpleQA, and WMDP, DoRA-RBAC performs comparably to standard LoRA. There is no significant performance difference between the two methods in single-domain settings, confirming that the modular decomposition does not inherently degrade utility.

2. Multi-Domain Composition (The Core Finding)

The primary empirical finding is that geometry-aware merging provides no consistent advantage over standard Euclidean averaging in multi-domain settings.

• Performance: On SimpleQA (2-domain and 3-domain combinations), Euclidean and Riemannian merging produced virtually identical accuracy scores (e.g., Llama 2-domain: 0.00532 for both; Mistral 2-domain: ~0.0123 for both).
• Interference: The hypothesis that enforcing orthogonality reduces interference was not supported. The "gain" from merging is attributed to the properties of modularity itself rather than the geometric structure of the composition.

3. Geometric Diagnostics

• Angular Alignment: The study found that cosine similarity between adapter updates predicts when geometry-aware merging collapses to Euclidean behavior. As updates become more collinear (high cosine similarity), the performance gap between the two merging strategies shrinks to zero.
• Limitation of Geometry: However, angular alignment is a weak predictor of overall composition performance. Even when adapters are orthogonal, interference persists. This suggests that interference is not governed primarily by linear parameter-space geometry.

4. Security and Privacy

• Utility vs. Privacy: While domain-specific managers improve target-domain utility (average UGI +0.1178 on SimpleQA), they do not provide strong privacy guarantees.
• DDI Results: The Domain Distinguishability Index (DDI) AUC remained around 0.65, significantly above chance (0.5). This indicates that modular composition does not fully isolate domain behaviors; models retain distinguishable "behavioral fingerprints" of their constituent domains.

Conclusions and Significance

The paper concludes that adapter interference is not primarily driven by linear overlap in parameter space, but rather by interactions in shared nonlinear representations and downstream activations.

• Refutation of Geometric Hypothesis: The assumption that enforcing orthogonality or using Riemannian merging would resolve interference is incorrect. When adapter update directions are nearly collinear (as observed in practice), geometry-aware merging effectively reduces to ordinary arithmetic mean, offering no advantage.
• Implications for Access Control: DoRA-RBAC is best viewed as a modular composition and serving framework rather than a mechanism for strict isolation. While it enables efficient multi-domain inference, it does not guarantee privacy. Formal privacy defenses (e.g., differentially private training) remain necessary for high-assurance deployments.
• Future Direction: The authors suggest that future work should focus on representation-level objectives or activation-space constraints rather than parameter-space geometry to achieve reliable compositionality.

The study serves as a cautionary note against over-relying on parameter-space geometric properties to solve the complex problem of adapter interference in large language models.