ABLE: Representing and Mapping LLMs via Attribution-Based Large-model Embedding

Technical Summary: ABLE (Attribution-Based Large-model Embedding)

Problem Statement
The rapid expansion of the Large Language Model (LLM) ecosystem has resulted in a heterogeneous collection of models with diverse architectures, scales, and training origins, often lacking clear documentation. This opacity complicates critical tasks such as provenance auditing, security analysis (e.g., tracing backdoor propagation), and intelligent model selection. Existing representation methods face significant limitations in this setting:

• Internal Feature Analysis: Methods analyzing parameters or activations are effective for homogeneous families but struggle with structural heterogeneity, requiring complex weight alignment strategies that do not scale well.
• External Output Probing: Black-box methods relying on outputs (e.g., log-likelihoods or generated text) scale well but often conflate models with similar surface behaviors but different internal computations. Furthermore, comparing outputs across models with different tokenizers and vocabularies is difficult.

Methodology
The authors propose ABLE (Attribution-Based Large-model Embedding), a training-free framework that constructs model representations in the "interpretability space" by aggregating gradient-based feature attributions. The pipeline consists of three stages:

1. Attribution Computation:

   • The method utilizes Gradient × Input (GI) attributions on a predefined probing dataset of multi-choice questions (MCQs).
   • For each model, the attribution of every input token to the log-probability of each candidate answer option is computed. This captures the model's input-sensitivity patterns rather than just its final prediction.
   • The result is a high-dimensional token-level attribution vector ($a_{token}$).

2. Cross-Tokenier Alignment:

   • To address the incompatibility of different tokenizers, ABLE employs a tokenizer-agnostic alignment process.
   • Token-to-Character Mapping: Token-level attributions are distributed uniformly across the character spans of the original text.
   • Character-to-Word Aggregation: Character-level attributions are summed based on whitespace delimiters to form word-level attributions.
   • This yields a unified word-level attribution vector ($a_{word}$) for all models, regardless of their underlying tokenizer.

3. Dimensionality Reduction:

   • The high-dimensional word-level vectors are projected into a compact low-dimensional embedding space ($K$ dimensions) using Johnson-Lindenstrauss (JL) random projection.
   • This preserves pairwise distances between models with high probability while enabling efficient downstream processing.

Theoretical Guarantees
The paper provides a stability analysis under standard regularity assumptions for differentiable Transformer-style models:

• Lipschitz Continuity: The authors prove that ABLE induces a Lipschitz-continuous map from model parameters to embeddings. This ensures that small perturbations in model parameters result in bounded changes in the embedding space, preventing chaotic behavior.
• Finite-Sample Convergence: The paper establishes concentration bounds showing that the empirical representation (computed from a finite set of samples) converges to the population-level representation as the sample size increases.

Experimental Results
The framework was validated on 239 open-source LLMs spanning parameter sizes from 70M to 70B and various families (e.g., Llama, Mistral, Qwen, Bloom). Key findings include:

• Relation Prediction: ABLE outperformed baselines (including output-based methods like PhyloLM and Log-Likelihood) in predicting whether two models are related (derived from the same source). It achieved the highest accuracy (86.7%) and precision (83.6%), demonstrating its ability to detect subtle derivation relationships that output-only methods miss.
• Model Routing: In a model routing task (predicting which model answers a question correctly), a router using frozen ABLE features achieved 67.6% accuracy, slightly surpassing the fully trained EmbedLLM baseline (66.5%). Crucially, ABLE offers a training-free advantage, allowing new models to be added to the routing pool without retraining the entire system.
• Benchmark Score Prediction: Using ridge regression, ABLE features predicted performance on standard benchmarks (ARC, MMLU, TruthfulQA, etc.) with Spearman correlations ranging from 0.81 to 0.89.
• Candidate Selection: ABLE successfully narrowed the search space for task-specific model selection (e.g., for math or reasoning tasks), achieving high NDCG@10 scores (up to 0.990) in selecting promising base models for adaptation.

Significance and Claims
The paper claims that ABLE offers a novel paradigm that bridges the gap between internal parameter analysis and external output probing. By leveraging the interpretability space, ABLE captures model-specific input-sensitivity patterns that are robust to architectural and tokenizer differences.

The authors emphasize that ABLE is not a replacement for existing methods but a complementary view that remains scalable for heterogeneous ecosystems. The framework provides a principled, training-free approach for:

1. Provenance Auditing: Identifying hidden relationships and potential copyright issues.
2. Model Routing: Efficiently selecting models for specific tasks without retraining.
3. Performance Estimation: Predicting benchmark scores to guide model selection.

The work concludes that while computing attributions incurs a one-time cost, the resulting reusable embeddings facilitate efficient downstream analysis, offering a stable and interpretable way to map the evolving landscape of LLMs.