🤖 machine learning

Symbolic Mechanistic Data Attribution: Tracing Training Influence to Learned Behavioral Policies

This paper introduces Symbolic Mechanistic Data Attribution (SMDA), a framework that bridges mechanistic interpretability and data attribution by analytically decomposing how specific training examples shape high-level symbolic behavioral policies, thereby revealing systematic safety gaps and feature-level interference in large language models.

Original authors: Reza Habibi, Darian Lee, Magy Seif El-Nasr

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Reza Habibi, Darian Lee, Magy Seif El-Nasr

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) like a giant, complex orchestra. For a long time, we've treated this orchestra as a "black box": we know what music it plays (the answers it gives), but we have no idea which specific instruments are playing which notes, or how the conductor decided to start the song.

Mechanistic Interpretability is the field trying to figure out the orchestra's internal wiring. It's like finding out that a specific violinist (a "neuron") always plays a high note when the topic is "politics," or that a specific drumbeat (a "circuit") triggers when the model is about to lie.

But there's a missing piece: Data Attribution. We know how the orchestra plays, but we don't know which sheet music (training data) taught them to play that way. Did a specific song in the training set teach the violinist to play the high note? Or did a different song teach the drummer to stop?

This paper introduces a new tool called SMDA (Symbolic Mechanistic Data Attribution) to solve this puzzle, specifically looking at why AI models sometimes say "I can't do that" (refusal) when asked to do something harmful.

Here is how SMDA works, using simple analogies:

1. The "Symbolic Policy" (The Rulebook)

Instead of trying to trace influence back to a single, tiny neuron (which is like trying to find a single grain of sand on a beach), SMDA builds a simple rulebook (a "symbolic policy") that explains the model's behavior.

The Analogy: Imagine the AI's refusal behavior is a judge deciding whether to ban a book. The judge doesn't just look at the book; they look at a checklist of 75 specific "features" (like "Does it contain violence?" or "Is it about religion?").
The Innovation: SMDA uses a "Sparse Autoencoder" (SAE) to find these 75 features. It's like a super-advanced translator that turns the AI's complex, messy internal thoughts into clear, human-readable concepts (e.g., "Religious Stereotyping" or "Violent Scenes").
The Goal: SMDA fits a simple math equation (Ridge regression) to this checklist. This equation tells us how much weight the AI gives to each feature when deciding to say "No."

2. The "What-If" Experiment (Tracing the Influence)

Once the rulebook is built, SMDA asks: "If we had added one specific training example to the AI's history, how would that change the rulebook?"

The Analogy: Imagine you are teaching a student. You want to know if showing them a picture of a burning building (a "harmful" example) changed their rule for "When to call 911."
The Two Pathways: SMDA looks at two ways this training example changes the rulebook:
1. The "What I See" Pathway ( $\Delta X$ ): Did the training example change how the AI sees the world? (e.g., Did it make the AI more sensitive to the word "fire"?)
2. The "What I Decide" Pathway ( $\Delta Y$ ): Did the training example change the AI's final decision? (e.g., Did it make the AI more likely to say "I can't" regardless of what it sees?)

By separating these two, SMDA can tell you exactly why a specific training pair had an effect.

3. What They Found (The Results)

The researchers tested this on Llama-3.2-3B-Instruct, a popular AI model, using 200 training examples (some harmful, some harmless). Here is what they discovered:

The "Gap" in the Rulebook: They found that the AI's rulebook had a strange flaw. For categories like "religious stereotyping" or "making fun of protected groups," the AI could detect the harm (the feature lit up), but the rulebook gave those features a negative weight.
- Translation: The AI knew it was seeing something bad, but its internal rulebook was telling it to ignore it and comply. It's like a security guard who sees a thief but has a rulebook that says, "If you see a thief, let them pass." SMDA exposed this broken rule.
The "Cross-Feature Interference" (The Unintended Side Effects): This is a crucial finding. When they trained the AI on a prompt about "murdering a warthog" (a violent, harmful topic), they expected it to strengthen the "Violence" rule.
- The Surprise: That single training example also accidentally changed the rules for completely unrelated topics, like "Copyright requests" or "Religious questions."
- The Analogy: It's like trying to teach a student to be careful with knives, but in the process, you accidentally taught them to be afraid of spoons. The training data "spilled over" and messed up rules it wasn't supposed to touch.
Bad Training Data: They found specific training examples that were "miscalibrated." For instance, a prompt asking about "thick and long penises" (a sexual/harmful topic) was supposed to teach the AI to refuse sexual content. Instead, SMDA showed that this example mostly taught the AI to refuse general, harmless questions.
- Translation: The AI learned the wrong lesson. It thought, "If I see this weird question, I should just stop answering everything," rather than "I should only refuse sexual questions."

Why This Matters

Before SMDA, if you wanted to know why an AI refused a request, you either had to:

Guess (Black-box influence functions): "This training data probably caused it."
Manually inspect circuits: "Let's look at neuron #452..." (This is slow and hard to scale).

SMDA bridges the gap. It gives you a diagnostic tool that is:

Fine-grained: It explains why (e.g., "This training pair changed the 'Religion' rule, not the 'Violence' rule").
Scalable: It doesn't require a human to manually map every single neuron.

In short: SMDA is like a forensic accountant for AI training data. It doesn't just say "This data changed the AI"; it provides a detailed receipt showing exactly which internal rules were tweaked, which ones were broken, and which training examples caused the AI to learn the wrong lessons. This helps researchers fix the "rulebook" to make AI safer and more reliable.

Technical Summary: Symbolic Mechanistic Data Attribution (SMDA)

Problem Statement

Existing data attribution methods, including Mechanistic Data Attribution (MDA), are effective at tracing training examples to specific, pre-characterized low-level circuits (e.g., neurons, attention heads, or known SAE features). However, they struggle to explain how training data shapes high-level behavioral decisions where the underlying circuit is not known a priori. This gap limits the ability to diagnose complex safety phenomena such as harmful compliance, sycophancy, or refusal behaviors, which often emerge from the interaction of multiple components rather than a single identifiable unit. Furthermore, standard influence functions operate as black boxes, while manual circuit analysis is not scalable to high-level policies.

Methodology: Symbolic Mechanistic Data Attribution (SMDA)

The authors propose SMDA, a framework that attributes training influence to a learned symbolic policy over Sparse Autoencoder (SAE) features, rather than to a hand-specified circuit component. The method operates through the following steps:

Symbolic Policy Construction:
- The framework extracts SAE feature activations from a specific layer (Layer 10 of Llama-3.2-3B-Instruct) for a held-out evaluation set.
- It selects a subset of features ( $d=75$ ) that show substantial activation differences between harmful and harmless prompts.
- A closed-form Ridge regression is fitted to model the target behavior (refusal propensity) as a function of these SAE features. The output is a symbolic policy weight vector $W$ , where each entry represents the contribution of a specific feature to the refusal decision.
Influence Computation:
- For a given supervised fine-tuning (SFT) training pair $z_i$ , SMDA approximates the effect of a single gradient step on the symbolic policy weights $W$ .
- The shift in weights, $\Delta W_i$ $Δ W_{i}$ , is analytically decomposed into two pathways using a first-order Taylor expansion:
  - $\Delta X$ Pathway: Changes in the internal SAE feature activations caused by the gradient update.
  - $\Delta Y$ Pathway: Changes in the model's output refusal log-probabilities caused by the gradient update.
- The final influence score is the dot product $\langle \Delta W_i, W \rangle$ , indicating whether the training pair pushes the policy further in its current direction (positive) or against it (negative).
Experimental Setup:
- Model: Llama-3.2-3B-Instruct.
- SAE: A BatchTopK SAE trained on the residual stream of Layer 10, yielding 12,288 features.
- Data: 200 SFT training pairs (100 harmful refusal, 100 harmless compliance) and an evaluation set of 11,000 prompts split into various configurations (harmful-harmless, harmful-natural, harmful-balanced).

Key Contributions

Framework Innovation: Introduction of SMDA, which shifts the attribution target from predefined low-level units to a learned, interpretable symbolic policy over SAE features. This allows attribution in settings where the underlying circuit structure is unknown.
Analytical Decomposition: Derivation of a closed-form influence computation that separates training effects into feature-activation ( $\Delta X$ ) and output-probability ( $\Delta Y$ ) pathways, providing mechanistic insight into how training data alters behavior.
Empirical Analysis: Application of SMDA to the refusal behavior of Llama-3.2-3B-Instruct, analyzing 200 SFT pairs to reveal systematic gaps in safety behavior and cross-feature interference patterns.

Key Results

The analysis of the symbolic policy and training influences yielded three primary findings:

Systematic Gaps in Safety Behavior:
- The symbolic policy's coefficients revealed that the base model reliably detects harmful content (features activate) but fails to translate this detection into refusal decisions.
- Features associated with religious stereotyping, protected-group characterization, and argumentative framing carried large negative coefficients (predicting compliance). This indicates the model represents the harm but has not learned the corresponding refusal policy for these categories.
Qualitative Differences in Training Influence:
- The $\Delta X / \Delta Y$ decomposition showed that harmful and harmless training pairs exert qualitatively different influences.
- For certain features, harmful pairs primarily shifted the model's output probability ( $\Delta Y$ ), whereas harmless pairs distributed influence more evenly across both pathways. This suggests safety fine-tuning is not a uniform process.
Cross-Feature Interference:
- Individual training pairs frequently exhibited "catastrophic interference," where a pair targeting one semantic category (e.g., violence) significantly shifted the coefficients of semantically unrelated features (e.g., copyright requests).
- SMDA successfully identified "miscalibrated" training pairs where the dominant effect fell on unintended features (e.g., a refusal pair about sexual content primarily suppressing general informational requests), highlighting potential inefficiencies in safety data curation.

Significance and Claims

The paper claims that combining mechanistic interpretability with data attribution yields a diagnostic tool that is more fine-grained than black-box influence functions and more scalable than manual circuit analysis.

From Circuits to Policies: SMDA bridges the gap between understanding individual components and understanding decision-level behavior. It treats the symbolic policy itself as the object of attribution, revealing not just what features exist, but how the model weighs them.
Diagnostic Utility: The framework provides interpretable signals for safety engineering, such as identifying training pairs that cause unintended cross-feature interference or revealing specific categories where the model detects harm but fails to refuse.
Limitations and Scope: The authors are modest about the current scope, noting that SMDA attributes to the symbolic policy (a 75-feature Ridge regression with ~64–70% test accuracy) rather than the full model. They explicitly state that while the method identifies training pairs with consistent influence profiles, it does not perform causal validation via retraining experiments to verify that removing such pairs improves downstream safety. The work is presented as a necessary precondition for future intervention studies rather than a fully validated data curation tool.