💻 computer science

Towards Explainability of SLMs by investigating Token Level Activation

This paper proposes the Activation Flow Network (AFN), a lightweight and model-agnostic framework that quantifies token-level importance in BERT by analyzing Layer-8 hidden-state activation strengths, demonstrating that semantically meaningful tokens exhibit significantly higher activation magnitudes than structural tokens and thereby offering an interpretable alternative to attention-centric methods.

Original authors: Sayantani Ghosh, Rajashik Datta, Amit Kumar Das, Amlan Chakrabarti

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sayantani Ghosh, Rajashik Datta, Amit Kumar Das, Amlan Chakrabarti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but secretive, robot librarian named BERT. This librarian has read almost everything ever written and can answer questions or understand feelings in sentences. But there's a catch: no one knows how it actually thinks. It's like a "black box"—you put a sentence in, and an answer comes out, but the gears inside are hidden.

For a long time, researchers tried to peek inside by looking at the librarian's "attention." They thought, "If the librarian is looking at a word, that word must be important!" But they discovered a trick: the librarian often stares intently at boring things like commas, periods, or words like "the" and "is," while ignoring the actual juicy words like "beautiful," "disaster," or "Canada." It was like watching a security camera that only zooms in on the door handle and ignores the person walking through the door.

The New Idea: Measuring "Energy" Instead of "Gaze"

The authors of this paper decided to try a different approach. Instead of asking, "Where is the librarian looking?" they asked, "How much energy is the librarian using to think about each word?"

They focused on a specific room in the librarian's brain called Layer 8. Think of the librarian's brain as a 12-story building:

Floors 1–5: The ground floor. Here, the librarian just notices basic things like capital letters or if a word is a noun or a verb.
Floors 10–12: The penthouse. This is where the final answer is decided.
Floor 8 (The Sweet Spot): This is the "consolidation zone." It's where the librarian stops looking at grammar and starts really understanding the meaning of the story.

The researchers built a tool called the Activation Flow Network (AFN). Instead of watching the librarian's eyes, they measured the "electricity" (mathematically called the L2 Norm) flowing through each word as it passed through Floor 8.

What They Found: The "High-Voltage" Words

When they measured this energy, they found a clear pattern:

The "High-Voltage" Words: Content-rich words like "prime," "minister," "Canada," "beautiful," or "enjoying" lit up like bright neon signs. They had the highest energy.
The "Low-Voltage" Words: Structural words like "the," "of," or punctuation marks like "?" were dimmer. They were there, but they weren't carrying the main weight of the meaning.

The Analogy: Imagine a team of movers packing a house. The "High-Voltage" words are the heavy sofas and pianos—they require the most effort to move. The "Low-Voltage" words are the empty boxes or the tape; they are necessary for the job, but they don't take up much space or energy. The researchers found that Floor 8 is where the heavy lifting happens.

The "Ripple Effect" Experiment

To prove this, the researchers played a game of "spot the difference." They gave the librarian two very similar sentences:

"Enjoying a beautiful day at the park!"
"Enjoying a beautiful walk at the beach!"

They watched how the "energy" changed when they swapped "day" for "walk" and "park" for "beach."

The Obvious: The words that changed ("day" and "park") had a massive jump in energy.
The Surprise: Even the words that didn't change (like "beautiful" or "at") had a significant energy shift!

The Analogy: It's like dropping a stone in a pond. When you change one word (the stone), the ripples (energy shifts) spread out and change the water around it, even the parts that didn't get hit directly. This showed that Floor 8 is deeply connected; changing one part of the story forces the whole sentence to re-think its meaning.

The "Bucket" Test

Finally, they tried to separate the "important" words from the "background" words using a simple rule. They drew a line (a threshold) and put all the words with high energy into a HIGH bucket and the rest into a LOW bucket.

They found that:

The HIGH bucket only had about 25% of the words (the important ones).
But, these few words were responsible for 76% of the total meaning changes.

The Analogy: It's like a concert where 25% of the musicians (the soloists) are playing the melody that everyone remembers, while the other 75% are playing the background rhythm. If you mute the soloists, the song loses its soul. If you mute the background, you can still hear the song, even if it feels a bit empty.

The Big Takeaway

This paper doesn't just say "BERT is smart." It gives us a new way to see how it thinks. By measuring the "energy" of words in the middle of the process (Layer 8), they turned the "black box" into a "glass box."

They showed that BERT isn't just staring at punctuation; it's actually focusing its mental energy on the words that carry the real meaning. This helps us understand that when the model processes a sentence, it's doing a lot of heavy lifting on the "content" words to make sense of the world, while the "structure" words just hold the frame together.

Technical Summary: Towards Explainability of SLMs by Investigating Token Level Activation

Problem Statement
Despite the revolutionary impact of Transformer-based models like BERT on Natural Language Understanding (NLU), their internal mechanisms remain largely opaque. Traditional interpretability methods relying on attention weights often fail to provide genuine semantic explanations, frequently highlighting structurally important but semantically weak tokens (e.g., punctuation, function words) rather than meaningful content. Furthermore, existing interpretability approaches often lack model-agnosticism, require expensive gradient computations, or fail to isolate specific intermediate processing dynamics, particularly within the middle layers (6–9) where syntactic and semantic information are believed to consolidate. There is a critical need for a lightweight, efficient method to quantify and rank the representational importance of individual tokens based on internal activation strengths rather than attention correlations.

Methodology
The authors propose a novel, model-agnostic framework termed the Activation Flow Network (AFN), which focuses exclusively on Layer 8 of the bert-base-uncased model. The methodology proceeds through the following stages:

Token-Level Activation Strength Quantification:
Instead of analyzing attention heads, the framework extracts the hidden state vectors ( $h_i$ ) for each token at Layer 8. The representational strength ( $S_i$ ) of a token is defined as the L2 Norm (Euclidean norm) of its 768-dimensional hidden state vector:
$S_i = \|h_i\|_2$
This scalar value serves as a direct measure of the token's activation magnitude and representational priority.
Threshold-Based Activation Bucket Formulation:
To distinguish between dominant semantic activations and structural background noise, the authors define an empirical activation boundary ( $\tau$ ) using the upper-quartile (75th percentile) of the activation distribution for a given sentence:
$\tau = Q_{0.75}(S)$
Tokens are partitioned into two groups:
- HIGH-activation: $S_i > \tau$ (Semantically salient tokens).
- LOW-activation: $S_i \leq \tau$ (Structurally supportive tokens).
  The framework calculates a contribution ratio ( $R_H$ ) to determine if HIGH-activation tokens dominate representational shifts.
Comparative Activation Shift Analysis:
To probe sensitivity to context and task instructions, the authors introduce the Token-level Activation Shift ( $\Delta_i$ ), calculated as the L2 Norm of the difference between hidden state vectors of corresponding tokens in two similar sentences or under different prompt conditions:
$\Delta_i = \|h_{A,i} - h_{B,i}\|_2$
This metric quantifies the geometric displacement in the vector space caused by contextual perturbations (e.g., changing "day" to "walk") or goal-oriented prompts (e.g., "Summarize" vs. "Translate").

Key Contributions

Layer 8 as a Semantic Consolidation Zone: The study identifies Layer 8 as a critical transition point where structural information gives way to semantic consolidation, balancing syntactic and semantic processing.
Activation Magnitude over Attention: The work demonstrates that hidden-state norms (L2) provide a more reliable signal for semantic importance than attention weights, which often mislead by focusing on punctuation.
Lightweight Interpretability: The AFN framework offers a computationally efficient, non-intrusive alternative to gradient-based attribution methods, requiring only a single forward pass to extract hidden states.
Quantification of Semantic Shifts: The introduction of the Activation Shift metric allows for the quantification of how specific word changes or task instructions alter the model's internal representation trajectory.

Results
Experimental analysis on a corpus of 732 sentiment-bearing sentences and specific case studies yielded the following observations:

Semantic Dominance: Content-bearing words (e.g., "prime," "minister," "Canada," "beautiful") consistently exhibited the highest activation strengths in Layer 8, ranking significantly higher than structural tokens like articles or punctuation.
Layer Comparison: Comparisons between Layer 8 and Layer 9 confirmed that Layer 8 is more sensitive to content-rich tokens, whereas Layer 9 shows relatively lower activation for these same tokens, suggesting a shift in focus from semantic content to syntactic structure in deeper layers.
Activation Bucketing: The upper-quartile threshold successfully separated tokens, with HIGH-activation tokens (content words) consistently occupying the top ranks. In perturbation experiments, HIGH-activation tokens contributed approximately 76% of the total semantic activation shift while comprising only ~25% of the token count.
Contextual Ripple Effect: Unchanged structural tokens (e.g., "at," "the") showed significant activation shifts when the context changed (e.g., "day at the park" vs. "walk at the beach"), indicating that Layer 8 dynamically re-encodes the entire phrase to maintain semantic consistency.
Prompt Sensitivity: Activation shifts were observed in the [CLS] token and other tokens when the input was prefixed with different task instructions ("Summarize" vs. "Translate"), demonstrating that high-level prompts steer the intermediate representational space.

Significance and Claims
The paper claims to transform BERT from a "black box" into a more transparent "glass box" by revealing that activation magnitudes concentrate around semantically informative tokens. The authors assert that their framework provides a computationally efficient and interpretable alternative to attention-centric analysis. By isolating Layer 8 and utilizing L2 norms, the research offers a direct way to identify which tokens carry the maximum representational strength during semantic processing. The work suggests that understanding these activation flows can help researchers better comprehend how BERT balances structural and semantic information and how task instructions influence the model's internal trajectory, without the need for complex gradient computations or model-specific probing classifiers.