📊 statistics

A Composite Activation Function for Learning Stable Binary Representations

This paper proposes the Heavy Tailed Activation Function (HTAF), a smooth composite approximation of the Heaviside function that enables stable gradient-based training for binary and spiking neural networks, and demonstrates its effectiveness in creating interpretable Implicit Concept Bottleneck Models with discrete feature representations.

Original authors: Seokhun Park, Choeun Kim, Kwanho Lee, Sehyun Park, Insung Kong, Yongdai Kim

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seokhun Park, Choeun Kim, Kwanho Lee, Sehyun Park, Insung Kong, Yongdai Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "On/Off" Switch That Breaks the Brain

Imagine a neural network (a type of AI) as a giant team of workers passing notes to each other to solve a puzzle. To make decisions, these workers use "activation functions," which act like light switches.

Standard AI: Uses dimmer switches. They can be slightly on, half-on, or fully on. This makes it easy for the team to learn because if a worker makes a mistake, they can gently nudge the switch up or down to fix it.
The Goal (Binary AI): The researchers want to use On/Off switches (like the Heaviside function). This is great because it saves massive amounts of memory and energy (like a lightbulb that is either fully bright or totally dark). It also makes the AI's thinking easier to understand.
The Catch: You cannot gently nudge an On/Off switch. If it's off, it stays off. If you try to calculate how to fix a mistake (using gradients), the math breaks because the switch doesn't move smoothly. It's like trying to steer a car that only has a "Go" and "Stop" button with no steering wheel.

The Solution: The "Heavy-Tailed" Switch (HTAF)

The authors created a new type of switch called HTAF (Heavy-Tailed Activation Function). Think of it as a smart, hybrid switch that tricks the training process.

The "Sweet Spot" (Near Zero): When the input is small (near zero), this switch behaves like a steep, aggressive ramp. It allows the "learning signal" (the gradient) to flow through strongly. This ensures the AI can learn quickly and fix mistakes, just like a standard dimmer switch.
The "Tail" (Far from Zero): When the input gets very large, most standard switches (like the Sigmoid function) flatten out completely, causing the learning signal to vanish (the "gradient vanishing problem"). HTAF is different. It has a heavy tail, meaning it keeps the learning signal flowing even when the numbers get huge. It's like a long, gentle slope that never quite flattens out, ensuring the AI never gets "stuck" or loses its way.

The Analogy: Imagine a slide at a playground.

Standard switches are like a slide that gets so flat at the bottom that you get stuck and can't slide down anymore (gradient vanishing).
HTAF is a slide that is steep at the top (so you start fast) but has a long, gentle curve at the bottom that keeps you moving all the way to the end, no matter how high you started.

How They Used It: Two Main Tricks

The paper shows two ways they used this new switch:

1. Training "Binary" Networks Without Breaking Them

They used HTAF to train networks that want to be binary (Spiking Neural Networks, Binary Neural Networks, and Deep Heaviside Networks).

During Training: They use the smooth HTAF switch. The AI learns normally because the math works.
During Testing (Inference): Once the learning is done, they swap the HTAF switch out for the strict On/Off switch.
The Result: The AI ends up with a binary brain (saving memory and energy) but learned just as well as a standard AI. In their tests, this method worked better than previous tricks (like "Straight-Through Estimators") which often produced messy or biased results.

2. "Implicit Concept Bottleneck Models" (ICBMs)

This is the most creative application. Usually, to make an AI explainable (so humans can understand why it made a decision), we have to manually label what the AI is looking for (e.g., "Does the bird have a red beak?"). This is expensive and slow.

The authors built a model called ICBM that finds these concepts on its own:

The Process: They force the AI's internal "thoughts" (features) to become binary (0 or 1) using HTAF.
The Magic: Because the AI is forced to make binary decisions, it naturally groups images into "has this feature" or "doesn't have this feature."
The Explanation: Once the AI learns these binary groups, they ask a Large Language Model (like a smart chatbot) to look at the groups and give them a name.
- Example: The AI might find a group of birds that are "on the ground" vs. "in the water." The chatbot names this concept "Ecomorphological Niche."
The Benefit: They get a model that is as accurate as a standard "black box" AI, but it also gives you a list of human-readable reasons for its decisions, without needing humans to label the concepts beforehand.

The Bottom Line

The paper introduces a new mathematical tool (HTAF) that acts as a bridge. It allows researchers to train AI models using standard, smooth math, but then snap them into a rigid, binary state for the final product.

Why it matters: It makes AI faster, cheaper to run, and easier to understand, without sacrificing how well it performs.
The Claim: They proved mathematically that this switch prevents the "learning signal" from dying out, and they showed through experiments that it works on images (like recognizing birds or cars) and even on large language models.

Note: The paper focuses entirely on image datasets (like CIFAR, CUB-200) and text datasets (like Winogrande). It does not claim to work for medical diagnosis or clinical uses, nor does it predict future applications beyond the specific experiments they ran.

Technical Summary: A Composite Activation Function for Learning Stable Binary Representations

Problem Statement

Activation functions are fundamental to neural network performance, influencing prediction accuracy, robustness, and interpretability. While the Heaviside activation function ( $x \mapsto \mathbb{I}(x \geq 0)$ ) offers significant advantages in computational efficiency, memory usage, and interpretability—making it ideal for Spiking Neural Networks (SNNs), Binary Neural Networks (BNNs), and Deep Heaviside Neural Networks (DHNs)—it is inherently non-differentiable. This non-differentiability prevents the use of standard gradient-based optimization.

Existing solutions face critical limitations:

Surrogate Gradients (e.g., Straight-Through Estimator - STE): These methods approximate gradients during backpropagation but introduce a mismatch between forward and backward passes, often leading to biased optimization and suboptimal performance.
Smooth Surrogates (e.g., Sigmoid): Replacing Heaviside with smooth functions like sigmoid enables gradient descent but suffers from severe gradient vanishing, particularly when the function is steepened to approximate the binary step. This leads to unstable training dynamics.

The core challenge is to design an activation function that closely approximates the Heaviside function (enabling binary representations) while maintaining a stable, non-vanishing gradient flow suitable for deep network training.

Methodology: Heavy-Tailed Activation Function (HTAF)

The authors propose the Heavy-Tailed Activation Function (HTAF), a novel smooth composite function defined as:
$\text{HTAF}(x|\alpha_0, \alpha_1) := \sigma(\alpha_1 \psi(\alpha_0 x))$
where $\sigma$ is the sigmoid function, $\psi$ is the hyperbolic tangent function, and $\alpha_0, \alpha_1 \in \mathbb{R}^+$ are hyperparameters.

Theoretical Properties

The design of HTAF addresses the trade-off between binary approximation and gradient stability through two key mechanisms:

Binary Approximation: Theoretical analysis (Proposition 3.1) demonstrates that with appropriate choices of $\alpha_0$ and $\alpha_1$ , HTAF uniformly approximates the Heaviside function outside an arbitrarily small neighborhood of the origin.
Heavy-Tailed Gradient Behavior:
- Near Zero: The gradient at $x=0$ is determined by the product $\alpha_0 \alpha_1$ (Theorem 3.2b), allowing for a large gradient magnitude to facilitate learning near the decision boundary.
- In the Tails: For large $|x|$ , the gradient decays as $C(\alpha_0, \alpha_1)\exp(-2\alpha_0|x|)$ (Theorem 3.2a). Crucially, by choosing a small $\alpha_0$ , the gradient decay in the tail regions is slowed significantly compared to standard scaled sigmoid functions, preventing premature gradient vanishing.
- Parameter Control: The authors show that $\alpha_0$ controls the tail decay rate, while $\alpha_1$ controls the slope at the origin. This decoupling allows the network to maintain a large gradient at zero while ensuring gradients do not vanish too quickly for large inputs.

Applications

The paper applies HTAF to two primary domains:

Training Neural Networks with Heaviside Activations: HTAF replaces the Heaviside function during training for SNNs, BNNs, and DHNs. This enables end-to-end optimization using standard gradient descent without surrogate gradients. At inference, HTAF is reverted to the exact Heaviside function, yielding discrete, memory-efficient activations.
Implicit Concept Bottleneck Models (ICBMs): The authors introduce ICBMs, an extension of Concept Bottleneck Models (CBMs). In ICBMs, the final layer of a feature extractor uses HTAF during training to induce discrete, near-binary latent representations (implicit concepts). At inference, the Heaviside function is applied to produce exact binary features. Unlike traditional CBMs, ICBMs do not require pre-defined concept annotations; instead, they learn concepts directly from data, which are subsequently interpreted by Large Language Models (LLMs).

Key Contributions

Novel Activation Function: Introduction of HTAF, a smooth approximation of the Heaviside function that preserves large gradients near zero while exhibiting slow gradient decay in tail regions.
Theoretical Justification: Proof that HTAF can approximate the Heaviside function with $O(\epsilon)$ error and theoretically demonstrate how parameter tuning controls gradient decay to enable stable deep learning.
Surrogate-Free Training: Demonstration that HTAF enables stable, end-to-end gradient-based training of SNNs, BNNs, and DHNs, outperforming existing methods like STE and Scaled Sigmoid (SS) in prediction accuracy and stability.
Interpretable Models: Proposal of ICBMs, which leverage HTAF to learn discrete, class-specific implicit concepts without human annotation, achieving competitive prediction performance while offering interpretability.

Experimental Results

The authors conducted extensive experiments across various architectures and datasets:

Heaviside Network Training:
- On CIFAR-10 and CIFAR-100, HTAF-trained SNNs achieved performance competitive with STE (91.0% vs 91.2% on CIFAR-10).
- For BNNs and DHNs, HTAF consistently outperformed both STE and Scaled Sigmoid. For instance, on CIFAR-100, HTAF-trained BNNs achieved 46.1% accuracy compared to 45.2% (STE) and 31.3% (SS). The authors attribute SS's poor performance to gradient vanishing.
Implicit Concept Bottleneck Models (ICBMs):
- Replacing the final activation of feature extractors (ResNet, WideResNet, ViT) with HTAF did not degrade prediction performance on CIFAR-10, CIFAR-100, CUB-200, and Tiny-ImageNet. In some cases (e.g., CIFAR-100 with WideResNet), performance improved.
- Class Specificity: Analysis showed that implicit concepts learned by ICBMs are highly class-specific, with activation values concentrating near 0 or 1 for specific classes, unlike standard models.
- Interpretability: Using an LLM to interpret the binary latent nodes, the authors successfully identified semantically meaningful concepts (e.g., "Ecomorphological Niche," "Bill Morphology") that aligned with visual characteristics of the data.
Large Language Models (LLMs):
- Experiments replacing activation functions in the last $k$ layers of Llama-2-7b and Qwen-3.5-9b with HTAF showed that HTAF provides a more stable alternative to STE, exhibiting a smaller performance drop as the number of replaced layers increases.

Significance and Claims

The paper claims that HTAF provides a principled solution to the long-standing difficulty of training neural networks with binary activations. By theoretically and empirically demonstrating that HTAF maintains gradient flow in tail regions while approximating the Heaviside function, the authors enable:

Stable Optimization: Training of deep binary networks without the bias introduced by surrogate gradients or the instability of steep sigmoid approximations.
Efficiency and Interpretability: The ability to deploy memory-efficient, discrete models (SNNs, BNNs) and interpretable models (ICBMs) without sacrificing prediction accuracy.
Data-Driven Concepts: A new paradigm for CBMs that learns concepts directly from data, bypassing the cost and unreliability of manual or AI-generated concept annotations.

The authors note that while HTAF alleviates gradient vanishing through parameter tuning, it does not eliminate it completely, and determining the optimal number of concepts in ICBMs remains an open direction for future research.