Instance-wise Linearization of Neural Network for Model Interpretation

In the modern world, artificial intelligence has become a quiet partner in daily life, guiding decisions from medical diagnoses to loan approvals. At the heart of these systems are neural networks, computer programs designed to learn patterns from data much like a human brain learns from experience. These networks are built from layers of simple processing units that pass information along, transforming raw inputs into final answers. For years, these systems have been incredibly effective, yet they have remained largely mysterious to the people who rely on them. When a network makes a decision, it is often impossible to see exactly which pieces of information mattered most or how the system combined them to reach that conclusion. This lack of transparency has become a significant hurdle, as governments and scientists increasingly demand that these automated systems explain their reasoning before they can be trusted with critical tasks.

A team of researchers has now proposed a way to peel back the curtain on this black box, not by guessing how the system thinks, but by mathematically simplifying its thought process for a single moment in time. Their work suggests that while a neural network is incredibly complex when viewed as a whole, its behavior for any single specific prediction is actually quite straightforward. By treating a single prediction as a unique event, the researchers found they could strip away the network's non-linear complexities and rewrite its entire decision-making process as a simple, direct calculation within a specific local region. This approach allows them to see exactly how every single piece of input data contributes to the final result, turning a mysterious algorithm into a transparent map of cause and effect.

The core of this discovery rests on a simple observation about how these networks operate. Neural networks use special components called activation units to decide whether to pass a signal forward or stop it. These units create a non-linear behavior, meaning the relationship between input and output is not a straight line, which is what makes the system so powerful but also so hard to understand. However, the researchers noted that for any single prediction, the network follows only one specific path through these activation units. Once that path is chosen, the complex, winding behavior of the network collapses into a linear process. In this specific instance, the network is no longer making a complicated, curved decision; it is simply multiplying the input by a set of numbers and adding a constant value.

To prove this, the team developed a method to rewrite the forward journey of data through a neural network. They took the standard layers used in computer vision, such as those that scan images for edges or shapes, and showed that each one could be converted into a standard matrix multiplication. This includes the convolution layers that scan images, the pooling layers that reduce image size, and even the skip connections that help deep networks learn better. By converting every single layer into this linear format, they could combine them all into one giant equation. The result is a single formula where the input is multiplied by a large matrix of weights and added to a bias term to produce the output. This formula acts as a precise representation of the network's decision for that specific image within its local linear region, revealing exactly how much each pixel contributed to the final score.

The researchers tested this method on several well-known image recognition models, including those trained to identify handwritten digits and complex objects like cars and animals. They discovered a surprising split in how these networks make decisions. For simpler models trained on basic digit images, the main calculation part of the formula carried almost all the weight of the decision, while the constant addition term was negligible. However, for the more complex models trained on difficult datasets, the constant term became the dominant force. In these advanced networks, the constant value alone could often determine the majority of the prediction, while the detailed calculation of the input features played a smaller role. This finding challenges the common assumption that the network's internal calculations are always the primary driver of its choices, suggesting that for many modern systems, a fixed bias is doing the heavy lifting.

This new way of looking at neural networks offers a powerful tool for understanding not just what a model predicts, but how it arrives at that prediction. Because the method breaks the decision down into a direct contribution from every input feature, it can generate a detailed map showing which parts of an image helped the network decide on a specific label and which parts worked against it. In experiments with handwritten digits, the researchers showed that this map correctly highlighted the strokes that defined a number, such as the loop of a seven or the vertical line of a one. Unlike other methods that might guess at importance, this approach calculates the contribution of every pixel based on the network's actual math for that specific image, provided the network utilizes piece-wise linear activation functions.

The utility of this technique extends beyond simple image classification into the realm of unsupervised learning, where networks are used to organize data without human labels. The researchers applied their method to a technique called parametric t-SNE, which uses a neural network to squeeze high-dimensional data down into a two-dimensional map for human viewing. Usually, it is difficult to understand why a network places two data points close together on this map. By applying their linearization method, the team could trace the journey of each data point through the network and see exactly which features caused it to land in a specific spot. They found that samples placed near each other often shared similar feature contributions, but sometimes, points that appeared close together were actually driven by very different internal reasons, a nuance that would have been invisible with traditional analysis tools.

Ultimately, this work provides a flexible framework for demystifying neural networks without needing to rebuild them from scratch. By recognizing that a single prediction is a linear event within a local region, the researchers have shown that the decision process can be reformulated into a clear, mathematical statement. This does not mean the networks are simple, but rather that their complexity can be paused and examined one step at a time. Whether the goal is to ensure an AI is making fair decisions or to help a scientist understand how a model sees the world, this approach offers a direct window into the machine's logic, replacing speculation with precise calculation.