Finding Multiple Interpretations in Datasets

Technical Summary: Finding Multiple Interpretations in Datasets

Problem Statement
In domains such as biology, researchers frequently analyze global characteristics of deep learning models (e.g., feature importance, representation layers) to derive insights about underlying phenomena. The standard workflow involves training a model to satisfactory performance, extracting a global property, and interpreting that property as a conclusion about the domain. However, this approach is philosophically limited by the potential for underspecification. In high-dimensional datasets, multiple distinct models may achieve similar predictive performance while relying on entirely different sets of features. Consequently, a conclusion derived from a single "best" model (e.g., "Gene X is critical") may be invalid if alternative models with equivalent performance exist where Gene X is unimportant. The paper argues that researchers must explore the model space to ensure that domain-specific conclusions are not merely artifacts of a specific model initialization or training trajectory.

Methodology
The authors propose a method to generate a set of models that are performance-equivalent but interpretation-dissimilar. The core technical contribution is an optimization framework that trains a new model $\theta_{n+1}$ to maximize its distance from a set of existing models $\{\theta_1, ..., \theta_n\}$ while maintaining performance within an acceptable tolerance.

The approach is formalized as an optimization problem:

1. Objective: Maximize a distance function $\delta(\theta_i, \theta_{n+1})$ between the new model and existing models.
2. Constraint: The primary loss $\lambda(\theta_{n+1})$ must remain below $\lambda(\theta_i) + \epsilon$.

To make this tractable in a deep learning setting, the authors convert the constrained problem into a differentiable objective function (Equation 2):
$$\text{minimize } \lambda(\theta_{n+1}) + \alpha \sum_{i=1}^{n} \frac{1}{1 + \delta(\theta_i, \theta_{n+1})}$$
Here, the term $\frac{1}{1 + \delta}$ acts as a regularization penalty. If the new model is similar to an existing one (small $\delta$), the penalty is high; if it is dissimilar (large $\delta$), the penalty is low.

Algorithm 1 (NextModel) implements this by initializing a model randomly and iteratively updating its parameters using the gradient of the combined loss (primary loss + distance penalty).

Distance Function Implementation
For the specific application of gene expression analysis, the distance $\delta$ is defined based on the ranking of feature importances.

• Algorithm 2 (RankLossDistance) computes the distance by comparing the top $n$ important genes of an existing model against a new model.
• It utilizes a differentiable ranking algorithm (softrank) on input layer weights.
• It identifies the top $n$ genes of the reference model and calculates an "offset" based on the importance of the $(n+1)$-th gene.
• The distance is computed as a weighted sum of the positive differences between the new model's gene ranks and this offset. This effectively penalizes the new model if it selects any gene that was highly ranked in the reference model's top 25.

Experimental Results
The method was evaluated on the METABRIC dataset (breast cancer gene expression data) using DeepType models.

• Setup: Ten control models were trained using standard hyperparameters. Subsequently, three "distant" models were generated sequentially using Algorithm 1 and the RankLossDistance function.
• Performance: The three new models achieved test and validation accuracies comparable to the control models (e.g., Model 1: 0.8220 test accuracy; Model 2: 0.7822; Model 3: 0.8056), though Model 3 showed a significant increase in sparsity loss.
• Interpretation Divergence:
  • Control Models: Across 10 control runs, 65 unique genes appeared in the top 25 importance lists, with 9 genes appearing in the top 25 for all 10 models.
  • Distant Models: The three new models produced a total of 75 unique genes in their respective top 25 lists (the maximum possible, as $3 \times 25 = 75$). Crucially, only 26 of these 75 genes overlapped with the top 25 genes found in any of the control models.
• Observation: The distant models achieved high performance while relying on gene sets that were largely disjoint from the control models and from each other.

Key Contributions and Significance
The paper claims the following contributions:

1. Demonstration of Underspecification: The study confirms that even on well-studied datasets like METABRIC, clusters of models exist that offer equivalent predictive power but rely on fundamentally different biological mechanisms (gene sets).
2. Method for Exploration: The proposed regularized training approach, utilizing a differentiable distance term, is shown to be an effective mechanism for surfacing these alternative interpretations without sacrificing overall accuracy.
3. Workflow Implication: The authors argue that relying on a single "best" model for biological interpretation is risky. Their method provides a scalable way to explore the model space, ensuring that domain insights are robust and not dependent on a single arbitrary solution.

Caveats and Future Directions
The authors note that as more models are added to the set, primary performance metrics may eventually degrade (as seen with Model 3's sparsity loss), suggesting that the space of "good" solutions is finite. They propose that the rate of this degradation could serve as a metric for the level of underspecification in a pipeline. Future work includes applying this to other biological datasets, investigating theoretical bounds on the error-distance tradeoff, and exploring dynamic selection of the regularization weight $\alpha$.