🤖 machine learning

Extending Explainable Ensemble Trees (E2Tree) to regression contexts

This paper extends the Explainable Ensemble Trees (E2Tree) methodology, originally designed for classification, to regression contexts by leveraging dissimilarity measures to graphically represent both predictor-response relationships and predictor associations, thereby enhancing the transparency of random forest models.

Original authors: Massimo Aria, Agostino Gnasso, Carmela Iorio, Marjolein Fokkema

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Massimo Aria, Agostino Gnasso, Carmela Iorio, Marjolein Fokkema

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart team of detectives (a Random Forest) working together to solve a mystery. Each detective looks at a small piece of evidence and makes a guess. When they all vote, their combined guess is incredibly accurate. However, because there are so many detectives and they all talk over each other, it's impossible to tell exactly how they reached their final conclusion. The team is a "black box"—you get the answer, but you don't see the logic.

This paper introduces a new tool called E2Tree (Explainable Ensemble Trees) that acts like a translator. It takes the chaotic, complex decisions of the detective team and organizes them into a single, clear, easy-to-read flowchart.

Here is the breakdown of what the paper does, using simple analogies:

1. The Problem: The "Black Box"

In the world of machine learning, tools like Random Forests are great at predicting things (like how much a car will cost or how fast a plant will grow). But they are like a giant, tangled ball of yarn. You can pull the end and get a result, but you can't see the pattern inside.

The Goal: The authors want to untangle that yarn without losing the strength of the knot. They want to keep the high accuracy of the Random Forest but make it look like a simple decision tree that a human can understand.

2. The Solution: E2Tree

The authors previously created E2Tree for "classification" tasks (sorting things into categories, like "Cat" vs. "Dog"). In this paper, they teach E2Tree how to handle regression tasks.

Regression Analogy: Instead of sorting things into boxes, regression is like predicting a specific number, such as the exact speed of a car or the price of a house.
The Magic Trick: E2Tree doesn't just look at one variable at a time. It looks at how the variables "dance" together. It uses a special math trick called a dissimilarity matrix.
- Think of it like a seating chart: Imagine you have a room full of people. The Random Forest has already grouped similar people together at different tables. E2Tree looks at how often two specific people ended up at the same table across all the different groups the detectives formed. If they sat together often, they are "similar." If they rarely sat together, they are "different."

3. How It Works (The Recipe)

The paper describes a step-by-step process to build this new, clear tree:

Measure Similarity: It calculates how often pairs of data points (like two specific cars or two specific flowers) end up in the same "group" inside the Random Forest.
Check the "Fit": It checks if the groups make sense. In the world of predicting numbers (regression), it asks: "Are the numbers in this group close to each other?" If the numbers are all over the place, the group isn't very useful.
Build the Tree: It starts splitting the data, just like a normal tree, but it uses those similarity scores to decide where to cut.
Stop Rules: It knows when to stop splitting. It stops if the groups are already very similar or if splitting them further doesn't change the outcome (like checking if two groups of people have the same average height before trying to separate them further).

4. The Proof: Two Examples

The authors tested their new "translator" on two real-world datasets to prove it works:

The Flower Test (Iris Dataset):
- They tried to predict the length of a flower's petal based on its other measurements.
- Result: The E2Tree successfully recreated the complex logic of the Random Forest. It showed a clear path: "If the petal width is small AND the species is X, then the petal length is Y."
- Verification: They compared the "map" made by the Random Forest with the "map" made by E2Tree. They were 90% similar, proving the translator didn't lose the original meaning.
The Car Test (Auto MPG Dataset):
- They tried to predict how many miles per gallon (MPG) a car gets based on its weight, engine size, and year.
- Result: The E2Tree created a visual map showing exactly how factors like "heavy weight" or "big engine" lower fuel efficiency. It showed the "If-Then" rules clearly (e.g., "If the car is heavy AND old, it gets low MPG").
- Verification: Even with more complex data (more car features), the E2Tree map matched the Random Forest's logic about 75% of the time.

5. Why This Matters

The paper claims that E2Tree is a powerful tool because:

It's Transparent: It turns a confusing "black box" into a clear flowchart.
It's Accurate: It keeps the high prediction power of the original Random Forest.
It Shows Relationships: It doesn't just say "Weight is important." It shows how weight interacts with other factors (like engine size) to create the final result.

In short: The authors have built a bridge between the super-smart but confusing "Random Forest" and the human need for clear, logical explanations. They proved that this bridge works not just for sorting things into categories, but also for predicting specific numbers.

Technical Summary: Extending Explainable Ensemble Trees (E2Tree) to Regression Contexts

Problem Statement

Ensemble methods, particularly Random Forests (RF), have revolutionized supervised learning by offering high predictive accuracy through the aggregation of weak learners. However, these models function as "black boxes," lacking transparency regarding how they process input values to generate predictions. This opacity hinders user comprehension and trust, especially in critical domains like medicine and finance. While existing approaches to interpretability exist—ranging from inherently interpretable models (e.g., decision trees) to post-hoc explanation tools (e.g., variable importance, Shapley values)—they often struggle to balance model accuracy with the ability to elucidate complex, non-linear interactions between features.

The specific challenge addressed in this paper is the extension of the Explainable Ensemble Trees (E2Tree) methodology. Originally proposed by Aria et al. (2024) for classification tasks, E2Tree provides a graphical, tree-like representation of RF models that accounts for both predictor effects and inter-predictor associations. The authors note that while E2Tree successfully explained classification RFs, a corresponding methodology for regression contexts (continuous outcome models) was missing.

Methodology

The paper proposes an extension of the E2Tree framework to handle regression problems. The core objective is to derive a single, interpretable tree-like structure that faithfully represents the decision-making logic of a trained Random Forest regression model.

1. Dissimilarity Matrix Construction

The methodology begins by defining a dissimilarity matrix $D$ derived from the RF model. Let $H = \{H_1, \dots, H_B\}$ be a Random Forest of $B$ weak learners trained on a sample of size $n$ . For any two observations $u_i$ and $u_j$ , the dissimilarity $d_{ij}$ is defined as $1 - O_{ij}$ , where $O_{ij}$ is a weighted co-occurrence measure.

$O_{ij}$ quantifies how often a pair of observations falls into the same node across the sequence of trees, weighted by a local goodness-of-fit measure $W(t)_{ij|b}$ for node $t$ in tree $b$ :
$O_{ij} = \frac{\sum_{b=1}^{B} I(u_i \land u_j) \cdot W(t)_{ij|b}}{\max(\sum_{b} u_i W(t)_{i|b}; \sum_{b} u_j W(t)_{j|b})}$

2. Regression-Specific Goodness-of-Fit

A key contribution of this paper is the adaptation of the local goodness-of-fit measure $W(t)_{ij|b}$ for regression. In the classification context, this measure was based on purity; in regression, the authors propose using the Normalized Mean Square Error (NMSE):
$NMSE(t)_{ij|b} = \frac{MSE(t)_{ij|b}}{MSE_{max} \cdot p(t)}$
Where:

$MSE(t)_{ij|b}$ is the mean square error at node $t$ .
$p(t)$ is the proportion of observations in node $t$ .
$MSE_{max}$ is the theoretical maximum MSE, calculated based on the range of the response variable $Y$ within the node (following Jacobson, 1969): $MSE_{max} = \frac{(\max(Y|b) - \min(Y|b))^2}{9}$ .

The local fit measure is then defined as:
$W(t)_{ij|b} = \begin{cases} 0 & \text{if } 1 - NMSE(t)_{ij|b} \le 0 \\ 1 - NMSE(t)_{ij|b} & \text{otherwise} \end{cases}$

3. Tree Construction and Stopping Rules

The algorithm constructs the E2Tree by recursively splitting the data based on the dissimilarity matrix $D$ . To prevent overfitting and ensure the resulting tree is parsimonious, the authors introduce specific stopping criteria beyond the standard RF splits:

NMSE Threshold ( $\gamma$ ): A node becomes terminal if the calculated $NMSE(t)_{ij|b}$ falls below a specified threshold $\gamma$ (set to 0.05 in experiments).
Distributional Equivalence: For each potential split, a Mann-Whitney test is performed to verify if the distributions of predicted values in the child nodes are identical. If the distributions are not significantly different, the split is rejected, and the parent node is treated as terminal.

Key Contributions

Methodological Extension: The primary contribution is the formal extension of E2Tree from classification to regression. This involves redefining the local goodness-of-fit metric to utilize NMSE rather than classification purity.
Unified Representation: The method provides a single tree-like structure that captures the global logic of a Random Forest, including complex interactions between predictors, while maintaining the predictive accuracy of the original ensemble.
Dual Explanation Capability: The resulting E2Tree offers both global explanations (patterns and rules followed by the model) and local explanations (detailed logic for specific instances), bridging the gap between black-box performance and human interpretability.
Open-Source Implementation: The authors provide an R package (e2tree) and code availability, facilitating reproducibility and further application of the method.

Experimental Results

The authors validated the proposed method using two datasets from the UCI Machine Learning repository: the Iris dataset (adapted for regression on Petal Length) and the Auto MPG dataset (predicting fuel efficiency).

Iris Dataset: The E2Tree successfully reconstructed the structure of the RF's co-occurrence matrix. A hierarchical cluster analysis comparing the original RF matrix ( $O_{ij}$ ) and the E2Tree-estimated matrix ( $\hat{O}_{ij}$ ) yielded a Fowlkes-Mallows Index (FMI) of approximately 90%, indicating a high degree of similarity in the clustering structure.
Auto MPG Dataset: Despite the dataset's larger size and non-linear relationships, the E2Tree effectively visualized the decision paths. The comparison of heatmaps for $O_{ij}$ and $\hat{O}_{ij}$ resulted in an FMI of approximately 75%.
Interpretability: In both cases, the E2Tree provided clear "If-Then" pathways. For the Auto MPG data, the tree visualized negative effects of displacement, weight, and horsepower on MPG, and positive effects of model year and origin, offering insights that are not directly accessible from a standard fitted RF model.

Significance and Claims

The paper claims that E2Tree ensures machine learning models remain "tools of empowerment" by offering an understandable decision-making process without sacrificing the accuracy of the ensemble. The authors emphasize that their approach:

Interprets the ensemble process through a single, coherent tree structure.
Maintains the accuracy performance of the underlying Random Forest.
Provides a powerful tool for explaining interactions among predictors that determine final node predictions.

The authors conclude that this extension addresses both interpretability and explainability issues in both classification and regression contexts. They note that future work aims to generalize the approach to other ensemble-tree methodologies and develop further metrics to assess the fidelity of the E2Tree results against the original ensemble outputs. The paper does not claim to replace black-box models but rather to provide a transparent, post-hoc explanation layer that preserves the model's utility.