📈 economics

Integrating Machine-Learned ESG Ratings into Hierarchical Risk Parity: Evidence from Classification Benchmarking and Sustainable Portfolio Optimization

This study demonstrates that integrating a machine-learning-optimized Logistic Regression classifier for ESG ratings into the Hierarchical Risk Parity framework enables "soft" portfolio integration that significantly improves ESG scores while preserving risk-adjusted performance, in contrast to "hard" exclusionary screening which drastically reduces diversification and returns.

Original authors: Chaima Ben Hassine, Heni Boubaker

Published 2026-07-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Chaima Ben Hassine, Heni Boubaker

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive ship, the Portfolio, sailing through the choppy waters of the stock market. Your goal is to keep the ship moving fast (making money) while keeping it stable (avoiding huge crashes) and ensuring you're not sailing toward an iceberg made of bad environmental or social practices (ESG).

For a long time, captains had two main ways to handle this:

The "Speed-Only" Crew: They looked only at how fast the ship could go. They packed the deck with the fastest, most aggressive sailors. This made the ship zoom, but it was dangerously crowded, and if one sailor stumbled, the whole ship tipped over.
The "Purity-Only" Crew: They threw everyone off the ship who didn't have a perfect "Green Badge." This made the ship very clean and ethical, but it left so few sailors that the ship was tiny, slow, and couldn't handle a storm.

This paper introduces a new, clever captain who uses a Machine Learning tool to find a middle path. Here is how they did it, what they found, and what they definitely didn't find.

The Magic Tool: Sorting the Sailors

First, the researchers needed a way to sort 866 companies into three groups: Low, Medium, and High ESG ratings. They tried nine different "sorting algorithms" (think of them as different types of super-smart robots) to see which one could best predict a company's rating based on its financial data and its specific Environmental, Social, and Governance scores.

The Big Surprise:
Everyone expected the most complex, high-tech robots (like Gradient Boosting and Neural Networks) to win. After all, they are the "heavy lifters" of the AI world.

What the paper argues against: The idea that "more complex is always better."
The Reality: The simplest robot, Logistic Regression, won the race. It got it right 89.58% of the time.
The Trap: The complex robots (Gradient Boosting) looked perfect during practice tests, scoring 100% (a perfect 1.000 cross-validation F1-score). But when they faced the real test, they crashed, dropping to about 79% accuracy. The paper explains this as "memorizing the practice questions" instead of actually learning. They overfit the data. The simple robot, however, stayed steady and reliable, proving that sometimes a straight line is better than a tangled knot.

The New Navigation System: HRP

Once the robot sorted the companies, the captain needed a way to build the ship. They used a method called Hierarchical Risk Parity (HRP).

The Analogy: Imagine building a tower of blocks. Instead of stacking them randomly, you group similar blocks together (like all red blocks, all blue blocks) and balance the weight so the tower doesn't tip. HRP does this with money, grouping companies that move together and balancing the risk so no single group can knock the whole portfolio over.

The Four Ways to Mix in "Green"

The researchers tested four different ways to mix the ESG ratings into this balancing act:

The "Hard Filter" (Pre-screening): Only let the "High ESG" sailors on board.
- Result: The ship became incredibly green (ESG score jumped 86% higher), but it became tiny and fragile. The ship's speed (Sharpe ratio) dropped by 85%. It was too concentrated and risky.
The "Soft Tilt" (Adjusting Returns): Give a tiny bonus to the speed of the "High ESG" sailors, but don't kick anyone off.
- Result: The ship got a little greener (ESG scores rose 6–12%) but kept its speed and stability. The risk-adjusted performance stayed around 0.50–0.54.
The "Grouping Tweak" (Adjusting Distances): Tell the robot to group "High ESG" companies closer together when building the tower.
- Result: Similar to the soft tilt, it improved the green score without wrecking the ship's balance.
The "Hybrid" (Mixing Both): Do both the speed bonus and the grouping tweak.
- Result: This was the sweet spot. It improved the ESG score by 6–12% while keeping the ship's stability (Sharpe ratio) between 0.50 and 0.54.

The Verdict: The "Sweet Spot" Frontier

The paper maps out a "frontier" (a map of possibilities).

On one end, you have the Pure Speed ship: Fast, but crowded and risky.
On the other end, you have the Pure Green ship: Very clean, but slow and fragile.
In the middle: The Soft Integration ships. These ships show that you can make your portfolio significantly greener (by 6–12%) without sacrificing your safety or speed.

What the paper is sure about:

Logistic Regression is the best tool for sorting these companies in this specific setup, beating out complex AI models that overfit the data.
Soft integration (tweaking the rules slightly) is much better than Hard exclusion (kicking people off the ship) if you want to keep your portfolio safe and diversified.
The "Hard Filter" strategy creates a ship that is too small (only 6.51 effective assets) compared to the balanced ships (which have 28 to 48 effective assets).

What the paper suggests but doesn't prove for the whole world:

The results are based on a specific group of 48 large US companies over a specific time (2020–2023). This period included the crazy pandemic crash and the inflation shock. The authors suggest these findings might look different in other markets or times, but within this test, the "Soft Tilt" worked best.
They found that Environmental and Social scores were the most important clues for the robot, while Governance scores mattered less in this specific dataset.

The Takeaway for a Curious Teen

If you want to build a portfolio that cares about the planet but doesn't crash and burn, don't just ban the "bad" companies (that makes your ship too small). Instead, use a simple, reliable AI to sort the companies, and then gently nudge your investment strategy to favor the "good" ones while keeping the whole ship balanced. It's like adding a little extra fuel to the eco-friendly engines without removing the safety rails. The paper shows you can do this and still keep your ship sailing smoothly.

Technical Summary: Integrating Machine-Learned ESG Ratings into Hierarchical Risk Parity

Problem Statement
Sustainable portfolio optimization increasingly requires the rigorous integration of Environmental, Social, and Governance (ESG) considerations into risk-based allocation frameworks. However, existing literature offers limited guidance on structurally embedding ESG metrics within Hierarchical Risk Parity (HRP), a robust alternative to Markowitz's mean-variance framework that avoids covariance matrix inversion. Current approaches typically impose sustainability constraints exogenously (e.g., exclusionary screening) rather than incorporating them as intrinsic components of the risk allocation process. Furthermore, the comparative effectiveness of machine learning (ML) methods for predicting ESG ratings from observable firm characteristics remains underexplored, with many studies relying on single-model pipelines that lack systematic benchmarking, cross-validation stability analysis, and rigorous handling of class imbalance. This creates a risk of specification error and limits generalizability across market conditions.

Methodology
The study proposes a two-stage pipeline that integrates ML-driven ESG classification directly into the HRP construction process.

Data and Feature Engineering:
- Dataset: The study utilizes ESG data for 866 S&P 500 companies from the S&P 500 ESG Sustainability Reports dataset. The target variable is constructed by binning composite ESG scores into three quantile-based categories: Low, Medium, and High.
- Features: A comprehensive set of 25 features is engineered, comprising 10 statistical descriptors of return series, 12 technical indicators (momentum, volatility, drawdown), and the three raw ESG pillar scores (E, S, G).
- Class Imbalance: To address moderate class imbalance, Synthetic Minority Oversampling Technique (SMOTE) is applied strictly within the training partitions of cross-validation folds to prevent data leakage.
Machine Learning Benchmarking:
- Algorithms: Nine classification algorithms spanning linear, ensemble, kernel-based, probabilistic, instance-based, and neural paradigms are benchmarked. These include Logistic Regression (LR), Gaussian Naive Bayes, Random Forest, Extra Trees, Gradient Boosting, LightGBM, SVM (RBF), K-Nearest Neighbors (KNN), and Multi-Layer Perceptron (MLP).
- Validation Protocol: A rigorous leakage-averse protocol is employed, featuring an 80/20 stratified train-test split, stratified 5-fold cross-validation, and nested grid search for hyperparameter tuning.
- Evaluation Metrics: Performance is assessed via test accuracy, macro F1-score, Matthews Correlation Coefficient (MCC), Cohen's kappa, AUC-ROC, AUC-PR, and cross-validation stability (standard deviation of F1 across folds).
ESG-HRP Integration Strategies:
The study designs and tests four integration mechanisms to embed sustainability into HRP:
- Strategy 1 (Hard Pre-screening): Restricts the investment universe to assets classified as "High ESG."
- Strategy 2 (ESG-Adjusted Returns): Modifies expected returns via a utility tilt: $\mu^{ESG}_i = \mu_i + \alpha \cdot (ESG_i - \overline{ESG})$ .
- Strategy 3 (ESG-Modified Distance): Blends financial correlation distance and ESG Euclidean distance in the hierarchical clustering phase: $d^{combined}_{ij} = (1-\beta)d^{financial}_{ij} + \beta d^{ESG}_{ij}$ .
- Strategy 4 (Hybrid): Simultaneously applies return adjustments and distance modifications with independently tunable parameters $\alpha$ and $\beta$ .
Portfolio Evaluation:
Backtests are conducted over the 2020–2023 period (encompassing the COVID-19 crash, recovery, and inflationary shock) on a subset of 48 liquid assets. Performance is evaluated using risk-adjusted metrics (Sharpe, Sortino, Calmar, Omega), diversification measures (Effective Number of Assets, Herfindahl Index, Gini Coefficient), and portfolio-level ESG scores.

Key Contributions

Systematic Model Selection: The study provides a comprehensive comparative evaluation of nine ML algorithms for ESG classification, identifying Logistic Regression as the optimal model. It explicitly documents the risk of overfitting in high-capacity ensemble methods (Gradient Boosting, LightGBM) that achieve perfect cross-validation scores but suffer significant test-set degradation.
Structural ESG Integration: It proposes and validates four distinct mechanisms for embedding ESG data into HRP, moving beyond simple exclusionary screening to structural modifications of return expectations and hierarchical clustering distances.
Empirical Trade-off Analysis: The research quantifies the "ESG–Sharpe–diversification frontier," demonstrating that soft integration mechanisms can meaningfully raise portfolio ESG scores while preserving risk-adjusted performance and diversification, whereas hard pre-screening imposes severe concentration and performance costs.
Methodological Rigor: The study emphasizes cross-validation stability, class balancing within folds, and multi-metric assessment (including AUC-PR and MCC) to ensure model reliability for institutional deployment.

Results

Classification Performance: Logistic Regression emerged as the superior classifier, achieving 89.58% test accuracy, a macro F1-score of 0.8836, and an MCC of 0.844. It demonstrated exceptional cross-validation stability (CV F1 = 0.9809 ± 0.0044).
Overfitting Detection: Gradient Boosting and LightGBM achieved perfect cross-validation F1-scores (1.000) but degraded to ~79% test accuracy, revealing severe overfitting to fold-specific patterns. This highlights the danger of relying solely on validation metrics.
Feature Importance: The three ESG pillar scores (E, S, G) were the dominant predictors, with Environmental and Social scores carrying the highest weights. Technical indicators, particularly short-horizon volatility and momentum, provided complementary predictive signal.
Portfolio Performance (2020–2023):
- Hard Pre-screening: Increased the average portfolio ESG score by 86% (to 72.5) but reduced the Sharpe ratio by 85% (to 0.352) and collapsed the effective number of assets to 6.5.
- Soft Integration (Hybrid/Adjusted): Improved average ESG scores by 6–12% (to ~42–44) while maintaining Sharpe ratios in the 0.50–0.54 range and preserving effective asset counts near 47.
- Traditional HRP: Served as a baseline with a Sharpe ratio of 0.786 and an average ESG score of 39.0.
- Mean-Variance: Achieved the highest Sharpe ratio (2.375) but with extreme concentration (Effective N = 2.77) and zero exposure to high-ESG assets.

Significance and Claims
The paper claims to establish a transparent and stable foundation for ESG-informed HRP. It argues that:

Simplicity Prevails: Contrary to the assumption that complex non-linear models dominate financial prediction, a linear model (Logistic Regression) offers superior generalization, stability, and interpretability for ESG classification tasks.
Soft Integration is Superior: "Soft" integration mechanisms (adjusting returns or clustering distances) allow investors to accommodate sustainability objectives with limited diversification loss and manageable performance trade-offs. In contrast, "hard" exclusionary screening imposes strong normative constraints at the cost of markedly increased concentration and reduced risk-adjusted performance.
Practical Deployability: The proposed framework offers a practically deployable solution for institutional portfolio management, balancing fiduciary duties with ESG mandates. It provides a quantitative basis for calibrating integration intensity based on investor preferences, showing that marginal improvements in ESG scores can be achieved with moderate reductions in risk-adjusted returns.

The study positions itself as a proof-of-concept for ESG-augmented HRP, emphasizing that its findings are specific to the S&P 500 large-cap universe and a single ESG provider, while offering a modular framework adaptable to broader universes and multiple data sources.