💻 computer science

Deeper network methods for solving 3D American Heston-CIR model

This paper proposes three deeper network methods—SoftNet, LinNet, and AddNet—utilizing multiplicative ensembling of sub-networks with feature reuse, linear aggregation, and early exits to effectively solve the 3D American Heston-CIR model while mitigating vanishing gradients and performance degradation in deep architectures.

Original authors: Chinonso Nwankwo, Weizhong Dai

Published 2026-10-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chinonso Nwankwo, Weizhong Dai

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of finance, predicting the future price of an asset is a high-stakes game of probability. Investors and banks rely on complex mathematical models to determine the value of options, which are contracts giving the right to buy or sell something at a specific price by a certain date. One of the most difficult challenges in this field is pricing "American" options. Unlike their European counterparts, which can only be exercised on a single specific date, American options can be exercised at any time before they expire. This flexibility creates a moving target: the model must constantly calculate not just the value of the option, but also the exact moment it becomes most profitable to cash out. This critical moment is known as the early exercise boundary. When the model involves multiple changing factors—such as the asset's price, the volatility of that price, and fluctuating interest rates—the problem becomes a three-dimensional puzzle that is notoriously difficult to solve with traditional computer methods.

For decades, scientists have turned to artificial intelligence, specifically deep learning, to tackle these complex puzzles. These computer programs, known as neural networks, are designed to learn patterns from data by stacking layers of processing units. The deeper the stack, the more complex the patterns the network can theoretically understand. However, a persistent problem has plagued these deep networks: as they grow taller, they often become unstable. The signals used to teach the network can fade away or explode in magnitude before reaching the top layers, causing the system to fail to learn anything useful. This phenomenon, often called the vanishing gradient effect, has historically limited how deep these networks could be made, especially for the intricate, non-linear problems found in financial mathematics.

In a recent study, researchers Chinonso Nwankwo and Weizhong Dai set out to break through this barrier. They focused on a specific, highly complex financial model called the American Heston-CIR model, which tracks three moving parts simultaneously: the asset price, its volatility, and the interest rate. Their goal was to design a new type of neural network that could train effectively with far more layers than ever before, allowing it to capture the subtle, irregular behaviors of the market without collapsing under its own complexity. Instead of simply stacking more layers on top of each other, they invented three distinct architectural strategies, which they named SoftNet, LinNet, and AddNet. These new designs were built to bypass the limitations that had previously stopped deep networks from working on such difficult tasks.

The researchers began by testing the limits of standard, conventional networks. They found that when they tried to use a traditional design with just thirteen hidden layers, the network performed poorly. It failed to accurately predict the value of the options or the point at which they should be exercised. The deeper the network went, the worse it became, confirming that the standard approach was hitting a wall. To fix this, the team introduced their three new methods. The first, SoftNet, works by breaking the network into smaller blocks and connecting them in a way that allows information to be reused and shared efficiently, much like a train where each car passes its cargo to the next while keeping its own load. This design allowed the network to grow deeper without losing the signal. The second method, AddNet, takes a different approach by creating multiple "early exits" within the network. Instead of forcing every piece of information to travel all the way to the very end, the network makes partial decisions at various stages and combines them at the end. This ensures that even if the deepest layers struggle, the earlier, shallower layers still contribute valuable insights.

The third and most efficient design, LinNet, combines the best of both worlds. It uses a system of adding linear units together before passing them through specialized processing blocks. This structure prevents the computational cost from exploding as the network gets deeper, making it highly efficient. The researchers pushed LinNet to an extreme depth, training it with up to one hundred hidden layers. Remarkably, unlike the conventional networks that failed at thirteen layers, LinNet maintained high accuracy even at this extreme depth. The results showed that these new architectures could predict the value of the options and the early exercise boundary with a level of precision that matched or exceeded established, highly respected mathematical methods. The networks successfully mapped out the complex surfaces of the financial data, including the sensitive derivatives known as "Greeks," which measure how the option price reacts to small changes in the market.

A key finding of the study was that these new networks achieved their success without relying on the usual safety nets that deep learning experts typically use to prevent errors, such as dropout or batch normalization. The researchers demonstrated that the specific way they structured the layers—using special activation functions and multiplicative ensembling—was enough to keep the network stable. They also discovered that by concentrating their training data in the most difficult areas of the problem, specifically near the boundaries where the market behavior is most erratic, the networks learned much faster and more accurately. The study concluded that while standard deep learning hits a wall around thirteen layers for this type of problem, the new SoftNet, AddNet, and LinNet architectures can go much deeper, with LinNet proving particularly robust and efficient. This work suggests that by rethinking how neural networks are built, rather than just making them bigger, scientists can solve some of the most stubborn problems in financial modeling, opening the door to more accurate pricing for complex, real-world financial instruments.

Technical Summary: Deeper Network Methods for Solving 3D American Heston-CIR Model

Problem Statement
The paper addresses the challenge of solving the three-dimensional American Heston-CIR model, a complex financial partial differential equation (PDE) involving a free boundary (the early exercise boundary). The model incorporates three stochastic factors: the underlying asset price ( $S$ ), variance ( $\upsilon$ ), and interest rate ( $r$ ). The objective is to simultaneously approximate the value function ( $V$ ), the early exercise boundary ( $\hat{s}_f$ ), and the Greeks (derivatives of the value function).

The authors identify a critical limitation in existing deep learning approaches for this specific task: conventional Deep Neural Networks (DNNs) with softmax units suffer from performance degradation, overfitting, and the vanishing gradient effect when the network depth exceeds approximately 13 hidden layers. While architectures like DenseNet and ResNet have succeeded in image classification (CNNs), their direct application to non-linear regression problems in high-dimensional financial modeling has proven unsuitable, particularly when dealing with the singularities and irregularities found near the early exercise boundary.

Methodology
To overcome the limitations of conventional DNNs, the authors propose three novel network architectures: SoftNet, LinNet, and AddNet. All three models utilize a "multiplicative ensembling" strategy, combining a deep main sub-architecture with shallow supportive sub-architectures to satisfy terminal conditions exactly.

SoftNet (Feature Reuse via Concatenation):
- Structure: Comprises a deep first sub-architecture and a shallow second sub-architecture (two hidden layers).
- Mechanism: The first sub-architecture is built using "softmax blocks" (sub-networks with softmax activation). It employs feature reuse via concatenating shortcut connections between these blocks, inspired by DenseNet but adapted for regression.
- Design: The input to each subsequent softmax block (from the third block onwards) is the concatenation of the original input features and the outputs of previous blocks. A modular variant, M-SoftNet, is also introduced, which uses multiple pathways with different feature reuse strategies (dense, partial, and none) that are concatenated before the final ensembling.
AddNet (Early Exits and Multi-Mixture):
- Structure: Designed to be resource-efficient, utilizing early exits (local exits) from intermediate layers of the main deep sub-architecture.
- Mechanism: The network generates $n$ single-mixture deep learning solutions. Each solution is formed by multiplicatively ensembling an output from a "local exit" (or the final layer) of the main architecture with outputs from two shallow supportive sub-architectures.
- Ensembling: These $n$ solutions are then additively ensembled with adaptive weights ( $\hat{\pi}_j$ ). The authors propose an adaptive weighting strategy where shallower local exits are assigned higher weights as the network depth increases, mitigating the impact of vanishing gradients in deeper layers.
LinNet (Aggregation of Linear Units):
- Structure: The most efficient of the three, designed to train with extreme depth (up to 100 hidden layers).
- Mechanism: Instead of concatenating features (which increases width and computational cost), LinNet uses systematic aggregation of linear units. It alternates between "softmax blocks" and "aggregating units" where linear outputs are summed before being passed to the next softmax block.
- Stability: The authors argue that the softmax activation function acts as a classification mechanism that controls the growth of combined linear units, ensuring numerical stability without the need for explicit regularization techniques like dropout or batch normalization.

Key Contributions

Architectural Innovation: The proposal of three distinct deep network architectures (SoftNet, LinNet, AddNet) specifically tailored for solving 3D American option pricing problems, moving beyond standard CNN-based or conventional DNN approaches.
Mitigation of Vanishing Gradients: The designs successfully circumvent the vanishing gradient effect that plagues conventional DNNs beyond 13 layers. The authors demonstrate that while standard DNNs fail at depth 13, their proposed models maintain predictive accuracy at depths of 25, 35, 52, and even 100 layers (specifically for LinNet).
Exact Terminal Condition Imposition: All models incorporate a multiplicative ensembling layer with a shallow second sub-architecture designed to ensure that terminal conditions (e.g., $V=0$ at maturity) are satisfied in exact form, removing bias terms from hidden layers.
Handling Singularities: The methodology includes a grid stretching strategy using the sinh function to concentrate sampling points near the singularity at $(0,0,0,0)$ , where the early exercise boundary exhibits irregularities.

Experimental Results
The models were tested on the 3D American Heston-CIR model with parameters including varying strike prices ( $K$ ), maturities ( $T$ ), and volatility/interest rate ranges.

Performance vs. Depth: Conventional DNNs (C-DNN) with 13 hidden layers produced poor results (e.g., value function errors > 9.0 vs. reference ~1.36). In contrast, SoftNet, AddNet, and LinNet with 13+ layers produced results consistent with established numerical methods (Boyarchenko & Levendorski˘I, Medvedev & Scaillet, Longstaff & Schwartz).
LinNet Efficiency: LinNet demonstrated the highest efficiency, achieving reasonable predictive performance with up to 100 hidden layers. It required the least training and prediction runtime compared to SoftNet and AddNet.
Accuracy: The models accurately predicted the value function, the early exercise boundary, and the Greeks (Delta, Gamma, Vega, Rho, Vanna, Vomma). The predicted surfaces were smooth, and the Greeks fell within expected theoretical bounds (e.g., Delta in $[-1, 0]$ for puts).
Dataset Sensitivity: Experiments showed that increasing the training dataset size (from $10^4$ to $14^4$ points) generally improved performance, particularly for M-SoftNet-10.

Significance and Claims
The paper claims that the proposed trio of networks (SoftNet, LinNet, AddNet) offers a viable solution for training deep neural networks on complex, high-dimensional financial PDEs where conventional architectures fail.

LinNet is highlighted as the most efficient solution, capable of training with 100 hidden layers without regularization, making it suitable for resource-constrained environments and extremely deep tasks.
SoftNet and AddNet provide alternative strategies (feature reuse and early exits, respectively) that also mitigate vanishing gradients, though SoftNet incurs higher computational costs due to concatenation.
The authors modestly note that while their models perform well without regularization, the inclusion of techniques like dropout or batch normalization could further enhance generalization for even more complex, higher-dimensional tasks. They position this work as a foundation for future research into full parametrization of American pricing models involving correlation matrices and higher dimensions.

More like this