🤖 machine learning

HyperNOs: Automated and Parallel Library for Neural Operators Research

HyperNOs is an automated, parallel PyTorch library that leverages Ray-tune for efficient hyperparameter optimization and offers a flexible, user-friendly interface for training and evaluating neural operators like Fourier and convolutional models across diverse datasets and resolutions.

Original authors: Massimiliano Ghiotto

Published 2026-02-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Massimiliano Ghiotto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to predict how water flows through a complex network of pipes, or how heat spreads through a metal plate. In the world of science, these are called "Partial Differential Equations" (PDEs). Traditionally, solving these problems is like trying to calculate every single drop of water or every atom of heat by hand—it takes forever and requires massive supercomputers.

Enter Neural Operators. Think of these as a special kind of AI that doesn't just learn from specific pictures of pipes; it learns the rules of how fluids and heat behave. Once trained, it can predict the outcome for any new pipe setup instantly, regardless of how detailed the map is.

However, there's a catch. Building these AI models is like trying to tune a giant, complex radio with thousands of dials. If you turn the dials (called hyperparameters) just slightly wrong, the signal is static. If you get them right, the music is crystal clear. Finding the perfect combination of dials usually requires a human expert to guess, test, fail, and guess again for days.

HyperNOs is a new software library (a toolbox for programmers) that automates this tuning process. Here is how it works, broken down into simple concepts:

1. The "Auto-Tuner" for Science

Imagine you have a radio with 20 dials. Instead of you turning them one by one, HyperNOs is like a team of 50 robots working in parallel. They all try different combinations of dials at the same time.

The Magic: It uses a smart system (called Ray-tune) that learns from the failures. If a combination of dials sounds bad, the robots stop wasting time on it and focus on the combinations that sound promising.
The Result: It finds the "perfect tune" for the AI much faster and better than a human could, leading to predictions that are significantly more accurate.

2. The Two Main "Musicians"

The paper focuses on two specific types of AI "musicians" (architectures) that HyperNOs helps tune:

FNO (Fourier Neural Operator): Think of this as a musician who listens to the entire song at once to understand the melody. It looks at the whole picture globally. It's great for smooth, wave-like patterns.
CNO (Convolutional Neural Operator): Think of this as a musician who listens to small, local chunks of the song and pieces them together. It's great for detailed, local textures.
HyperNOs can tune both of these musicians to play their best.

3. Special Features of the Toolbox

The paper highlights a few clever tricks the library uses to make life easier for scientists:

The "Fair Fight" Mode: Sometimes you want to compare two different AI models, but one is huge and the other is tiny. That's not a fair test. HyperNOs has a feature that forces both models to have the exact same number of "brain cells" (parameters). This ensures you are comparing their intelligence, not just their size.
The "Mix-and-Match" Datasets: Imagine you have data from a small pipe and data from a giant pipe. Usually, you can't mix them. HyperNOs lets you feed both into the AI at the same time, teaching it to handle different sizes and resolutions effortlessly.
The "Physics Safety Net": Sometimes the AI might guess something that breaks the laws of physics (like water flowing uphill without a pump). HyperNOs allows you to add a "rulebook" to the training, forcing the AI to respect physical laws while it learns.

4. What Did They Find?

The authors tested this library on a variety of scientific puzzles, including:

How fluids move (Navier-Stokes).
How heat spreads (Heat equation).
How waves travel (Wave equation).
How air flows over a wing (Euler equation).

The Big Discovery:
When they used the "best settings" found by humans in previous studies, the AI made errors. But when they let HyperNOs automatically find the best settings, the errors dropped dramatically.

In one specific test (smooth transport), the error went from 0.40% down to 0.036%. That is a massive improvement—like going from a blurry photo to a 4K image.
Even when they forced the AI to use the same number of "brain cells" as the old human-tuned models, the automated tuning still made the model perform better.

Summary

HyperNOs is a tool that takes the hard, boring work of tuning scientific AI models and hands it over to a smart, parallel computing system. It proves that by letting machines find the best settings for other machines, we can solve complex physics problems faster and more accurately than ever before. It's not just about building a better model; it's about building a better process for finding the best model.

Technical Summary: HyperNOs: Automated and Parallel Library for Neural Operators

Problem Statement

The field of scientific machine learning has seen significant advances in using Neural Operators (NOs) to solve partial differential equations (PDEs) and model complex dynamical systems. Unlike traditional neural networks that operate on finite-dimensional vectors, NOs learn mappings between infinite-dimensional function spaces, offering resolution independence and superior generalization across different discretizations. However, training these models presents a critical bottleneck: hyperparameter optimization.

The hyperparameter space for NOs is vast and complex, encompassing architectural choices (layers, width, activation functions), optimization parameters (learning rates, schedulers), and regularization techniques. Manual tuning is labor-intensive, error-prone, and often requires significant domain expertise. Furthermore, standard grid or random searches are computationally prohibitive given the cost of training NOs on large datasets. There is a lack of accessible, automated tools that can systematically explore this space using parallel computing to identify optimal configurations for diverse architectures and datasets.

Methodology

The paper introduces HyperNOs, a PyTorch library designed to automate and parallelize the exploration of neural operators. The methodology is built upon the following core components:

Architecture Integration: The library natively supports two state-of-the-art NO architectures:
- Fourier Neural Operators (FNOs): Which utilize the Fast Fourier Transform (FFT) to parameterize integral operators in the frequency domain, capturing global dependencies.
- Convolutional Neural Operators (CNOs): Which extend Convolutional Neural Networks (specifically U-Nets) to the continuous setting, satisfying the Continuous-Discrete Equivalence (CDE) condition and ensuring resolution invariance through bandlimited function spaces.
Automated Optimization Engine: HyperNOs leverages the Ray-tune library to implement state-of-the-art hyperparameter optimization.
- It utilizes the Tree-structured Parzen Estimator (TPE) and adaptive TPE algorithms for efficient search.
- It employs the ASHA (Asynchronous Successive Halving Algorithm) to terminate underperforming trials early, significantly reducing computational waste.
- The system supports parallel computing, allowing multiple trials to run simultaneously across available CPU and GPU resources.
Modular Design: The library is structured to be highly extensible. Users define:
- Search Spaces: Using Ray-tune primitives (e.g., choice, randint, uniform).
- Model Builders: Functions that instantiate models (FNO or CNO) based on configuration dictionaries.
- Dataset Builders: Functions to load and preprocess data, supporting standard benchmarks and custom datasets.
- Loss Functions: Customizable objectives, including relative $L_p$ errors and physics-informed losses.
Advanced Functionalities:
- Fixed Complexity Training: A utility to constrain optimization to models with a specific number of trainable parameters, enabling fair comparisons between architectures of equal complexity.
- Multi-Resolution and Multi-Dataset Training: The ability to concatenate datasets with different resolutions or from different sources to train a single model, leveraging the resolution-invariance property of NOs.
- Physics-Informed Learning: Support for weak imposition of physical laws via modified loss functions (e.g., adding PDE residuals) and strong imposition via model wrappers (e.g., enforcing boundary conditions).

Key Contributions

The paper presents the following specific contributions:

HyperNOs Library: A user-friendly, open-source PyTorch library that streamlines the entire workflow of NO research, from dataset loading to hyperparameter optimization and final model training.
Comprehensive Benchmarking: The library includes implementations of standard datasets (Poisson, Darcy, Navier-Stokes, Wave, Euler, Transport, and Allen-Cahn equations) and provides a framework for evaluating both FNO and CNO architectures.
Optimization Strategies: The integration of advanced algorithms (TPE, ASHA) and parallel execution to handle the high computational cost of NO training.
Reproducibility and Analysis: The library automatically saves checkpoints, optimal configurations, and trained models. It provides tools to analyze the dependence of performance on hyperparameters and model complexity (degrees of freedom).

Results

The authors conducted extensive numerical experiments on the representative benchmarks mentioned above, comparing three configurations:

State-of-the-Art (s.o.t.a.): Using hyperparameters reported in previous literature (specifically [31]).
Same Degrees of Freedom (Same DoF): Optimized configurations constrained to have the same number of trainable parameters as the s.o.t.a. models.
Our Hyperparams: Unconstrained optimization using the HyperNOs library to find the best possible configuration.

Key Findings:

Performance Gains: Automated optimization consistently outperformed manual s.o.t.a. configurations. For example, on the Smooth Transport Equation with FNOs, the relative mean $L_1$ error dropped from 0.40% (s.o.t.a.) to 0.036% (optimized), representing an order-of-magnitude improvement.
Complexity vs. Performance: Even when constrained to the same number of parameters as the s.o.t.a. models, the optimized configurations often yielded better results (e.g., reducing error from 0.40% to 0.17% for the Smooth Transport Equation with FNOs).
Scalability: The experiments demonstrated that increasing the number of trainable parameters (via unconstrained optimization) further reduced errors, with the best FNO configuration (250M parameters) achieving a 0.036% error on the Smooth Transport Equation.
Architecture Comparison: Both FNO and CNO architectures benefited from the optimization, with CNOs showing particular strength in problems requiring translation equivariance or specific resolution invariance properties.

Significance and Claims

The paper claims that HyperNOs addresses a critical gap in the scientific machine learning ecosystem by providing an automated, scalable, and reproducible framework for studying neural operators.

Efficiency and Automation: By offloading the tuning process to machines using parallel computing, the library saves researchers time and eliminates the bias and limitations of human intuition in navigating complex hyperparameter spaces.
Fair Comparison: The ability to fix the number of trainable parameters allows for rigorous, fair comparisons between different architectures (e.g., FNO vs. CNO) without confounding factors related to model size.
Accessibility: The library lowers the barrier to entry for researchers, allowing them to focus on model design, interpretation, and application rather than the tedious mechanics of hyperparameter tuning.
Foundation for Future Work: The authors position HyperNOs as a stepping stone for future developments, aiming to integrate more architectures (DeepONets, Graph Neural Operators, Transformers) and further streamline the deployment of NOs in real-world scientific and engineering applications.

The paper concludes that automated hyperparameter optimization is not merely a convenience but a fundamental requirement for unlocking the full potential of neural operators in solving complex, real-world problems.