An implication of activation functions for Physics-Informed Neural Networks (PINNs) to solve non-autonomous differential equations

In the vast landscape of modern science and engineering, predicting how systems change over time is a fundamental challenge. Whether tracking the orbit of a satellite, modeling the spread of a disease, or simulating the flow of heat through a material, scientists rely on a specific type of mathematical rule known as a differential equation. These equations describe how a quantity changes in relation to itself and its environment. For decades, solving these equations has been the domain of classical numerical methods, which break problems down into tiny, manageable steps. While reliable, these traditional approaches can become computationally expensive and struggle with complex, high-dimensional problems. In recent years, a new approach has emerged, borrowing from the field of artificial intelligence. Researchers have begun using neural networks—computer systems inspired by the human brain—to learn and approximate the solutions to these equations. These systems, known as Physics-Informed Neural Networks, are trained not just on data, but on the physical laws themselves, allowing them to find solutions that respect the underlying science.

However, for these neural networks to function, they require a specific internal mechanism to introduce complexity and non-linearity, much like a switch that allows a circuit to do more than just pass electricity straight through. This mechanism is called an activation function. It determines how the network processes information at each step, shaping its ability to learn and approximate the correct answer. For a long time, the choice of which activation function to use in these scientific solvers has been somewhat of a guess, with researchers often picking one based on habit rather than evidence. This uncertainty matters because the wrong choice can lead to inaccurate predictions or a system that fails to learn the solution at all. The question remains: which of the many available options is truly the best for solving the complex, time-dependent equations that govern our physical world?

A recent study set out to answer this question with a systematic and rigorous approach. The researchers focused on a specific class of problems: non-autonomous ordinary differential equations. In plain terms, these are equations where the rules of change are not static but evolve over time, often driven by external factors like time itself or changing environmental conditions. To test their theories, the team constructed a standard neural network architecture and subjected it to a battery of ten distinct mathematical challenges. These challenges included equations driven by exponential growth, polynomial curves, and trigonometric waves, covering a wide spectrum of behaviors found in nature. Crucially, the researchers kept every other variable in their experiment identical. They used the same network structure, the same number of training cycles, and the same data points for every test. The only thing that changed was the activation function. They tested nine of the most common functions used in the field, ranging from the well-known Sigmoid and ReLU to more modern variants like GELU, ELU, and SiLU.

The results of this extensive simulation were clear and decisive. The study found that no single function was perfect for every single scenario, but one stood out as the most reliable performer across the board. The GELU activation function emerged as the top choice, delivering the most accurate solutions in sixty percent of the experiments. It consistently produced the smallest errors when compared to the known, exact answers for these equations. Following closely behind were two other functions, ELU and SELU, which also demonstrated strong capabilities. In contrast, some of the older, more traditional choices, specifically Sigmoid and Softmax, performed poorly. The researchers attributed this failure to a phenomenon known as saturation, where these functions become so flat in certain regions that they stop learning effectively, causing the network to stall. The study also noted that while functions like ReLU are popular in general artificial intelligence, they were not the best fit for these specific scientific tasks, likely because their sharp, angular nature creates difficulties when calculating the precise derivatives required by physics-based solvers.

To ensure these findings were not a fluke, the researchers expanded their testing to include a second set of six additional equations. The results held firm. In this extended group, GELU again took the top spot in roughly two-thirds of the cases, with ELU taking the lead in the remainder. No other function managed to secure the first position in this validation group. The study suggests that the success of GELU and ELU is rooted in their mathematical smoothness. Unlike functions with sharp corners or flat plateaus, these smooth functions allow the network to calculate changes and gradients more reliably, which is essential when the network is being asked to satisfy the strict laws of physics embedded in the equations. This smoothness helps the training process converge more stably, leading to more accurate final results.

The implications of this work are practical and immediate for scientists and engineers who rely on these computational tools. The study provides a clear, evidence-based guideline for selecting the right tool for the job. Rather than treating the choice of activation function as a random or universal setting, the researchers suggest that the specific nature of the problem should dictate the choice. For general time-dependent problems, GELU appears to be the most robust default option. For problems involving polynomial interactions, ELU offers a strong alternative. Meanwhile, the study explicitly advises against using saturating functions like Sigmoid for these types of residual-based solvers, as they are likely to hinder performance. By identifying the most effective activation functions, this research helps refine the capabilities of neural networks, paving the way for more efficient and accurate methods of solving the differential equations that describe our universe. The work does not claim to have solved every problem in the field, but it offers a solid foundation for future developments, pointing toward a more systematic and informed approach to building the next generation of scientific computing tools.