🤖 machine learning

ArcGate: Adaptive Arctangent Gated Activation

This paper introduces ArcGate, an adaptive arctangent gated activation function with learnable parameters that dynamically optimizes non-linearity, demonstrating superior accuracy and noise resilience over standard baselines in remote sensing tasks using ResNet-50 and ViT architectures.

Original authors: Avik Bhattacharya, Siddhant Dnyanesh Gole, Subhasis Chaudhuri, Alejandro C. Frery, Biplab Banerjee

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Avik Bhattacharya, Siddhant Dnyanesh Gole, Subhasis Chaudhuri, Alejandro C. Frery, Biplab Banerjee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a massive team of workers (a neural network) to sort through thousands of satellite photos to identify things like forests, cities, or water. In the past, every worker in this team was given the exact same rulebook for how to react to what they see.

The most popular rulebook was called ReLU. It was simple: "If you see something bright, shout it out. If it's dark, stay silent." It worked well, but it had a flaw: if a worker saw something even slightly dark, they would shut down completely and stop learning. Also, this rulebook was rigid; it couldn't change its mind based on whether the worker was a beginner (looking at simple shapes) or an expert (looking at complex patterns).

This paper introduces a new, super-smart rulebook called ArcGate.

The "Shape-Shifting" Rulebook

Instead of giving every worker the same fixed rule, ArcGate gives them a customizable toolkit. Think of it like a Swiss Army knife for decision-making.

The Old Way (ReLU): Like a light switch. It's either ON or OFF. If you push it too hard the wrong way, it breaks (the "dying neuron" problem).
The New Way (ArcGate): Like a dimmer switch with a built-in AI. It has seven special knobs that the network can turn and adjust while it's learning.

These knobs allow the network to decide:

How steep the hill is: Should the worker react slowly to small changes, or snap to attention immediately?
Where the center is: Should the worker ignore very faint signals, or pay attention to them?
How sharp the turn is: Should the reaction be a smooth curve or a sharp corner?

By turning these knobs, the network can morph its behavior. It can act like a gentle curve for beginners, a sharp switch for experts, or even a smooth ramp for complex data. It essentially says, "I will figure out the perfect rule for this specific job and this specific layer of my brain."

Why It's a Game-Changer for Satellite Photos

The researchers tested this new system on three different sets of satellite images (PatternNet, UC Merced, and EuroSAT). Here is what happened:

1. It's the Best at Sorting
When the network used ArcGate, it got better at identifying land types than any other method. On one test (PatternNet), it got 99.67% accuracy. That's like getting almost every single answer right on a massive exam. It beat the old "ReLU" method and other modern methods like GELU and SiLU.

2. It's Tougher in the Storm (Noise Robustness)
Satellite photos often have "noise"—like static on a TV, caused by clouds, sensor glitches, or atmospheric interference.

When the researchers added a moderate amount of "static" (noise) to the images, the old ReLU system crashed. Its accuracy plummeted from nearly 100% down to about 39%. It got confused and gave up.
The ArcGate system, however, stayed calm. It dropped to about 66%. While that's lower than its perfect score, it is 26% better than the old system.
The Analogy: Imagine two people trying to hear a whisper in a noisy room. The ReLU person covers their ears and stops listening entirely when the noise gets loud. The ArcGate person adjusts their hearing, filters out the static, and keeps listening.

3. It Learns to Grow Up
The researchers looked inside the network to see how the "knobs" were set. They found something fascinating:

Early layers (the beginners) used a smooth, gentle curve.
Deep layers (the experts) turned up the "gain" or strength. They made the reaction steeper and more intense.
Why? As the signal travels through 50 layers of the network, it can get weak (like a whisper traveling down a long hallway). The deeper layers learned to "turn up the volume" to make sure the important details didn't fade away. The old ReLU system couldn't do this; it was stuck with the same setting from top to bottom.

The Bottom Line

The paper claims that ArcGate is a flexible, self-adjusting activation function that lets deep learning networks adapt their own "personality" to the data they are seeing.

It doesn't just follow a rigid rule; it learns the best rule for every part of the network.
It handles messy, noisy satellite data much better than current standards.
It achieves record-breaking accuracy on remote sensing tasks without needing a supercomputer to run it (the extra "knobs" it learns add almost no extra weight to the system).

In short, ArcGate is like giving a neural network the ability to say, "I'm not just a robot with a fixed rule; I'm smart enough to figure out the best way to react to what I'm seeing."

Technical Summary: ArcGate: Adaptive Arctangent Gated Activation

Problem Statement

Activation functions are fundamental to deep neural networks, governing non-linearity, feature learning, convergence, and robustness. While historical choices like Sigmoid and Tanh suffered from vanishing gradients, and modern fixed-shape functions like ReLU, GELU, and SiLU offer computational efficiency, they lack structural flexibility. These conventional activations do not adapt to the specific requirements of different network layers or data distributions. Furthermore, fixed activations often exhibit limitations such as the "dying ReLU" problem (zero gradients for negative inputs) or non-differentiable "kinks" at the origin, which can destabilize training in deep architectures. In the context of remote sensing, where data often exhibits high dynamic ranges and is susceptible to sensor noise and atmospheric disturbances, the rigidity of standard activation functions may hinder optimal performance and resilience.

Methodology

The paper proposes ArcGate (Adaptive Arctangent Gated Activation), a flexible, parametric activation function designed to autonomously optimize its non-linearity. ArcGate is constructed via a three-stage non-linear transformation involving seven learnable parameters per layer: $a, c, p, \alpha, \beta, \gamma, \delta$ .

The formulation proceeds as follows:

Monotonic Transition: A smooth transition function $u(x; a, c)$ maps the input to $(0, 1)$ using an arctangent function. The parameter $a$ controls steepness, and $c$ sets the transition center.
Non-linear Stretching (Gating): An adaptive gating function $v(x; p)$ is derived by applying an odds transformation to $u(x)$ , raising it to the power $p$ , and mapping it back to $(0, 1)$ via arctangent. The parameter $p$ controls the sharpness of the gate, allowing transitions from sigmoid-like to near piecewise-linear behavior.
Affine Combination: The final activation $F(x)$ combines the gating function with linear and bilinear terms:
$F(x; \alpha, \beta, \gamma, \delta) = (\alpha x + \beta)v(x; p) + (\gamma x + \delta)$
Here, $\alpha$ regulates the slope in the active region, $\beta$ controls the non-linear contribution, $\gamma$ introduces a linear path (leakage), and $\delta$ provides a constant shift.

Mathematical Properties

Generalization: ArcGate forms a functional superset capable of replicating classical activations (ReLU, Sigmoid, Tanh, SiLU, GELU, Leaky ReLU) as special cases or close approximations by tuning its parameters.
Continuity and Differentiability: Unlike ReLU, ArcGate is $C^\infty$ (infinitely differentiable) everywhere due to its composition of arctangent and power functions, ensuring stable backpropagation.
Numerical Stability: The use of arctangent as a squashing mechanism prevents overflow issues common in exponential-based functions (like Sigmoid) during half-precision training.
Learnable Bias: The parameters $c$ and $\delta$ allow the activation to learn an internal bias independent of weight layers, adapting to specific data statistics.

Experimental Setup

The authors evaluated ArcGate using ResNet-50 and Vision Transformer (ViT-B/16) architectures on three remote sensing benchmarks:

PatternNet: A high-resolution land-cover dataset.
UC Merced Land Use: A standard land-use classification dataset.
EuroSAT MSI: A 13-band multispectral dataset.

For multispectral inputs (EuroSAT), models were adapted to 13 channels. Training utilized AdamW optimization over 50 epochs. ArcGate parameters were initialized to a "Soft ReLU" configuration ( $a=5.0, p=1.0$ , etc.) to ensure stable early training.

Key Results

1. Classification Accuracy

ArcGate consistently outperformed standard baselines (ReLU, Leaky-ReLU, SiLU, GELU, etc.) across all datasets and architectures.

PatternNet: Achieved a peak Overall Accuracy (OA) of 99.67% with ResNet-50, surpassing the ReLU baseline (99.57%) and SiLU (99.01%).
UC Merced: Achieved 98.15% (ResNet-50) and 98.33% (ViT-B/16), outperforming ReLU and SiLU.
EuroSAT: Demonstrated competitive performance, with ResNet-50 achieving 98.47% OA.

2. Noise Robustness

A stress test involving Gaussian noise ( $\sigma \in [0, 0.2]$ ) on the PatternNet dataset revealed ArcGate's superior structural resilience.

Under moderate noise ( $\sigma = 0.1$ ), the ReLU-based model's accuracy dropped drastically to 39.34%.
The ArcGate model maintained an accuracy of 65.99%, representing a 26.65% absolute improvement over the ReLU baseline.
This resilience is attributed to ArcGate's smooth, fully differentiable transition, which naturally regularizes the response and limits noise amplification compared to ReLU's sharp discontinuity.

3. Layer-wise Adaptation and Ablation

Depth-Dependent Evolution: Analysis of learned parameters showed that the gating slope ( $\alpha$ ) increases with network depth (from 0.99 in shallow layers to 1.12 in deep layers). This suggests the network autonomously learns to amplify signal intensity in deeper layers to counteract attenuation and enhance feature saliency.
Granularity: An ablation study confirmed that Layer-wise Learnable parameters (343 extra parameters total) outperformed Global Shared parameters by 0.42% and Fixed parameters by 0.66%. The computational overhead was negligible (<0.002% of total parameters).
Initialization: Initializing with a "Soft-ReLU" configuration yielded faster convergence and near-optimal results compared to random or identity initialization.

Significance and Claims

The paper posits that ArcGate is a robust and adaptive general node activation function specifically beneficial for high-resolution Earth observation tasks. Its primary significance lies in:

Autonomous Adaptation: Unlike fixed activations, ArcGate allows the network to shape its non-linearity to the specific feature hierarchy and data distribution, evidenced by the depth-dependent parameter evolution.
Enhanced Robustness: The smooth, differentiable nature of the function provides significant resilience against the signal perturbations common in remote sensing (e.g., sensor noise), outperforming standard ReLU by a wide margin in noisy conditions.
Unified Framework: ArcGate unifies various activation behaviors (rectifying, saturating, linear, smooth non-monotonic) into a single parametric family, demonstrating that classical functions are special cases of this broader formulation.

The authors conclude that while ArcGate shows promise for SAR and optical image analysis, future work will explore its application in lightweight architectures and complex-valued networks.