🤖 AI

No One-Size-Fits-All Neurons: Task-based Neurons for Artificial Neural Networks

This paper proposes a two-step framework for designing task-specific artificial neurons that leverage intrinsic inductive biases to outperform traditional universal neurons, demonstrating that moving from task-based architecture design to task-based neuron design enhances feature representation and achieves competitive performance across various benchmarks.

Original authors: Feng-Lei Fan, Meng Wang, Hang-Cheng Dong, Jianwei Ma, Tieyong Zeng

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Feng-Lei Fan, Meng Wang, Hang-Cheng Dong, Jianwei Ma, Tieyong Zeng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Stop Using the Same Tool for Every Job

Imagine you are a master carpenter. For the last decade, you've been building every single piece of furniture—chairs, tables, bookshelves, and cabinets—using the exact same type of hammer. It's a good hammer, and it works well enough for many things. But you've noticed that sometimes, a screwdriver or a chisel would do a much better job for specific tasks.

In the world of Artificial Intelligence (AI), most networks are built like that carpenter. They use the same "neuron" (the basic building block of the brain-like computer) for every single problem, whether it's predicting the weather, recognizing a cat in a photo, or analyzing stock prices.

This paper argues: Why use the same hammer for everything?

The authors propose that just like the human brain uses different types of neurons for different jobs, AI should too. They call these "Task-based Neurons." Instead of forcing every problem into a generic box, they want to design a custom tool specifically for the job at hand.

The Problem with the "Universal" Hammer

Current AI networks are "homogeneous," meaning every single neuron inside them is identical. They all do the same math: they take inputs, multiply them by numbers, add them up, and pass the result through a simple filter (like a light switch that turns on or off).

The authors say this is inefficient. It's like trying to fix a watch with a sledgehammer. You can do it, but you might break the watch, or it might take you twice as long.

The Solution: A Two-Step "Custom Tool" Factory

The paper introduces a new framework to build these custom neurons. Think of it as a factory that designs the perfect tool for a specific job before you even start building the machine.

Step 1: The Detective (Vectorized Symbolic Regression)

First, the system acts like a detective looking at the data to find the hidden "recipe" or pattern.

The Analogy: Imagine you are trying to guess the secret recipe for a soup. You taste the ingredients (data) and try to write down the exact formula.
The Innovation: Usually, trying to find this formula is slow and messy, especially if you have hundreds of ingredients (high-dimensional data). The authors invented a method called VSR (Vectorized Symbolic Regression).
How it works: Instead of guessing the recipe for every single ingredient separately, VSR assumes all ingredients follow the same type of rule. It groups them together and solves the puzzle much faster. It finds the mathematical "shape" of the problem. For example, it might realize, "Oh, this specific problem isn't a straight line; it's actually a curve that looks like a parabola."

Step 2: The Builder (Parameterizing the Formula)

Once the detective finds the formula, the builder takes that formula and turns it into a working neuron.

The Analogy: The detective found the blueprint for a custom wrench. The builder takes that blueprint and builds the wrench, but leaves the handle size adjustable so it can be tightened or loosened later.
The Result: This new neuron is no longer a generic "on/off" switch. It is a specialized tool that knows exactly how to handle the specific data it was designed for.

Why This Matters: The "Specialist" vs. The "Generalist"

The paper tested this idea on real-world problems, like predicting asteroid sizes and analyzing particle collisions.

The Result: The "Task-based Networks" (made of these custom neurons) performed better than the standard networks, even when they were smaller and used fewer computer resources.
The Takeaway: A network built with specialists (custom neurons) can solve a problem with a lighter, simpler structure than a network built with generalists (standard neurons). It's like having a team of expert surgeons versus a team of general practitioners; for a specific surgery, the experts get the job done faster and with less risk.

What About Images? (The "Compression" Trick)

The authors also tried this on images (like photos). You can't run a custom formula on a photo with millions of pixels directly; it would be too slow.

The Analogy: Imagine trying to describe a whole city by listing every single brick in every building. It's impossible. Instead, you first take a photo of the city and shrink it down to a small, detailed map (a "latent vector").
The Process: They used an "autoencoder" (a tool that compresses images into a small summary) to turn the photo into a small list of numbers. Then, they used their "Detective" (VSR) to find the best formula for that small list. Finally, they turned that formula into a special filter for the image.
The Outcome: Even on complex image tasks, these custom filters outperformed standard ones, especially on the hardest parts of the problem (the "worst-performing" classes of images).

Summary: The "One-Size-Fits-All" Myth is Over

The paper concludes that the era of "one-size-fits-all" neurons is ending.

Old Way: Build a huge, complex network using the same basic brick for everything, hoping it's big enough to solve the problem.
New Way: Look at the problem first, design a custom brick that fits that specific problem perfectly, and build a smaller, more efficient network with those custom bricks.

The authors have shared their code, inviting others to stop using the same hammer for every job and start designing the right tool for the task.

Technical Summary: No One-Size-Fits-All Neurons: Task-based Neurons for Artificial Neural Networks

Problem Statement
Current deep learning research predominantly focuses on designing novel network architectures (e.g., shortcuts, Neural Architecture Search) while almost exclusively utilizing a single, homogeneous type of neuron: the linear neuron (an inner product followed by a nonlinear activation). This approach contrasts with biological reality, where the human brain exhibits significant neuronal diversity and relies on "task-based" neurons rather than a universal type. While recent studies have introduced specific non-linear neurons (e.g., quadratic or polynomial), these remain fixed designs that may not be optimal for all tasks. The paper posits that there is no "one-size-fits-all" neuron and that the field should shift from designing task-based architectures to designing task-based neurons that possess an intrinsic inductive bias specific to the data at hand.

Methodology
The authors propose a two-step framework to prototype task-based neurons, centered around Vectorized Symbolic Regression (VSR):

Vectorized Symbolic Regression (VSR):
- Goal: To identify optimal analytical formulas that fit input data by utilizing a predefined space of base functions (e.g., polynomials, trigonometric functions).
- Mechanism: Unlike traditional Symbolic Regression (SR) which often yields heterogeneous formulas for different variables, VSR enforces homogeneity. It stacks all input variables into a vector and regularizes each variable to perform the same computation.
- Operators: The search space includes vector operators (Hadamard product, inner product) and scalar operators (+, −, ×, ÷) applied element-wise.
- Search Algorithm: The framework employs Genetic Programming (GP) with mutation, crossover, and selection to evolve expression trees. The output is a mathematical formula representing a basic pattern in the data.
- Advantages: VSR addresses the computational inefficiency of traditional SR in high dimensions, enables parallel computation (crucial for GPU acceleration), and reduces the search space complexity to $O(d)$ parameters.
Neuron Parameterization and Network Integration:
- Reparameterization: The fixed constants in the formula discovered by VSR are replaced with learnable parameters. This transforms the static formula into a dynamic aggregation function for a neuron.
- Activation Functions: Standard activation functions (e.g., ReLU, Sigmoid) are retained and attached to the aggregated output, as they are proven to be effective.
- Architecture Adaptation:
  - Fully Connected Networks: The learned formula is expanded into polynomial terms (e.g., $x, x \odot x, x \odot x \odot x$ ), each multiplied by a learnable weight matrix.
  - Convolutional Networks: For images, an autoencoder first reduces dimensionality. The VSR formula is then applied to latent vectors, and the resulting terms are converted into learnable convolution kernels.
- Theoretical Basis: The paper presents a theorem (Theorem 1) suggesting that if a single-layer network using task-based neurons achieves a lower optimal error ( $\|e^*_1\|$ ) than a generic neuron, then a deeper network composed of these neurons ( $\|e^*_l\|$ ) will also achieve a lower error, provided the architecture is consistent.

Key Contributions

Task-Based Neuron Framework: A novel roadmap for NeuroAI that designs neurons based on the specific task rather than using a universal architecture. This introduces a new dimension to deep learning model design.
Vectorized Symbolic Regression (VSR): The first application of symbolic regression to the design of deep learning neurons. VSR solves the scalability issues of traditional SR in high-dimensional spaces and facilitates parallel computing.
Empirical Validation: Systematic experiments across synthetic data, public benchmarks, and real-world applications demonstrating that task-based neurons and their associated networks outperform preset neurons and state-of-the-art models.

Results

Synthetic Data: VSR successfully recovered ground-truth polynomial formulas from noisy data. Task-based networks (TN) achieved significantly lower Mean Squared Error (MSE) than Linear Networks (LN) even with fewer parameters and shallower depths.
Tabular Data: On real-world datasets (High-energy Particle Collision and Asteroid Prediction), TN outperformed advanced models like XGBoost, LightGBM, CatBoost, TabNet, and various Transformer-based tabular models (TabTransformer, FT-Transformer). TN achieved the best performance on both datasets with minimal capacity (single-layer variants).
Comparison with KANs: TN demonstrated superior or highly competitive performance against Kolmogorov-Arnold Networks (KAN) and its variants (ChebyKAN, FourierKAN, FastKAN) across 16 regression and classification datasets, often with fewer parameters and FLOPs.
Image Data: When applied to the 20 worst-performing classes of ImageNet using MobileNet-V2, ResNet-18, and ViT-B-32 backbones, TN consistently outperformed conventional neurons, achieving higher Top-1 accuracy with fewer parameters.
Scalability: Experiments showed that while traditional SR struggles with dimensions above 80, VSR maintains robust performance and scales effectively up to 400 dimensions, outperforming SR in high-dimensional regimes.
Ablation: Comparisons with neurons using random polynomials confirmed that the formulas learned via VSR are not arbitrary but capture essential data patterns.

Significance and Claims
The paper claims that the proposed framework represents a significant step toward NeuroAI by introducing neuronal diversity into artificial networks. It argues that:

Efficiency: Task-based neurons can achieve high performance with shallower networks and fewer parameters compared to generic neurons, suggesting that the "depth" of a network can be traded for "neuron expressiveness."
Adaptability: The method is flexible; while the initial experiments used polynomials, the framework can extend to other base functions (e.g., trigonometric functions) to suit specific physical or signal-processing tasks.
Feasibility: The approach is computationally feasible, with VSR converging reliably and TN inference times being comparable to linear networks.
Future Potential: The authors suggest that task-based neurons can be integrated into architectures discovered by Neural Architecture Search (NAS), offering a synergistic path for future AutoML research.

The paper maintains a modest tone, acknowledging that while the results are promising, the VSR process requires further optimization for speed and that the selection of base functions could be automated more broadly in future work. It does not claim to solve all deep learning challenges but offers a viable alternative to the current paradigm of homogeneous neurons.