💻 computer science

The Imposed and Emergent Pieces of Convolution Under an Energy Budget

This paper uses an evolutionary search under an energy budget to decompose convolutional neural networks into imposed priors and emergent structures, revealing that sparse connectivity and weight sharing arise naturally while compact filters require specific mutation mechanisms and composition on the channel axis hits a capacity wall without supervision.

Original authors: Vasili Gavrilov

Published 2026-09-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vasili Gavrilov

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Deep inside the modern artificial intelligence that recognizes faces, translates languages, and drives cars, there is a specific kind of mathematical engine called a convolutional network. For decades, scientists have built these engines by hand, carefully wiring them to mimic how the human brain processes sight. They have hardwired rules into the code: that nearby pixels should be connected, that a pattern found on the left side of an image should be recognized on the right side, and that the same filter should scan the whole picture. These rules are called priors, or pre-existing assumptions, and they make the machines efficient. But a lingering question has remained: are these rules necessary ingredients that must be forced into the system, or are they natural features that would eventually grow on their own if the system were left to evolve? If we stripped away the human instructions and let a network grow from a chaotic, fully connected mess, would it naturally discover the elegant, organized structure of a convolutional network, or would it remain a tangled knot?

A researcher named Vasili Gavrilov set out to answer this by treating the network not as a machine to be tuned, but as a living organism to be evolved. He started with a "seed" network that was completely dense, meaning every possible connection between its parts existed, creating a tangled web with no structure at all. He then placed this network under a strict energy budget, a rule that charged a cost for every single connection it kept. The goal was to survive and solve a task while spending as little energy as possible. The researchers used a process similar to natural selection, where the network would randomly prune away connections that were not useful, and occasionally try to grow new ones, all while trying to maintain its ability to perform the task. By watching which parts of the network survived this pressure, they could map exactly which pieces of a convolutional network are imposed by human design and which pieces emerge naturally from the struggle for efficiency.

The experiment revealed that some parts of the network do indeed grow on their own. When the network was forced to save energy, it quickly learned to cut away useless connections and focus only on the inputs that mattered for the task. This is known as feature selection, and it happened cleanly and reliably. The network also naturally adopted the rule of weight sharing, where the same set of numbers is used to scan different parts of the input. This is the core of translation invariance, the ability to recognize a pattern regardless of where it appears. However, the researchers found that this efficiency came with a catch. While the network wanted to share weights, it could not figure out how to make the shared filter compact and aligned on its own. The filter remained wide and messy.

The breakthrough came when the researchers changed how the network was allowed to mutate. In the first attempts, the network could only cut or add one single connection at a time. But because the weights were shared, cutting one connection did not actually save any energy, since that same weight was still being used elsewhere. The network was stuck. The solution was to let the network mutate the entire shared feature at once, rather than just a single edge. This is similar to how a gene in biology affects every instance of a trait it builds, not just one cell. Once the researchers allowed the network to edit the whole shared group of connections as a single unit, the compact, tidy filter finally emerged. The network spontaneously tightened its focus, creating a small, efficient window that scanned the input, exactly like the filters humans have designed by hand.

The study also explored how these networks build depth and complexity. When the task required the network to see a larger area, the network naturally grew taller, stacking more layers to increase its view. However, it tended to overshoot, building more layers than strictly necessary. When the task required the network to combine two different types of patterns, the network struggled to figure out how to split its work. If the researchers simply gave the network a single label for the combined task, the network failed to organize itself into two specialized channels; it would either collapse into one or waste energy on a redundant third channel. It only succeeded in splitting its work when the researchers provided extra guidance, teaching it to find the first pattern and then the second separately before combining them. This revealed a hard boundary: while some structure emerges naturally, the ability to decompose a complex task into distinct parts does not appear without supervision.

Finally, the researchers compared their directed evolutionary search against a purely random approach. They found that the guided search, which used the energy budget to climb toward better solutions, consistently found cleaner, more efficient structures than random chance. It produced filters that were more organized and slightly more accurate. This confirmed that the energy budget provided a real path for the network to follow, rather than just a flat landscape where random luck would be just as good. The work does not claim to have solved the problem of building artificial intelligence, nor does it suggest these small, synthetic tasks will immediately translate to real-world vision. Instead, it offers a precise map of what nature-like evolution can and cannot achieve on its own. It shows that while the pressure of efficiency can grow sparsity and weight sharing, the specific, compact architecture of a convolutional filter requires a specific kind of mutation to unlock, and the ability to break a complex problem into independent parts often requires a teacher to guide the way.

Technical Summary: The Imposed and Emergent Pieces of Convolution Under an Energy Budget

Problem Statement
The paper addresses a fundamental question in deep learning architecture design: Which components of a convolutional neural network (ConvNet) must be imposed as inductive priors by human designers, and which can emerge naturally from an exhaustive search under an energy (connection) budget? Rather than using search to find a high-performing architecture, the author uses an evolutionary search as a "lens" to map the boundary between imposed structure and emergent structure. The goal is to determine if a fully connected, dense network can evolve into a sparse, convolutional structure solely through the pressure of minimizing connection costs while maintaining task accuracy.

Methodology
The study employs a Genetic Algorithm (GA) to evolve network topology from an exhaustive, fully-connected seed.

Evolutionary Framework: The network starts as a dense, all-ones connectivity mask. The objective is to minimize energy (the number of connections/parameters) subject to a constraint that validation accuracy remains above a target threshold.
Operators: The search utilizes biological analogs:
- Prune: Removes connections (downhill energy move).
- Clone/Translate: Duplicates blocks and tiles them with shared weights (weight tying).
- Recombine: Swaps in different block types (heterogeneity).
Mutation Mechanism: A critical distinction is made between mutating single edges versus mutating "shared features." In weight-sharing scenarios, removing a single connection does not reduce the parameter count if that weight is reused elsewhere. Therefore, the author introduces grouped mutation, where a mutation acts on the entire shared feature (offset), removing all duplicated edges simultaneously to restore the energy gradient.
Experimental Setup: Experiments are conducted on small, synthetic tasks (e.g., spike detection, motif recognition, two-operation composition) using C99 probes to ensure reproducibility. Results are averaged over independent seeds (typically 24–250) and compared against random search baselines and supervised ceilings.

Key Results

Emergence of Sparsity vs. Convolution:
- Under an energy budget, feature selection and sparsity emerge cleanly. The search prunes the dense seed to a sparse subnetwork where surviving connections land on task-relevant inputs.
- However, a compact, aligned local receptive field does not emerge from pruning alone. At comparable densities, surviving connections are no more contiguous than chance. The search finds a sparse subnetwork, not a convolution.
The Mechanism of Emergent Compact Filters (Sharing/Energy Decoupling):
- A compact filter only emerges if the mutation unit is the shared feature, not the single edge. When mutation acts on the whole shared offset (grouped mutation), the kernel contracts to the optimal size (e.g., $K=3$ ).
- This "sharing/energy decoupling" is the paper's primary mechanistic finding: a compact shared filter cannot emerge under per-connection pruning because sharing decouples connection count from parameter count. The energy gradient only exists when the mutation acts on the gene (the shared weight) globally.
Directed Search vs. Random Search:
- The directed energy GA finds "tidier" (more contiguous) filters and achieves modestly higher accuracy than random search at matched evaluation budgets.
- The structural gap (contiguity) is significant (6–8 standard errors of the mean), while the accuracy gap is smaller but clear. This confirms that directed search provides value when the objective (energy minimization) has a climbable gradient, reconciling previous null results on flat benchmarks like NAS-Bench-101.
Depth and Reuse:
- Depth: Emergent depth grows with the required receptive field but consistently overshoots the theoretical minimum.
- Reuse: Reusing a single block type (stacking) is near-optimal for tasks requiring a single operation type. However, for tasks requiring multiple distinct operations (e.g., a fine local motif and a far-reaching spike), recombination (mixing different block types) is required. Uniform reuse fails on these heterogeneous tasks.
The Composition Boundary (Channel Axis):
- On a two-operation task (requiring detection of two distinct motifs), the search fails to cleanly discover the decomposition without supervision.
- The energy budget selects a channel count that overshoots the requirement (e.g., selecting 3 channels for a 2-operation task), and the channels often fail to specialize (collapsing onto one motif).
- In 2D tasks requiring both horizontal and vertical filters, a "capacity wall" appears at $K=3$ operations. Four interventions (more channels, nonlinear readout, lateral inhibition, deeper extractors) fail to move this boundary, suggesting a structural limit to emergent composition on the channel axis without per-operation supervision.
Weight Sharing Efficiency:
- Weight sharing is confirmed as highly data-efficient. A single tiled block with shared weights significantly outperforms locally connected baselines, especially when data is scarce. This advantage grows with input size and saturates, independent of the evolutionary search mechanism.

Significance and Claims
The paper claims its contribution is the decomposition of convolution into imposed priors and emergent pieces, along with a specific mechanism explaining why compact filters emerge only under specific mutation conditions.

The Map: The author provides a boundary map of what emerges (sparsity, feature selection, depth trends) and what does not (compact filters without grouped mutation, clean compositional decomposition without supervision).
The Mechanism: The "sharing/energy decoupling" explains that compact filters emerge only when mutation acts on the shared feature (the gene) rather than a single edge, restoring the energy gradient necessary for the filter to tighten.
The Negative: The paper explicitly reports the failure of emergent composition on the channel axis. It argues that without supervision, the search does not naturally discover the correct number of specialized components for multi-part tasks, often overshooting and under-specializing.
Modesty: The author emphasizes that the tasks are synthetic and small-scale. They do not claim the method scales to real-world vision or audio directly. Instead, they position the work as a "characterization" tool that reveals the limits of what energy budgets can achieve, distinguishing between what must be engineered (priors) and what can be evolved. The results verify textbook inductive bias facts (e.g., weight sharing efficiency) against fair baselines rather than claiming them as new discoveries.

Technical Summary: The Imposed and Emergent Pieces of Convolution Under an Energy Budget

More like this