🤖 machine learning

On the Depth Scalability of Logic Gate Networks

This paper introduces Input-Anchored Logic Gate Networks (IALGNs), a novel architecture that overcomes the depth scalability limitations of traditional Logic Gate Networks by anchoring each layer to the original input, thereby enabling stable optimization and consistent accuracy improvements across networks exceeding 100 layers.

Original authors: Taegun An, Dohun kim, Haebeom Lee, Changhee Joo

Published 2026-07-27

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Taegun An, Dohun kim, Haebeom Lee, Changhee Joo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a super-smart robot brain, but instead of using the usual squishy, floating-point numbers that most computers use, you decide to build it entirely out of tiny, on/off light switches. This is the world of Logic Gate Networks (LGNs). Think of these networks as a massive, intricate maze of light switches where each switch can only be "on" (1) or "off" (0). In the real world, these are the fundamental building blocks of digital circuits, the kind that run your calculator or your video game console. For decades, scientists have known a secret rule about these circuits: the deeper you build the maze (adding more layers of switches), the smarter and more powerful the brain can become. It's like adding more floors to a skyscraper; theoretically, you should be able to fit more offices and do more complex work.

However, there's a catch. When researchers tried to train these deep, digital brains using modern computer learning techniques, they hit a wall. No matter how many layers they added, the brain didn't get smarter. In fact, it often got dumber. It was as if they were building a 100-story skyscraper, but every time they added a new floor, the elevator stopped working, and the people on the top floors couldn't talk to the ground floor. The big question was: Is the problem that the construction crew (the training algorithm) is too clumsy to build deep towers, or is there something wrong with the blueprint itself?

This paper, titled "On the Depth Scalability of Logic Gate Networks," dives right into that mystery. The authors, a team from Korea University, discovered that the problem wasn't just the construction crew. Even when they fixed the training methods to make sure the "elevator" worked perfectly, the deep towers still failed to get smarter. They realized the blueprint was missing a crucial feature: the upper floors were completely cut off from the original raw materials. To fix this, they invented a new design called Input-Anchored Logic Gate Networks (IALGNs). Instead of letting every floor only talk to the floor immediately below it, they added a special "anchor" cable that runs straight from the ground floor (the original input) to every single floor in the building. This simple change allowed the deep layers to keep refining their work using the original information, rather than just guessing based on what the floor below them said. The result? These new, anchored networks actually do get smarter as they get deeper, consistently improving their accuracy on tasks like recognizing images from the MNIST, CIFAR-10, and CIFAR-100 datasets, even when they stretch over 100 layers deep.

The Problem: The "Telephone Game" of Deep Brains

To understand why the old designs failed, imagine a game of "Telephone" (or "Chinese Whispers"). You whisper a message to the person next to you, who whispers it to the next, and so on. In a standard deep Logic Gate Network, every layer of the brain only receives information from the layer directly above it. If you have 100 layers, the 100th layer is trying to understand the original image based only on what the 99th layer whispered to it.

The authors found that as the chain gets longer, the message gets garbled. The 99th layer might have already summarized the image into a very abstract concept, and by the time it reaches the 100th layer, the original details are lost. The network ends up just re-mixing the same old, blurry summary over and over again, rather than learning something new.

For a long time, scientists thought the problem was just that the "whispering" was too quiet. They thought the signal was dying out before it reached the top. They tried to fix this with better "megaphones" (techniques like skip-biased initialization and straight-through estimators) to make sure the signal stayed loud. And guess what? It worked! The signal stayed loud all the way to the top. But here's the twist: even with a loud signal, the deep networks still didn't get smarter. The 100th layer was shouting a loud, garbled message, but it was still just a garbled message. The problem wasn't the volume; it was the content. The top layers were still cut off from the original source.

The Solution: The "Direct Line" Blueprint

The authors realized that to make a deep network work, every single layer needs a direct line back to the original input. They call this the Input-Anchored design.

Imagine you are building a tower of blocks. In the old design, each new block could only look at the block directly underneath it. If the block below was a bit wobbly or had lost its shape, the new block would just copy that wobble. In the new IALGN design, every block has two inputs:

The Spine: It looks at the block directly underneath it (the hidden feature from the previous layer).
The Anchor: It has a direct cable running all the way down to the very first block (the original input).

This means that even the 100th layer can still "see" the original image clearly, while also looking at what the 99th layer has figured out so far. It's like having a team of editors working on a story. In the old system, Editor 100 only gets to read what Editor 99 wrote. If Editor 99 made a mistake or forgot a detail, Editor 100 is doomed. In the new system, Editor 100 can read Editor 99's draft and go back to read the original notes from the author. This allows them to fix mistakes and add new, useful details that the previous editor missed.

What They Found: Proof in the Numbers

The team tested this idea on three famous image datasets: MNIST (simple handwritten numbers), CIFAR-10 (small pictures of 10 different objects like cats, dogs, and airplanes), and CIFAR-100 (the same but with 100 different, harder-to-tell-apart objects).

They built networks with the same width (number of switches per layer) but varied the depth (number of layers).

The Old Way (Randomly Wired): When they made these networks deeper, the accuracy didn't go up. In fact, for the harder CIFAR-100 dataset, making the network 150 layers deep actually made it perform worse than a shallow one. The extra layers were just adding noise.
The New Way (Input-Anchored): When they used the IALGN design, the deeper the network got, the smarter it became. On CIFAR-10, a shallow network got about 53% accuracy, but a 100-layer deep network hit 56.41%. On CIFAR-100, the deep network reached 30.60% accuracy, a steady climb that the old networks couldn't match.

The authors didn't just stop at the final score. They peeked inside the "brain" to see what was happening. They froze the network at different layers and asked, "How much information is in this layer?" They found that in the new IALGNs, the information kept getting richer and more useful as you went deeper. In the old networks, the information got stuck or degraded.

They also proved that this wasn't just because the new design had "more freedom" to connect wires. They tested a version where they gave the old networks more freedom to connect to other hidden layers, but without the direct anchor to the input. That didn't help. The secret sauce was specifically the anchor to the original input.

Why This Matters

This paper suggests that for digital, logic-based brains, simply making them deeper isn't enough. You have to design them so that every part of the brain can still "see" the original problem. It's a reminder that in complex systems, connectivity matters just as much as size. You can't just stack layers on top of each other and hope for the best; you need to make sure the top layers have a direct line to the bottom.

The authors show that with this simple "anchor" trick, we can finally build Logic Gate Networks that are truly deep, scalable, and capable of handling complex tasks like recognizing objects in pictures. It's a small change in the blueprint, but it turns a tower that collapses under its own weight into a skyscraper that keeps growing taller and smarter.

Technical Summary: On the Depth Scalability of Logic Gate Networks

Problem Statement

Logic Gate Networks (LGNs) are neural architectures that perform computation through compositions of Boolean operations, offering a structured and interpretable alternative to real-valued neurons. While Boolean circuit theory suggests that increasing circuit depth strictly enlarges representational power, existing LGNs fail to exhibit reliable depth–performance scaling in practice. Most reported architectures remain shallow, and increasing depth frequently leads to accuracy saturation or degradation.

The prevailing explanation attributes this failure to optimization collapse, where gradients vanish as they propagate through many relaxed Boolean operations. While recent advances—such as skip-biased initialization and straight-through estimators (STE)—have largely stabilized optimization in deep LGNs, a critical gap remains: even with stable optimization, randomly wired LGNs (RWLGNs) do not benefit from increased depth. This paper posits that depth scalability requires satisfying two distinct conditions:

Optimization: The ability to train deep networks successfully.
Topology: A network structure that enables additional layers to perform useful computation rather than merely recombining existing hidden representations.

The authors argue that existing methods address the former but leave the latter unaddressed, leading to a topology-induced limitation where deeper layers fail to access the information necessary for progressive refinement.

Methodology: Input-Anchored Logic Gate Networks (IALGNs)

To address the topological limitation, the authors propose Input-Anchored Logic Gate Networks (IALGNs). This architecture introduces a minimal modification to the standard LGN topology:

Dual-Input Structure: Each logic gate in an IALGN receives two inputs:
1. Spine Pin: A feature propagated from the previous hidden layer, preserving a continuous computational path through the network.
2. Anchor Pin: A feature sampled directly from the original binarized input.
Mechanism: Unlike RWLGNs, where gates only compose features from the preceding layer, IALGNs allow every layer to refine accumulated computations by incorporating new, direct information from the original input. This creates an "input-conditioned refinement" process.
Random-k Anchor Relaxation: To improve flexibility without combinatorial search, the authors introduce a relaxation where each gate samples $k$ candidate input anchors and learns a relaxed selection among them. This preserves the input-anchoring principle while allowing the network to select the most informative input variables.
Training: IALGNs utilize the same differentiable Boolean-operation relaxations, loss functions, and optimization strategies (e.g., skip-biased initialization, STE) as standard DLGNs. The only architectural change is the source of one gate input.

Theoretical Analysis

The paper establishes a theoretical foundation for why this topology enables depth scaling:

Input Dependency (Proposition 1): In an IALGN, a computational path of depth $D$ depends on at most $D+1$ distinct input bits. This bound is tight.
Strict Depth Hierarchy (Theorem 1): For any integer $D \geq 1$ , there exists a Boolean function computable by a depth- $D$ path in an IALGN that cannot be computed by any depth- $(D-1)$ path. Specifically, the architecture can represent parity functions over $D+1$ bits at depth $D$ , whereas shallower paths are limited to $D$ bits.
Implication: This proves that increasing depth in IALGNs strictly enlarges the class of representable Boolean functions for individual paths, distinguishing the computational capability of the architecture from the trainability provided by optimization techniques.

Experimental Results

The authors evaluated IALGNs against RWLGNs and re-implemented LDLGNs across MNIST, CIFAR-10, and CIFAR-100, focusing on fixed-width depth scaling (increasing depth while keeping layer width constant).

Depth Scaling:
- RWLGNs: Performance saturates or degrades significantly as depth increases beyond 10–20 layers, even with stable optimization (Skip init + STE).
- IALGNs: Exhibit consistent accuracy improvements as depth increases, reaching stable performance gains even at depths of 100+ layers. On CIFAR-10, IALGNs improved from ~53.6% (depth 4) to ~56.5% (depth 150), whereas RWLGNs dropped to ~47.9%.
Linear Probes: Layer-wise linear probes on frozen hidden representations showed that IALGNs produce progressively more informative representations as depth increases. In contrast, RWLGN representations deteriorated or stagnated in deeper layers.
Activation Entropy: IALGNs achieved higher probe accuracy across comparable activation entropy levels, indicating that the improvement stems from richer computation rather than mere changes in activation statistics.
Ablation Studies:
- Replacing input anchoring with "hidden-k" relaxation (increasing flexibility in hidden-to-hidden connections) did not yield similar depth scaling benefits and often degraded performance at greater depths.
- This confirms that the gain arises from the input-anchored inductive bias rather than generic wiring flexibility.
Effective Depth: Analysis of effective computational paths revealed that IALGNs preserve longer chains of non-trivial Boolean operations throughout the network, whereas RWLGNs tend to collapse into skip-like computations.

Significance and Claims

The paper claims that scalable depth in Logic Gate Networks requires solving two distinct problems: optimization collapse and a topology-induced limitation in progressive representation refinement.

Optimization is Necessary but Insufficient: The authors demonstrate that stabilizing optimization (via STE and skip-biased initialization) is not enough to make additional depth useful; the network topology must also support the accumulation of new information.
Topology Enables Computation: By anchoring every layer to the original input, IALGNs ensure that deeper layers perform input-conditioned refinement rather than redundant recombination. This induces a strict path-wise depth hierarchy where increased depth strictly expands computational capability.
Empirical Validation: The consistent fixed-width depth–accuracy improvements across multiple datasets demonstrate that the proposed topology effectively converts increased depth into improved predictive performance.

The authors conclude that realizing the value of depth in discrete neural architectures requires the joint design of optimization strategies and information-access patterns. They position IALGNs as a step toward architectures where topology is designed not only for trainability but for progressively expanding computation across depth.