Technical Summary: Rank-Order N-of-M Codes for Sparse Distributed Memory

Problem Statement

Large Language Models (LLMs) struggle with continual learning, as incorporating new knowledge typically requires expensive retraining, and Retrieval-Augmented Generation (RAG) relies on external vector stores not designed for online associative memory. The CALM architecture (Nechesov and Ruponen, 2025) addresses this by integrating Sparse Distributed Memory (SDM) as an explicit episodic memory layer. However, CALM utilizes a threshold-binary encoder (binarizing embeddings against their mean) and explicitly identifies this as a limitation, noting that improved encoding schemes are a necessary future direction.

The core engineering question this paper addresses is: What encoding scheme should map continuous neural activations into the SDM address space to maximize retrieval robustness under noise? Specifically, does Rank-Order N-of-M coding (Furber et al., 2007)—which preserves relative magnitude information via geometrically weighted significance values—offer a tangible advantage over the threshold-binary approach used in CALM?

Methodology

The paper employs a rigorous experimental framework comparing three SDM variants and two encoding schemes across multiple dimensions:

1. Architectural Validation: The authors implemented four SDM variants (StandardSDM, RankOrderSDM, WheelSDM, and RDLIFSDM) based on the specifications in Furber et al. (2007) and Bose et al. (2005). They tested these across 10 random seeds to verify if the biologically motivated "WheelSDM" (spiking neurons) is numerically equivalent to the abstract "RankOrderSDM" and to confirm the documented failure of RDLIF neurons under interference.
2. Capacity Experiments: Memory capacity was measured at saturation points ($n=80, 120, 160$ patterns for $W=256, D=64$) and at the published default scale ($W=4096, D=256$).
3. Noise Robustness (BER) Decomposition: To disentangle the effects of encoding from learning rules, the authors conducted two distinct Bit Error Rate (BER) experiments:
   • Full Architecture Test: Compared Rank-Order SDM (using MAX-Hebbian learning) against Standard SDM (using accumulative bipolar-counter writing) under Gaussian magnitude perturbation.
   • Symmetric Perturbation Test: Applied identical Gaussian noise to the continuous activation vector, then re-encoded the noisy vector using both threshold-binary and rank-order schemes. This isolated the encoding-only advantage by holding the learning rule (accumulative SDM) constant.
4. Geometric Stability Analysis: Measured Kendall tau (rank correlation) and Jaccard overlap (active-set membership) to quantify how rank ordering degrades compared to binary thresholding under noise.
5. Energy Comparison: Calculated component-level encoding energy for Rank-Order coding versus SpikingMamba's SI-LIF neurons (Huang et al., 2026) using 45nm energy constants, while noting that system-level decoder costs dominate the total energy budget.
6. Real-World Data Validation: Pilot experiments were conducted on GloVe-100 word embeddings and all-MiniLM-L6-v2 sentence embeddings to verify findings on structured, non-Gaussian data.

Key Contributions and Results

1. Validation of Foundational Claims

The reimplementation confirmed that WheelSDM and RankOrderSDM are exactly equivalent, yielding a cosine similarity of 1.0000 across all seeds. Conversely, RDLIF neurons collapsed completely (0 correct recalls) under interference, validating the 2005 papers' decision to switch to the wheel model.

• Capacity: At saturation ($n=160, W=256$), RankOrderSDM outperformed StandardSDM by 13.4 percentage points (p < 0.001, Cohen's d = 4.61). At the larger scale ($W=4096$), the advantage was smaller (+0.8pp) but statistically present.

2. Disentangling Encoding from Learning (The Core Finding)

The paper identifies a critical distinction between the combined advantage of the full architecture and the isolated advantage of the encoding scheme:

• Combined Advantage: In the full architecture test (Rank-Order + MAX-Hebbian vs. Binary + Accumulative), Rank-Order encoding maintained perfect recall up to 20% noise, outperforming binary encoding by +38 to +66 percentage points at higher noise levels.
• Encoding-Only Advantage: When the learning rule was held constant (Accumulative SDM), the advantage of Rank-Order encoding over threshold-binary was marginal. Under symmetric perturbation, the improvement was +0.1 to +0.9 percentage points, reaching statistical significance only at specific noise levels (e.g., BER=0.25, p=0.036).
• Conclusion: The massive robustness gain (+66pp) observed in the full architecture is driven primarily by the MAX-Hebbian learning rule, not the encoding scheme alone. The encoding scheme itself provides a small, consistent, but modest geometric stability benefit.

3. Geometric Stability

Rank-order encoding demonstrates superior geometric stability compared to binary thresholding. At a noise level of 0.10, rank ordering maintained a Kendall tau of 0.847 and Jaccard overlap of 0.891. While binary bit-flip rates were low (0.020), the paper argues that rank degradation is "smooth and partial," whereas binary thresholding causes "sharp and discontinuous" address changes, making rank-order more robust to the specific nature of query noise in continual learning.

4. Energy and Real-Data Findings

• Energy: At the component level, Rank-Order encoding (assuming ideal parallel sort hardware) is 2x more energy-efficient than SpikingMamba's SI-LIF at $D=4$. However, system-level costs (address decoder traversal) dominate the architecture, making encoding optimization a secondary priority.
• Real Data: On GloVe-100 embeddings, the encoding-only advantage mirrored synthetic results (+0.1 to +0.4pp, significant at higher noise). On sentence embeddings (MiniLM), a ceiling effect was observed where all schemes achieved >0.999 recall due to low memory load, masking encoding differences.

Significance and Claims

The paper claims three primary contributions that offer actionable guidance for developers of memory-augmented AI systems like CALM:

1. Validation: It confirms that the theoretical properties of the 2007 Rank-Order SDM architecture hold up under modern reimplementation, specifically the equivalence of the wheel-model spiking neuron and the abstract rank-weighted dot product.
2. Mechanism Decomposition: It clarifies that the robustness of Rank-Order SDM is not solely due to the representation (encoding) but is heavily dependent on the MAX-Hebbian learning rule. For CALM developers, this implies that simply swapping the encoder yields only marginal gains; to achieve the full robustness potential, both the encoder and the write rule (replacing accumulative counters with MAX-Hebbian updates) must be changed.
3. Design Guidance: The paper argues that representation design is as critical as memory architecture. While the encoding-only advantage is modest (+1-5pp), it is consistent and directionally correct, suggesting that preserving relative magnitude information (via rank-ordering) is superior to simple thresholding for online learning scenarios where embeddings are continuously perturbed.

The authors conclude that while Rank-Order N-of-M codes offer a concrete, theoretically sound alternative to threshold-binary encoding, the learning rule is the dominant factor in noise robustness. The most immediate actionable step for systems like CALM is to adopt rank-order encoding, optionally coupled with MAX-Hebbian learning, to improve retrieval stability under continuous embedding updates.