Multi-Scale Structural Features for Continual, Comprehensible Visual Recognition in a Developmental Learning Framework

Technical Summary: Multi-Scale Structural Features for Continual, Comprehensible Visual Recognition

Problem Statement
Contemporary machine learning faces a fundamental triad of limitations: the inability to learn continually without overwriting past knowledge (catastrophic forgetting), the difficulty in reusing prior knowledge effectively, and the lack of comprehensible internal structures. Gradient-based neural networks struggle with this by construction; their shared weight space means updates for current data inevitably degrade competence on past data ("destructive adaptation"). Standard remedies like replay buffers, regularization, or task boundaries mitigate symptoms but rely on assumptions (retained data, known boundaries, routing signals) that do not exist in a genuine, non-stationary stream of experience.

While a previously proposed developmental, gradient-free framework (the "Modeller") addressed these issues by growing discrete, topological models through local variation and selection, its extension to visual inputs was limited. The prior visual instantiation relied on a minimal feature representation that capped recognition accuracy at approximately 0.50 on ten-class MNIST, failing to capture the necessary structural complexity for high-performance recognition.

Methodology
The paper introduces a developmental, gradient-free learning framework that processes visual inputs as relational networks. The system learns by refining these networks through local variation and selection, integrating each observation exactly once without storing past data or requiring task boundaries.

1. Multi-Scale Structural Feature Representation:
   The core contribution is a new visual feature representation that encodes shape structure across multiple scales simultaneously.

   • Principle: Regions of constancy are represented by their points of change. For 2D shapes, the system extracts contours from binarized images and identifies directional extrema (points where traversal direction changes sign along image axes).
   • Network Construction: These change points become typed nodes (defined by axis, direction, and local convexity). They are connected by directed edges representing contour continuity and spatial relations (horizontal/vertical).
   • Coarsening: The network is iteratively coarsened by removing the smallest adjacent maximum/minimum pairs of the same type. Crucially, edges from all levels of coarsening are accumulated into a single augmented network. Nodes are shared across levels, meaning long-range relations (spanning distant parts of the shape) and local relations coexist over the same node set. This allows the learner to retain reliable relations at any granularity while reducing non-recurring details.

2. Learning Process (Refinement and Selection):

   • Observation as Network: An input image is converted into a state network (a directed graph of feature instances and relations).
   • Variation: If a target (class or internal state) is active but unaccounted for, a new positive conditioning state variable (csv) is created, using the observed network as its source.
   • Selection (Refinement): Learning reduces the source of a csv toward what recurs. When a source matches a new observation where the target recurs, elements of the source absent in the new observation are removed. Because refinement only removes requirements and never adds them, the system guarantees that any observation satisfying a csv before a modification will still satisfy it afterwards.
   • Differentiation: If a csv matches but its target contradicts the observation, or if a partial match leaves a residual, new "upstream" csvs are spawned to carve out the specific residual, forming chains of progressively more specific structures.
   • Maintenance: Unreliable csvs are removed, and highly reliable ones are reintegrated into their downstream targets to prevent structural bloating.

3. Matching and Read-Out:

   • Chain-Aware Matching: The system locates where a learned pattern sits within an observation. Unlike prior methods that relied on positional proximity, this matcher uses the multi-scale relations. It grows correspondences incrementally, constrained by both local geometry and long-range spatial relations that span the shape. This makes the correspondence problem well-determined.
   • Absence-Aware Read-Out: Classification is performed by a local, gradient-free procedure that consumes the spatial statistics (positions and orientations) computed during matching. It treats the absence of an expected csv as evidence against a class. The score for a class is the sum of log-likelihoods derived from the presence of matching csvs and the absence of expected ones, weighted by their geometric fit to class-specific anchors versus a background pool.

Key Contributions

• Multi-Scale Representation: A unified relational network that captures shape structure at all levels of a coarsening hierarchy simultaneously, enabling the learner to retain long-range relations even as local details are refined away.
• Improved Learning Dynamics: An explicit account of the matching step and a local, absence-aware read-out mechanism that leverages learned spatial statistics without global classifiers or gradients.
• Demonstrated Continual Learning: The system learns all ten classes of MNIST from a replay-free, boundary-free stream, strictly sample-by-sample, achieving 0.874 accuracy.
• Comprehensibility: The learned models are explicit configurations of change points and relations that can be drawn and interpreted as caricatures of the classes.

Results
Evaluated on class-incremental MNIST (59,595 topologically consistent images), the system achieved the following:

• Accuracy: 0.874 (±0.018) held-out accuracy, a substantial increase from the prior 0.50 reported for the same framework.
• Retention: The system retains 95% of a class's accuracy at the end of the cycle it was trained on (WCR_end) and 90% at its lowest point (WCR_wst). In contrast, gradient-based baselines (even with replay) drop to near zero or require relearning the class within the same cycle.
• Comparison: The system matches or exceeds the performance of replay-based (e.g., DER++, Naive Replay) and regularization-based (e.g., SI, MAS) baselines at comparable storage, despite storing no past data.
• Model Growth: The model size saturates at approximately 187 conditioning state variables (csvs) after the initial learning phase, demonstrating that the system does not grow unboundedly. Negative structure (suppressors) eventually dominates positive structure, indicating that boundaries are where discrimination is maintained.

Significance and Claims
The paper claims that this approach demonstrates a qualitative capability outside the reach of statistical, gradient-based learning: the ability to integrate information one sample at a time while provably preserving responses to past observations.

• No Destructive Adaptation: Unlike neural networks that overwrite past competence, this system refines existing structure locally, ensuring earlier-learned classes are retained as new ones are introduced.
• Inherent Comprehensibility: The learned representations are human-interpretable structural descriptions rather than opaque weight vectors.
• Foundational Principle: The work validates the principle that a continuous entity can be represented by its points of change, and that multi-scale structural relations can be expressed as a single relational network for a structural continual learner.
• Limitations: The authors acknowledge the system is currently restricted to 2D shapes (MNIST) and relies on a hand-designed feature vocabulary (first-order change points on binarized contours). The 4/9 confusion remains a representational residual. The approach presupposes a representation rather than learning it, though the principle is argued to be universal.

The significance lies not merely in the accuracy, but in the manner of learning: a system that learns from a stream without replay, gradients, or task boundaries, yet retains what it learns and remains comprehensible.