Autonomous Collaborative Learning Among an Ensemble of Tsetlin Machines with Consensus-Based Inference

Technical Summary: Autonomous Collaborative Learning Among an Ensemble of Tsetlin Machines with Consensus-Based Inference

Problem Statement
The Tsetlin Machine (TM) is a rule-based machine learning algorithm grounded in propositional logic and Tsetlin Automata (TAs), offering computational simplicity and energy efficiency suitable for resource-constrained edge devices. While recent studies have explored Federated Learning (FL) with TMs, existing literature predominantly focuses on centralized server architectures (Horizontal or Vertical FL) or assumes homogeneous agent capabilities. There is a significant gap in the exploration of decentralized, distributed TM learning, particularly in vertical feature-partitioning settings where agents hold disjoint feature subsets of the same data instances. Furthermore, standard FL approaches often require exchanging model updates or gradients, which can be computationally expensive or privacy-invasive. The challenge lies in designing a framework where heterogeneous agents can collaboratively learn and reach a global consensus without exchanging raw data or revealing their internal model parameters, while operating under strict energy and communication constraints typical of sensor networks.

Methodology
The authors propose a decentralized collaborative learning paradigm utilizing a two-layer hierarchical TM architecture paired with a gossip-based communication protocol.

1. Architecture:

   • Input Layer: Each agent maintains a local TM that operates exclusively on its private, local Boolean feature subset. This layer learns to recognize patterns specific to the agent's local data partition.
   • Neighborhood Aggregation Layer: A second TM layer resides at each agent. Its input consists of the clause outputs from the agent's own Input Layer TM combined with the clause outputs received from its immediate neighbors in the network topology. This layer learns to detect higher-level, global patterns emerging from the local cluster of agents.
   • Privacy Preservation: Only binary clause outputs (single bits) are exchanged between neighbors. The internal structure of the clauses, the automata states, and the raw data remain strictly local and private.

2. Communication Protocol:

   • The system employs an asynchronous gossip-based protocol. Agents broadcast their clause outputs to immediate neighbors without a central coordinator.
   • For inference, a multi-source, all-to-all dissemination protocol is used to propagate class vote sums across the network, allowing every agent to eventually compute a globally consistent decision (consensus) based on the aggregated votes of the entire ensemble.

3. Training Workflow:

   • Training occurs in two stages per iteration. First, the Input Layer TM is trained on local data. Second, the Neighborhood Aggregation Layer TM is trained using the clause outputs from the local Input Layer and those received from neighbors.
   • In multi-class scenarios, agents maintain pairs of TMs (positive and negative clauses) for each class, updating them sequentially based on ground-truth labels and alternative class labels.

Key Contributions

• Decentralized Vertical FL Framework: The paper introduces the first known hierarchical TM framework designed for decentralized learning under vertical feature partitioning with aligned samples, eliminating the need for a central server.
• Heterogeneity Support: The paradigm accommodates heterogeneous agents with differing data acquisition means, local data distributions, and computational resources, facilitating integration in multi-modal sensing environments.
• Efficient Communication: By exchanging only binary clause outputs via a gossip protocol, the method significantly reduces communication overhead and preserves the confidentiality of both model parameters and local data.
• Consensus-Based Inference: The system achieves global consensus through local peer-to-peer exchanges, allowing the ensemble to recognize complex patterns that isolated local models cannot capture.

Experimental Results
The authors evaluated the framework using software implementations on three benchmarks:

1. MNIST and Fashion-MNIST:

   • Experiments involved partitioning 28x28 images into tiles assigned to agents in a 2D grid (e.g., 49 agents with 4x4 tiles, or 16 agents with 7x7 tiles).
   • Results: The two-layer hierarchical architecture consistently outperformed single-layer approaches. On MNIST, the ensemble achieved accuracies up to 94.65% (16 agents, 7x7 tiles), approaching the performance of a classic centralized TM (96.73%). On Fashion-MNIST, the ensemble reached 83.99%, comparable to centralized baselines.
   • The results demonstrated that the aggregation layer effectively learns the activity patterns of clauses within the cluster, providing superior discrimination compared to simple voting-sum aggregation.

2. Synthetic Heterogeneous Sensor Network:

   • A network of 25 agents with heterogeneous sensor data (varying scales, offsets, noise levels, and sensor counts) was simulated.
   • Baseline Comparison: A centralized Artificial Neural Network (ANN) trained on aggregated data achieved 93% accuracy, while individual agents trained on local data achieved only 43–57.5% accuracy due to data overlap and noise.
   • TM Performance: The decentralized TM ensemble achieved 93% accuracy, matching the centralized ANN.
   • Scalability: Accuracy was shown to converge toward the centralized baseline as the number of participating agents increased.
   • Topology Robustness: The framework was also tested on a general connected graph topology (non-grid) with 9 agents having varying neighbor counts. The system achieved 76% accuracy, comparable to the centralized ANN and the grid-based configuration with 9 agents.

Significance and Claims
The paper claims that this work establishes a viable paradigm for decentralized, privacy-preserving machine learning in edge and sensor network environments. By leveraging the logic-based, low-power nature of Tsetlin Machines and a consensus-driven inference mechanism, the proposed framework enables:

• Collaborative Intelligence: Agents can collectively solve complex classification tasks that exceed the capabilities of individual local models without sharing raw data.
• Resource Efficiency: The approach is tailored for low-power compute environments, avoiding the heavy arithmetic and communication costs associated with traditional deep learning federated learning.
• Scalability and Flexibility: The system functions effectively across different network topologies (grids and general graphs) and supports heterogeneous agents, making it suitable for real-world multi-modal sensing scenarios.

The authors note that the current evaluation assumes ideal, error-free communication channels and that future work will address realistic network impairments such as packet loss and latency.