🤖 machine learning

Single State Update Predictive Coding training for Time Series Forecasting and Anomaly Detection

This paper introduces a parallel training technique for Predictive Coding Networks that pairs a Generative PCN with an Encoding PCN to eliminate sequential backward error propagation, thereby enabling stable, continuous online learning for time series forecasting and anomaly detection.

Original authors: Matteo Cardoni, Sam Leroux

Published 2026-08-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Matteo Cardoni, Sam Leroux

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, machines are often taught to learn by looking at a finished product and working backward to understand how it was made. This method, known as backpropagation, is the standard engine behind modern deep learning, but it has a significant flaw: it requires the computer to process information in a strict, step-by-step sequence. Imagine a line of people passing a message from the back of the room to the front; if the person at the back changes their mind, the message must travel all the way back through the line again to correct the error. This sequential nature makes the process slow and difficult to run on small, portable devices that need to learn in real time. A different approach, called predictive coding, offers a way out. Instead of working backward, this paradigm suggests that the brain and machines learn by constantly guessing what will happen next and then adjusting their internal state only when reality differs from that guess. The goal is to make these adjustments happen all at once, across the entire system, rather than waiting for a signal to travel from one end to the other.

Researchers Matteo Cardoni and Sam Leroux at Ghent University have developed a new training technique that brings this parallel learning concept to life for time series data, such as video streams or sensor readings. Their work addresses a persistent bottleneck in predictive coding networks: even though the theory allows for parallel updates, the practical implementation often still gets stuck waiting for error signals to propagate sequentially from the output layer back to the input. To solve this, the team introduced a method that pairs two neural networks together. One network, acting as a generator, tries to predict the next moment in a sequence based on the previous one. The second network, acting as an encoder, observes the actual incoming data. Instead of forcing the generator to learn alone through a slow, iterative process, the encoder runs alongside it, matching its internal activity. This partnership allows the system to update its entire internal state in a single, unified step, rather than cycling through multiple rounds of correction.

The researchers tested this approach by creating a simulated environment where a digit from a standard set of images moved across a black background. They tasked a five-layer neural network with predicting the next frame of this movement. Under normal conditions, the digit moved diagonally, but the researchers introduced sudden anomalies, such as the digit abruptly changing direction, bouncing off walls, or even switching to a completely different number. The system had to learn these new patterns on the fly, treating the new behavior as normal once it was observed. In these experiments, the traditional method, which relies on multiple sequential steps to correct errors, struggled when the pattern changed. It often failed to adapt quickly, causing its predictions to drift apart from reality. In contrast, the new paired-network approach recovered almost immediately. When the movement pattern shifted, the system adjusted its internal representation in a single update and began predicting the new trajectory accurately, maintaining a steady and continuous learning curve.

The results showed that this single-state update technique provided a much more stable learning process. While the traditional method produced erratic spikes in error whenever the data changed, the new approach smoothed out these transitions, allowing the model to treat the new dynamics as normal without losing its footing. The study suggests that by using a supporting network to guide the learning process, the system can bypass the slow, sequential propagation of errors that typically limits these types of models. This finding is particularly significant for the future of artificial intelligence on edge devices, such as smartphones or sensors, where power and processing speed are limited. By enabling machines to learn continuously and in parallel, this technique offers a promising path toward systems that can adapt to a changing world in real time, without the heavy computational cost of waiting for a long chain of corrections to finish.

Technical Summary: Single State Update Predictive Coding for Time Series Forecasting

Problem Statement
Predictive Coding (PC) is a neural learning paradigm offering a low-complexity alternative to Backpropagation, distinguished by its ability to perform parallelizable layer-wise updates. However, a significant bottleneck remains in Predictive Coding Networks (PCNs): the requirement for sequential error propagation from the output layer back through all layers. While PCNs are often initialized via feedforward methods to accelerate convergence, this approach necessitates a number of inference steps equal to or greater than the network depth. This sequential dependency leads to vanishing update propagation and hinders the efficiency of online learning, particularly in time-series applications where rapid adaptation is critical.

Methodology: Guided PC Training
To tackle the sequential update bottleneck, the authors propose a "Guided PC" training technique that pairs two distinct networks: a Generative PCN (G-PCN) and an Encoding PCN (E-PCN). These networks are trained in parallel to match their neural activations, thereby overcoming the requirement for multiple state update iterations without sequential propagation.

The training process for a single time step $t$ involves four unified steps:

Parallel Feedforward Initialization:
- The G-PCN (acting as a decoder) maps the previous state $h_{t-1}^0$ to the current input representation $\mu_t^0$ via a temporal mapping layer ( $\theta_0$ ), subsequently predicting the next time-step data $\hat{x}_t$ .
- The E-PCN (acting as an encoder) takes the actual input $x_t$ and performs a standard feedforward pass to generate activations $\gamma_t$ .
Parallel State Update (Guiding & Internal Energy):
- Both networks update their states in parallel to minimize two energy functions:
  - Guiding Energy ( $G$ ): The squared difference between the activations of the two networks ( $\sum (\mu_l - \gamma_l)^2$ ).
  - Internal Energy ( $I$ ) at the output layer: The squared difference between the G-PCN's prediction and the actual input ( $(\hat{x}_t - x_t)^2$ ).
- This step ensures that all layers in both networks receive non-zero updates simultaneously.
G-PCN Internal State Refinement:
- The G-PCN performs a classic "Vanilla" PC state update to minimize its Internal Energy ( $I = \sum (h_l - \mu_l)^2$ ) across all layers. Because the states were already updated in Step 2, this step further refines the G-PCN's representation using the actual input $x_t$ as a reference. The E-PCN does not perform this specific step, relying instead on the Guiding Energy to align with the G-PCN.
Parallel Weight Update:
- The weights of the G-PCN are updated to minimize the Internal Energy ( $I$ ).
- The weights of the E-PCN are updated to minimize the Guiding Energy ( $G$ ).

In this architecture, the E-PCN acts as a support network (similar to a teacher model) trained to match the G-PCN's representation, while the G-PCN acts as the primary learner (student) that propagates updates using the E-PCN's guidance.

Key Contributions

Single State Update Mechanism: The paper introduces a technique that allows all layers of a PCN to be updated in a single, unified iteration, overcoming the requirement for multiple state update iterations typical of standard PC training via a specific dual-network setup.
Dual-Network Architecture: By pairing a Generative PCN with an Encoding PCN, the method enforces layer-wise activation matching, which accelerates state updates without requiring multiple inference steps per time frame.
Parallelization: Unlike prior acceleration methods, this approach provides parallel and hierarchical updates across the network depth using batched data.

Experimental Results
The authors evaluated the technique on a continuous, online learning task for time-series anomaly detection using a dataset of moving MNIST digits.

Setup: A 5-layer MLP was trained to predict the next frame. Anomalies were introduced as abrupt changes in movement direction (X/Y inversion) or digit value.
Performance: The Guided PC approach demonstrated superior stability compared to traditional "Vanilla" PC (which required 5 inference steps).
- Both methods learned initial dynamics quickly.
- Upon anomaly introduction, the Guided PC rapidly adapted to treat the new behavior as "normal," recovering quickly.
- The Vanilla PC diverged after the anomaly, likely due to slower and vanishing error signal propagation.
Metric: The Guided PC achieved lower Mean Squared Error (MSE) spikes and faster recovery times, maintaining stability even with only one unified state update step.

Significance and Claims
The paper claims that this training technique effectively tackles the sequential state update bottleneck inherent to Predictive Coding by overcoming the requirement for multiple state update iterations. By enabling parallel updates across all layers in a single step via a dual-network architecture, the method enhances training stability and convergence speed. The authors suggest that this efficiency makes the approach particularly promising for online learning on edge devices, where computational resources and latency constraints are critical. The results indicate that the Guided PC can maintain continuous learning stability in dynamic environments where traditional PC methods might fail to adapt quickly enough.

Technical Summary: Single State Update Predictive Coding for Time Series Forecasting

More like this