Nonparametric Bayesian Contextual Control: Integrating Automatisation and Prior Knowledge for Stable Adaptive Behaviour

This paper proposes the Nonparametric Bayesian Contextual Control (NP-BCC) model, which integrates repetition-based automatisation and schema-like prior knowledge into nonparametric contextual learning to achieve both stable and flexible adaptive behavior in dynamic environments.

The Gist

Imagine your brain is like a highly sophisticated chef working in a busy, ever-changing restaurant kitchen.

This paper introduces a new computer model (called NP-BCC) that tries to figure out how this chef manages to cook amazing meals in a familiar kitchen, while also being able to instantly adapt when the ingredients change or a new, strange recipe is handed to them.

Here is the breakdown of the problem and the solution, using simple analogies:

The Problem: The "Stability vs. Flexibility" Dilemma

Humans are amazing at two seemingly opposite things:

1. Stability: When you drive your usual route to work, you do it on autopilot. You don't have to think about every turn. This is efficient and safe.
2. Flexibility: If a road is suddenly closed, you instantly switch to a new route without panicking.

The big question for scientists is: How does the brain switch between "autopilot" and "active thinking" so smoothly?

Old computer models tried to solve this by assuming the brain has a fixed list of "scenarios" (like a menu with 50 dishes). But real life is messy. Sometimes you encounter a completely new situation that isn't on the menu. If the brain only has a fixed menu, it gets confused and crashes.

The Solution: A "Smart Chef" with Two Superpowers

The authors built a new model that gives the brain two superpowers to handle this messiness:

1. The "Muscle Memory" Superpower (Automatisation)

• The Analogy: Think of riding a bike. The first time you ride, you are wobbling and thinking hard. After 100 rides, you don't think about balancing; your body just knows what to do.
• How the Model Uses It: The model learns that if it keeps doing the same action in the same situation and it works, it should "lock in" that action.
• The Magic: This doesn't just make the chef faster; it actually helps the chef know which kitchen they are in. If the chef is automatically chopping onions, the brain thinks, "Ah, I must be in the 'Soup Kitchen' context." The action itself becomes a clue about the situation, making the brain more stable and less confused.

2. The "Recipe Template" Superpower (Schema-like Prior Knowledge)

• The Analogy: Imagine you walk into a new restaurant. You've never been there, but you know it's a "Pizza Place." Even before you see the menu, your brain guesses: "Okay, they probably have dough, sauce, and cheese. They probably don't serve sushi." You aren't starting from zero; you are using a template.
• How the Model Uses It: When the model encounters a totally new situation, instead of guessing randomly, it loads a "template." It assumes, "This new situation probably works like other situations I've seen."
• The Magic: This lets the model learn new tasks incredibly fast (like "one-shot learning"). It doesn't have to relearn that "food usually tastes good" every time it enters a new kitchen.

The Experiment: The "Slot Machine" Game

To test this, the researchers put their "Smart Chef" model into a video game that is like a row of slot machines (called a Multi-Armed Bandit task).

• The Game: There are 4 slot machines. In one "context," Machine A pays out 90% of the time. In another "context," Machine B pays out 90% of the time. The game switches contexts secretly every 100 tries.
• The Challenge: The model has to figure out which machine is the "winner" right now, and also figure out when the rules of the game have changed.

What happened?

1. The "Naive" Chef (No Superpowers): When the game got complicated (4 machines), the naive model got confused. It kept mixing up the rules, thinking Machine A was the winner when it was actually Machine C. It took forever to learn.
2. The "Smart" Chef (With Superpowers):
   • With Muscle Memory: The model became much more confident. Because it stuck to its winning habits, it realized faster when the game changed. It stopped mixing up the rules.
   • With Recipe Templates: When a totally new machine configuration appeared, the model guessed the rules based on its "templates." It learned the new rules in a fraction of the time it took the naive model.

Why Does This Matter?

This isn't just about computer games. The authors suggest this explains how our brains work in real life, and what happens when things go wrong.

• Addiction and Habits: The paper suggests that in conditions like substance use disorders, the "Muscle Memory" superpower gets stuck on "Overdrive." The brain becomes so obsessed with the old "recipe" (using drugs) that it can't see the new reality (that it's harmful). The brain is so confident in its old context that it ignores the warning signs.
• Mental Health: If our "Recipe Templates" are too rigid, we might misinterpret new situations as dangerous or familiar when they aren't.

The Bottom Line

The brain is a master of context. It doesn't just react to the world; it guesses what "world" it is in.

• Repetition (doing things over and over) helps us stay stable and confident.
• Templates (using past knowledge to guess the future) help us learn new things instantly.

By combining these two, the brain creates a perfect balance: it's stable enough to be efficient, but flexible enough to survive a surprise. This new computer model proves that you need both to be truly smart in a changing world.

Technical Summary

1. Problem Statement

Human cognition exhibits a remarkable ability to balance stability (performing well-learned actions efficiently in familiar contexts) and flexibility (rapidly adapting to novel situations). While contextual inference models (e.g., the COIN model) explain how agents infer latent task structures to switch between behaviors, they face two critical limitations:

1. Instability in Complex Environments: Purely normative, nonparametric models often struggle to learn stable contextual representations in high-dimensional or complex environments (e.g., multi-armed bandits with many options) without excessive data. They tend to suffer from "context mixing," where contingencies from different contexts are erroneously merged.
2. Lack of Biological Plausibility: Standard models often ignore well-established cognitive mechanisms that humans use to accelerate learning and stabilize behavior, specifically repetition-based automatisation (habits) and schema-like prior knowledge (using past structural knowledge to scaffold new learning).

The authors ask: How can a computational model integrate nonparametric structure learning with automatisation and schema priors to achieve both stable and flexible adaptive behavior?

2. Methodology: The NP-BCC Model

The authors propose the Nonparametric Bayesian Contextual Control (NP-BCC) model, an extension of the Bayesian Contextual Control (BCC) framework.

Core Architecture

• Generative Model: The agent operates within a Hierarchical Partially Observable Markov Decision Process (POMDP).
  • Bottom Level: A context-dependent Markov Decision Process (MDP) handling state transitions ($S$), rewards ($R$), and actions ($A$).
  • Top Level: A nonparametric layer inferring latent contexts ($C$) and their transition dynamics.
• Nonparametric Priors (HDP-HMM): Unlike the original BCC which assumes a fixed number of contexts, the NP-BCC uses a Hierarchical Dirichlet Process Hidden Markov Model (HDP-HMM).
  • This allows the agent to infer the number of contexts dynamically.
  • It utilizes a global Dirichlet Process ($G_0$) to define a shared set of possible contexts and local Dirichlet Processes ($G_j$) to define context-specific transition probabilities.
  • Novelty Detection: The model includes a "novel context" component ($K+1$) with uniform contingencies. If the surprise (free energy) under known contexts exceeds that of the novel context, a new context is instantiated.

Key Integrations

The model introduces two specific mechanisms to the standard nonparametric framework:

1. Repetition-Based Automatisation:

   • Implemented as a context-specific prior over policies ($\theta$).
   • The model learns pseudo-counts ($\omega$) for how often a specific policy is executed within a specific context.
   • Mechanism: As a policy is repeated, the prior becomes sharper, biasing future action selection toward that policy. Crucially, the execution of this automated action generates a "surprise signal" (via policy cross-entropy) that reinforces the belief in the current context, creating a feedback loop that stabilizes context inference.

2. Schema-Like Template Contexts:

   • Implemented as structured priors for newly opened contexts.
   • Instead of initializing a new context with uniform ignorance (random contingencies), the model selects a template from a library of pre-defined structures (e.g., "Bandit $m$ is optimal, others are not").
   • Mechanism: When a novel context is detected, the agent loads the template that minimizes initial prediction error. This provides a "scaffold" for rapid learning, allowing the agent to converge on the true contingencies in fewer trials.

Inference Scheme

• Variational Inference: The model uses online variational Bayesian inference to approximate the posterior distribution over hidden variables (contexts, policies, parameters).
• Free Energy Minimization: Action selection is driven by minimizing variational free energy, balancing goal-directed planning (expected rewards) with the prior bias for automatisation.
• Update Rules: The model updates hyperparameters for context probabilities ($\beta$), transition dynamics ($\eta$), reward contingencies ($\phi$), and policy priors ($\theta$) after every episode.

3. Key Contributions

1. Unified Framework: The NP-BCC is the first model to formally integrate nonparametric structure learning (unbounded context discovery) with automatisation and schema-based priors within a single Active Inference framework.
2. Mechanistic Explanation of Stability: It demonstrates that automatisation is not just a behavioral output but a computational input that stabilizes context inference. By making actions informative about the context, it reduces uncertainty.
3. Schema-Driven Learning: It provides a proof-of-principle that structured priors (templates) can drastically reduce the data requirements for learning complex task structures, mimicking human "one-shot" or "few-shot" learning capabilities.
4. Clinical Relevance: The model offers a computational account for Substance Use Disorders (SUDs), suggesting that pathological rigidity arises when automatisation and schema priors create self-reinforcing loops that bias context inference toward maladaptive habits, preventing the detection of negative outcomes.

4. Results

The authors tested the model using Non-Stationary Multi-Armed Bandit (MAB) tasks with varying complexities ($M=2, 3, 4$ arms).

• Naive Structure Learning (Baseline):

  • In simple tasks ($M=2$), the agent successfully learned context structure.
  • In complex tasks ($M=4$), naive agents failed to acquire stable representations with standard training lengths. They exhibited context mixing (confusing contingencies between contexts) and required significantly more trials (300 vs 100) to learn, often failing to distinguish between similar contexts.

• Impact of Automatisation:

  • Agents with moderate automatisation ($h \approx 0.025$) achieved the highest contextual certainty and behavioral accuracy.
  • Automatisation reduced the "surprise" noise during stable periods, allowing the agent to lock onto the correct context faster.
  • U-Shaped Relationship: Too little automatisation led to instability; too much led to rigidity (failure to switch contexts even when rewards changed), mimicking the "habitual" behavior seen in addiction.

• Impact of Template Contexts:

  • Agents using template contexts learned the structure of the $M=4$ task significantly faster than naive agents, even with short training regimes (100 trials).
  • Templates prevented context mixing by providing a strong initial hypothesis that was quickly refined, rather than starting from scratch.
  • 99% of template-using agents successfully learned the task structure, compared to lower success rates for naive agents.

• Combined Effect: The integration of both mechanisms allowed the agent to maintain high performance and stability in complex, dynamic environments where purely rational agents failed.

5. Significance and Implications

• Computational Psychiatry: The paper provides a formal mechanism for understanding behavioral rigidity in disorders like addiction. It suggests that the pathological persistence of substance use is not just a failure of reward learning, but a failure of contextual inference driven by excessive automatisation and biased schema deployment.
• Cognitive Science: It bridges the gap between contextual inference (how we switch tasks) and schema theory (how we use past knowledge). It posits that schemas are not just static memories but active priors that shape the very structure of new learning.
• AI and Robotics: The NP-BCC offers a blueprint for building AI agents that can operate in non-stationary environments. By incorporating "habit" (automatisation) and "intuition" (templates), agents can achieve robustness without sacrificing the ability to adapt to novel situations.
• Future Directions: The authors suggest extending the model to include offline learning (sleep-like consolidation) and investigating how higher-level processes might dynamically regulate the hyperparameters (e.g., the tendency to open new contexts) in biological systems.

In summary, the NP-BCC model demonstrates that stability and flexibility are not opposing forces but are co-regulated by the interplay of surprise-driven structure learning, repetition-based habit formation, and schema-guided prior knowledge.