Self-Orthogonalizing Attractor Neural Networks: A Principled Approach from the Free Energy Principle

Attractor networks—systems in which collective dynamics settle into stable patterns—have long provided a powerful metaphor for understanding both neural computation and artificial intelligence. In “Self-orthogonalizing attractor neural networks emerging from the free energy principle,” Spisák and Friston (2025) show how attractor architectures arise naturally from the Free Energy Principle (FEP), yielding networks that organize themselves, learn continuously, and encode information in nearly orthogonal representations. Below, we unpack the main ideas, findings, and implications of this work, focusing on:

• Why attractors matter and how the FEP provides a principled route to self-organizing networks

• The core mathematical derivation linking deep particular partitions to a Boltzmann/Hopfield-like form

• Learning and inference dynamics that emerge without hand-crafted rules

• The surprising tendency toward orthogonal attractor representations and its benefits

• Illustrative simulations demonstrating retrieval, generalization, sequence learning, and robustness

• Broader implications for neuroscience, continual learning, and neuromorphic computing

By the end, you’ll understand both how attractor networks can be viewed as a natural consequence of Bayesian free-energy minimization, and why this perspective matters for building more adaptable, efficient, and biologically plausible AI systems.

From Fixed-Point Landscapes to Self-Organizing Attractors

Attractors in Nature and the Brain. Across domains—from fluid vortices to flocking birds—complex systems often organize around attractor states, regions in state space toward which trajectories converge [Haken, 1978; Guckenheimer et al., 1984]. In neuroscience, attractor networks have been used to model working memory, decision-making, spatial navigation, and more. For example, ring-attractor dynamics underlie head-direction cells, creating a circular manifold on which the animal’s orientation is encoded [Zhang, 1996; Rolls, 2009].

Operational vs. Adaptive Self-Organization. The authors distinguish two levels of self-organization:

1. Operational self-organization: a preconfigured network settling into attractor states during operation.

2. Adaptive self-organization: a network that builds and refines itself through local interactions and learning. The latter mirrors how the brain not only operates as an attractor network but also develops and continuously adapts its architecture without an external “teacher.”

The Free Energy Principle (FEP) as a Unifying Lens. The FEP posits that any system maintaining itself in a changing environment must minimize a quantity called variational free energy (VFE), which balances accuracy (fitting sensory inputs) and complexity (keeping models simple) [Friston, 2009; Friston et al., 2023]. Under the FEP, a system’s internal states play the role of “inference,” using sensory data to infer external causes, while changes in coupling (synaptic weights) correspond to learning [Friston et al., 2016]. Spisák and Friston apply this principle to a deep hierarchy of “particular partitions,” showing that attractor architectures emerge naturally from FEP-driven self-organization.

Deep Particular Partitions → Boltzmann-Style Attractor Networks

Particular Partitions 101. A “particular partition” divides a system’s variables x into internal states (μ), external states (η), and a Markov blanket (sensory s and active a states) that mediates interactions [Friston et al., 2022; Friston and Ao, 2012]. Formally, this enforces conditional independence η⊥μ∣s,a, which in turn implies that internal states must minimize VFE:

\mu̇ = -∇_{\mu}F(s, a, \mu) \tag{1}

with

F(s,a,μ)=Eqμ(η)[ln⁡qμ(η)−ln⁡p(s,a,η)].(2)

From Macro to Micro: Nested (Deep) Partitions. Each “particle” (μ,s,a) can itself be partitioned into overlapping “subparticles” (σi,si,ai). These subparticles obey the same conditional independence at the micro-scale:

σi⊥σj∣sij,aij( i≠j ).

Recursively applying this idea yields a hierarchy of partitions, from the full network down to individual units (Figure 1B in the paper).

Parametrizing Subparticles with Continuous Bernoulli States. Spisák and Friston assume that each subparticle’s internal state σi∈[−1,+1] follows a continuous Bernoulli distribution with bias bi:

p(σi)∝e biσi.(3)

Sensory interactions between subparticles i and j are deterministic:

sij=Jij σj,aij=Jji σi,(4)

where Jij is a (possibly asymmetric) weight. Integrating out boundary states yields a joint distribution over all {σi}:

p(σ)  ∝  exp⁡ ⁣( ∑ibi σi  +  ∑i<j(Jij+Jji) σi σj).(5)

Crucially, even if J is asymmetric, only the symmetric part J†=12(J+JT) appears in the exponent, giving a Boltzmann-like energy:

E(σ)  =  −∑ibi σi  −  12∑i,j Jij† σi σj.(6)

Thus, at steady state, this system behaves as a continuous-state Hopfield/Boltzmann network, where high-probability regions correspond to stochastic attractors.

Inference and Learning Without Hand-Crafted Rules

1 Inference: Local Free-Energy Minimization ⇒ Sigmoid-Like Updates

Each node σi maintains a variational approximation q(σi) parameterized by bias bq. Writing VFE for node i (conditioned on all other nodes σ\i) gives:

F  =  Eq[ln⁡q(σi)−ln⁡p(σ\i,σi)],(7)

which—once you plug in the continuous Bernoulli form and rearrange—leads to a gradient condition ∂F/∂bq=0. Solving yields:

E[σi]  =  L(bi+∑j≠iJij σj),(8)

where L(⋅) is the Langevin function (a sigmoid arising from continuous Bernoulli). In practice, stochastic sampling from this CB distribution implements a Gibbs-like update akin to Hopfield networks, but now grounded in variational free-energy minimization. In other words, each node performs a local approximate Bayesian update, balancing its prior bi against weighted inputs ∑jJijσj.

2 Learning: Local Free-Energy Gradients ⇒ Hebbian/Anti-Hebbian Updates

When node i perturbs a weight Jij by δJij, it influences its total input  ui=bi+∑k≠iJik σk. Taking the derivative of VFE with respect to Jij yields a contrastive-Hebbian rule:

ΔJij  ∝  σi σj  −  L ⁣(bi+∑k≠iJik σk) σj.(9)

The first term is Hebbian (observed correlation), and the second is anti-Hebbian (prediction from existing weights). Unlike contrastive divergence, this update uses instantaneous samples, making it fully local and highly scalable. Over repeated presentations, if data are shown in random order, J converges to symmetric couplings (a classic Hopfield equilibrium). If data arrive in fixed sequences, the antisymmetric part persists, enabling heteroclinic/sequence attractors.

Orthogonal Attractor Representations: Why They Arise and Why They Matter

A central insight is that minimizing free energy implicitly trades off accuracy (fitting each input pattern perfectly) against complexity (keeping representations simple). Highly correlated attractors increase redundancy, inflating model complexity. The Hebbian/anti-Hebbian rule in equation (9) ensures that—when learning a new pattern correlated with existing attractors—the network only reinforces the residual, i.e., the component orthogonal to what’s already stored. Concretely:

1. Present pattern s(1); store attractor σ(1).

2. Present correlated pattern s(2). The network’s current weights already predict part of σ(2) from σ(1).

3. The update ΔJij is driven by σ(2)−σ^(2), where σ^(2) is what’s explained by σ(1).

4. Consequently, new attractors tend to be pushed into the orthogonal complement of the subspace spanned by existing ones.

Thus, learning under FEP naturally yields approximately orthogonal attractor states, which minimizes redundancy and maximizes the mutual information between hidden causes and observed data. In other words, orthogonal attractors serve as a compact, efficient basis for representing diverse inputs—enhancing both capacity and generalization.

Simulations: Demonstrating Key Capabilities

Spisák and Friston illustrate their theory through four simulations, all based on handwritten digits (8×8 pixels). Below we summarize the highlights:

Simulation 1: Orthogonal Basis Formation & Bayesian Inference

• Setup: A 25-unit network (5×5 images) is trained on just two correlated patterns (Pearson’s r = 0.77). Inference and learning run simultaneously, with inverse temperature iT=0.1 and learning rate α=0.01.

• Result: After 500 epochs (randomly sampling one of the two patterns per epoch), the network’s attractors—obtained via deterministic inference—have Pearson correlation r=−0.19. In other words, they become nearly orthogonal despite the input correlation.

• Bayesian Retrieval: With stochastic inference on a noisy version of one pattern, the network converges (via MCMC-like sampling) to the correct attractor—showing how macro-scale inference arises from local free-energy updates.

• Generalization: Introducing a new pattern in the subspace spanned by the two orthogonal attractors, the network reconstructs it successfully by combining attractor dynamics.

Simulation 2: Systematic Exploration of Learning Regimes

• Setup: A 10×10 network (for 8×8 digit patches) is trained on 10 digits (one exemplar per digit). The remaining 1,787 digits form a test set. The authors vary inverse temperature iT from 0.01 to 1 and evidence strength (bias magnitude) from 1 to 20, running 3,800 total models.

• Metrics:

• Key Findings:

Simulation 3: Sequence Learning via Asymmetric Couplings

• Setup: The network is trained on the fixed sequence of three digits (1 → 2 → 3 → 1 → …). Each epoch consists of a single step with bias corresponding to the current digit (inverse temperature iT=1, α=0.001).

• Result: The coupling matrix J becomes visibly asymmetric (Figure 5B). Decomposing J=J(sym)+J(asym):

Hence, by presenting data in a temporal order, the FEP-derived rule automatically generates both fixed attractors and the heteroclinic/sequence attractors needed to replay learned sequences.

Simulation 4: Resistance to Catastrophic Forgetting via Spontaneous Replay

• Setup: Take the well-trained network from Simulation 2 (with balanced regime) and run it for an additional 50,000 epochs with zero bias (i.e., no external inputs), but continue applying the local weight updates.

• Result: Even after long “free-running” (spontaneous) activity, the coupling matrix and performance metrics (retrieval and generalization R2) remain almost unchanged (Figure 6A–C). Learned attractors become slightly “soft,” but the network does not catastrophically forget earlier patterns. Spontaneous replay in attractor basins continually reinforces existing weights, providing a built-in mechanism for lifelong learning.

Why This Matters: Implications & Future Directions

A Unifying Theory of Self-Organizing Attractors

By deriving attractor networks directly from the FEP, the authors provide a first-principles explanation for why brains and other complex systems might adopt attractor architectures. Key points:

1. Inference = Free-Energy Minimization. Each node’s stochastic update aligns with approximate Bayesian inference.

2. Learning = Free-Energy Minimization. Coupling plasticity arises organically from local gradients of VFE, resembling predictive-coding and Hebbian/anti-Hebbian learning.

3. Macro-Scale Inference. The entire network—being itself a “particle” under FEP—performs global (MCMC-like) Bayesian sampling of the posterior when presented with new evidence.

4. Orthogonalization as an Information-Theoretic Imperative. Minimizing model complexity forces attractors to occupy distinct dimensions, maximizing mutual information and storage capacity.

Together, these points offer a principled route to adaptive self-organization, rather than prescribing hand-tuned update rules or energy functions.

Neuroscientific Relevance & Testable Predictions

• Orthogonal Resting-State Attractors. Large-scale fMRI analyses find that resting-state networks (RSNs) behave like attractors whose activity patterns are largely orthogonal [Englert et al., 2023]. The theory predicts that, if brains optimize free energy, their dominant attractor modes should be near-orthogonal—a hypothesis now amenable to empirical test.

• Sequence Replay & Solenoidal Flows. The existence of asymmetric couplings and solenoidal probability currents is consistent with findings on spontaneous sequence replay during rest and sleep. FEP suggests these flows should be divergence-free and preserve a Boltzmann-like prior, a prediction accessible via neurophysiological recordings.

AI Implications: Continual Learning & Neuromorphic Computing

• Built-In Continual Learning. Because learning and inference are two sides of the same VFE coin, the network inherently revisits and reinforces existing attractors during “idle” periods (zero bias). This mechanism sidesteps catastrophic forgetting—a major obstacle in deep learning—by continually replaying stored patterns whenever the system is active.

• Predictive Coding Architectures. The local update rule (9) ties directly into recent work showing that predictive‐coding networks can recover backpropagation as a special case [Millidge et al., 2022; Salvatori et al., 2023]. FEP-driven attractor nets may scale to complex, cyclic architectures without requiring separate forward and backward passes.

• Orthogonality for Memory Capacity. The tendency to orthogonalize attractors closely resembles projection-based associative memories (Kanter & Sompolinsky, 1987). Implementing FEP-derived learning may yield networks with higher capacity and more robust retrieval than standard Hopfield or Boltzmann machines.

• Thermodynamic and Neuromorphic Hardware. The inherently stochastic updates, precision modulation (via inverse temperature iT), and local learning rule map neatly onto neuromorphic and thermodynamic computing paradigms [Schuman et al., 2022; Melanson et al., 2025]. Energy-efficient hardware that natively implements stochasticity could naturally host these attractor dynamics.

Horizons: Hierarchical, Multi-Scale Active Inference

Because partitions can be nested arbitrarily, networks can be built in hierarchies—from individual neurons up to entire brain regions—each level performing local free-energy minimization. Fast neuronal updates at lower levels project onto slower dynamics at higher levels, echoing the center manifold theorem [Wagner, 1989]. Such a multi-scale FEP framework may explain how local circuit dynamics give rise to large-scale cognitive attractors, unifying single-unit plasticity and system-level behavior.

Conclusions

Spisák and Friston’s work elegantly demonstrates that attractor networks are not merely a convenient engineering tool but a natural consequence of Bayesian self-organization under the Free Energy Principle. By showing how local inference and learning rules fall out of free-energy gradients on nested partitions, they:

• Recover classic stochastic Hopfield/Boltzmann architectures as steady states.

• Reveal why coupling plasticity should be Hebbian with an anti-Hebbian corrective term.

• Prove that attractors will tend to be (approximately) orthogonal, optimizing capacity and generalization.

• Illustrate how sequence learning and robust replay emerge when data have temporal structure.

• Highlight an inherent antidote to catastrophic forgetting through spontaneous replay.

Beyond deepening our theoretical understanding of brain-like computation, this framework points toward future AI systems that can learn continuously, form compact orthogonal memories, and flexibly combine learned elements to generalize to novel situations—all within a principled variational framework. As we continue to scale up active inference and predictive coding to real-world tasks, FEP-derived attractor networks provide a promising blueprint for building more adaptive, efficient, and explainable machine intelligence.