Natural Gradient Learning in Predictive Coding through Lateral Circuits
Abstract
Predictive coding (PC) trains neural networks through local energy-minimization dynamics without depending on a global error signal that backpropagation (BP) requires. Although PC is a promising alternative algorithm for efficient learning, the computational cost of inference dynamics currently outweighs the resulting learning benefits in practice. In this work, we take inspiration from optimization theory to extract more learning benefit from each inference phase. We introduce Natural PC, which augments standard PC with a lateral circuit, enabling exact and efficient natural gradient learning locally in space and time. Our lateral circuit learns the activity correlations that determine the Fisher information matrix and implicitly computes its inverse, avoiding expensive matrix inversion. Across all tested image reconstruction benchmarks, Natural PC achieves state of the art in terms of sample and compute efficiency compared to existing PC approaches. Most notably, Natural PC scales to ImageNet and matches the performance of BP trained with AdamW and various second-order optimizers at an order of magnitude lower computational cost while also using two orders of magnitude fewer training samples. Our results demonstrate that local natural gradient learning makes PC competitive with BP in both sample efficiency and computational cost in reconstruction tasks.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a significant advancement in local learning algorithms by integrating natural gradient descent into Predictive Coding via lateral circuits, achieving state-of-the-art efficiency on ImageNet reconstruction. It successfully demonstrates that biologically inspired local dynamics can match the performance of global backpropagation with substantially lower sample and computational costs, offering a compelling alternative for efficient neural network training.
Methodology
The paper introduces "Natural PC," a modification to Predictive Coding (PC) that incorporates a lateral interneuron circuit to implement natural gradient descent. The core theoretical contribution is the derivation of the exact layerwise natural gradient for PC, showing that the Fisher information matrix corresponds to the Gram matrix of presynaptic activities. The authors demonstrate that lateral weights can learn this Gram matrix and that the interneuron dynamics implicitly compute its inverse via dynamical inversion, avoiding explicit matrix inversion costs. A significant theoretical insight is the unification of this approach with Newton and Gauss-Newton optimization, explaining why the method allows for unit step sizes (eliminating the learning rate hyperparameter) and achieves faster convergence. The biological plausibility of using lateral inhibition for decorrelation/whening to guide learning is a strong conceptual hook.
Results & evaluation
The experimental validation is robust and scales to a significant benchmark (ImageNet-1k). The paper reports that Natural PC matches the performance of Backpropagation (BP) with AdamW at an order of magnitude lower computational cost and with two orders of magnitude fewer training samples. It outperforms existing PC variants (iPC, ePC, bPC, PC-ALM) which fail to scale to ImageNet. The authors provide detailed ablations on the alignment of lateral weights with the true Fisher matrix and the effective step size, confirming the theoretical predictions. The results on smaller datasets (MNIST, CIFAR-10, etc.) further support the sample efficiency claims.
Reproducibility
The paper provides sufficient detail on the architecture (VGG-style decoder, 212.6M params), hyperparameters (30 Nesterov steps, specific precision values), and training protocols. The description of the lateral circuit dynamics and weight update rules is clear. However, as is common with arXiv preprints, specific code links are not provided in the text, which may hinder immediate reproduction without the authors' repository.
Limitations
The method is primarily validated on generative reconstruction tasks. The authors acknowledge that discriminative tasks (classification) suffer from a "target absorption problem" where the top layer explains the label, blocking credit propagation, and Natural PC does not solve this at scale. The method relies on the assumption of stationary activity statistics for the lateral weight convergence, which may be violated in non-stationary data streams. Additionally, the computational overhead per batch is higher than BP (12x more FLOPs per batch), though the total training cost is lower due to fewer required updates.
Broader impact
This work bridges the gap between biologically plausible local learning rules and high-performance deep learning. By making Predictive Coding competitive with Backpropagation in terms of sample and compute efficiency for large-scale vision tasks, it revitalizes interest in local learning algorithms. The insights into the role of interneurons in implementing natural gradient descent provide testable hypotheses for neuroscience. The elimination of the learning rate hyperparameter is a practical advantage for practitioners. The paper presents a significant advancement in local learning algorithms by integrating natural gradient descent into Predictive Coding via lateral circuits, achieving state-of-the-art efficiency on ImageNet reconstruction. It successfully demonstrates that biologically inspired local dynamics can match the performance of global backpropagation with substantially lower sample and computational costs, offering a compelling alternative for efficient neural network training.
Author of this paper? Send a correction.