Predictive coding, variational autoencoders, and biological connections

DOI10.1162/NECO_A_01458zbMATH Open1483.92017arXiv2011.07464OpenAlexW3211640812MaRDI QIDQ5037129FDOQ5037129

Authors: Joseph Marino 

Publication date: 25 February 2022

Published in: Neural Computation (Search for Journal in Brave)

Abstract: This paper reviews predictive coding, from theoretical neuroscience, and variational autoencoders, from machine learning, identifying the common origin and mathematical framework underlying both areas. As each area is prominent within its respective field, more firmly connecting these areas could prove useful in the dialogue between neuroscience and machine learning. After reviewing each area, we discuss two possible correspondences implied by this perspective: cortical pyramidal dendrites as analogous to (non-linear) deep networks and lateral inhibition as analogous to normalizing flows. These connections may provide new directions for further investigations in each field.

Full work available at URL: https://arxiv.org/abs/2011.07464

Recommendations

• Predictive coding approximates backprop along arbitrary computation graphs
• A predictive-coding network that is both discriminative and generative
• Predictive Coding and the Slowness Principle: An Information-Theoretic Approach
• Minimizing relative entropy in hierarchical predictive coding
• Learning probabilistic neural representations with randomly connected circuits

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Neural networks for/in biological studies, artificial life and related topics (92B20) Research exposition (monographs, survey articles) pertaining to biology (92-02)

Cites Work

• Title not available (Why is that?)
• Independent component analysis, a new concept?
• A Mathematical Theory of Communication
• Machine learning. A probabilistic perspective
• Title not available (Why is that?)
• Learning representations by back-propagating errors
• Deep learning
• Probabilistic Principal Component Analysis
• Covariance estimation: the GLM and regularization perspectives
• Neural networks and physical systems with emergent collective computational abilities
• A logical calculus of the ideas immanent in nervous activity
• Title not available (Why is that?)
• Title not available (Why is that?)
• Varieties of Helmholtz machine
• Title not available (Why is that?)
• Title not available (Why is that?)
• Fusion, propagation, and structuring in belief networks
• A Family of Nonparametric Density Estimation Algorithms
• Supervised and unsupervised learning with two sites of synaptic integration
• Optimal Whitening and Decorrelation
• Equivalence of Backpropagation and Contrastive Hebbian Learning in a Layered Network
• An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity
• Redundancy reduction with information-preserving nonlinear maps

Cited In (5)

• Toward optimality in scalable predictive coding
• Minimizing relative entropy in hierarchical predictive coding
• A predictive-coding network that is both discriminative and generative
• Predictive Coding and the Slowness Principle: An Information-Theoretic Approach
• Predictive coding approximates backprop along arbitrary computation graphs

Uses Software

• PMTK
• ImageNet
• NICE
• PixelVAE
• WaveNet
• NVAE

This page was built for publication: Predictive coding, variational autoencoders, and biological connections

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5037129)