A theory of capacity and sparse neural encoding

DOI10.1016/J.NEUNET.2021.05.005zbMATH Open1521.68109arXiv2102.10148MaRDI QIDQ6079092FDOQ6079092

Authors: Pierre Baldi, Roman Vershynin 

Publication date: 28 September 2023

Published in: Neural Networks (Search for Journal in Brave)

Abstract: Motivated by biological considerations, we study sparse neural maps from an input layer to a target layer with sparse activity, and specifically the problem of storing $K$ input-target associations $(x,y)$, or memories, when the target vectors $y$ are sparse. We mathematically prove that $K$ undergoes a phase transition and that in general, and somewhat paradoxically, sparsity in the target layers increases the storage capacity of the map. The target vectors can be chosen arbitrarily, including in random fashion, and the memories can be both encoded and decoded by networks trained using local learning rules, including the simple Hebb rule. These results are robust under a variety of statistical assumptions on the data. The proofs rely on elegant properties of random polytopes and sub-gaussian random vector variables. Open problems and connections to capacity theories and polynomial threshold maps are discussed.

Full work available at URL: https://arxiv.org/abs/2102.10148

Recommendations

• Lower bounds on the capacities of binary and ternary networks storing sparse random vectors
• Notions of associative memory and sparse coding
• Sparse coding for layered neural networks
• A comparative study of sparse associative memories
• Dilution and sparse coding in threshold-linear nets

zbMATH Keywords

threshold functionssparse representationssparse encodingneural mapsneural capacity

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Neural networks for/in biological studies, artificial life and related topics (92B20)

Cites Work

• Title not available (Why is that?)
• Support-vector networks
• The horseshoe estimator for sparse signals
• Bayesian Variable Selection in Linear Regression
• 10.1162/15324430152748236
• High-dimensional probability. An introduction with applications in data science
• Compressive sampling
• Compressed sensing
• The Generalized Lasso With Non-Linear Observations
• A mathematical introduction to compressive sensing
• Asymptotic geometric analysis. I
• Neural networks and physical systems with emergent collective computational abilities
• Asymptotic shape of a random polytope in a convex body
• Random projections of regular simplices
• EXTREMAL PROPERTIES OF ORTHOGONAL PARALLELEPIPEDS AND THEIR APPLICATIONS TO THE GEOMETRY OF BANACH SPACES
• Living on the edge: phase transitions in convex programs with random data
• Counting faces of randomly projected polytopes when the projection radically lowers dimension
• Smallest singular value of random matrices and geometry of random polytopes
• Counting the faces of randomly-projected hypercubes and orthants, with applications
• Random polytopes
• Gaussian polytopes: variances and limit theorems
• Universality in polytope phase transitions and message passing algorithms
• One-bit compressed sensing with non-Gaussian measurements
• Linear Inversion of Band-Limited Reflection Seismograms
• Central limit theorems for Gaussian polytopes
• Banach-Mazur distances and projections on random subgaussian polytopes
• Random projections of regular polytopes
• The geometry of random \(\{-1,1\}\)-polytopes
• Dimension reduction by random hyperplane tessellations
• Title not available (Why is that?)
• Gaussian polytopes: a cumulant-based approach
• Robust 1-bit Compressed Sensing and Sparse Logistic Regression: A Convex Programming Approach
• Random spaces generated by vertices of the cube
• Probability
• Title not available (Why is that?)
• The capacity of feedforward neural networks
• Cones generated by random points on half-spheres and convex hulls of Poisson point processes
• Polynomial threshold functions, hyperplane arrangements, and random tensors
• Expected intrinsic volumes and facet numbers of random beta-polytopes
• Deep learning in science

Cited In (6)

• Expansion of information in the binary autoencoder with random binary weights
• Lah distribution: Stirling numbers, records on compositions, and convex hulls of high-dimensional random walks
• Sparse coding for layered neural networks
• Lower bounds on the capacities of binary and ternary networks storing sparse random vectors
• Tractability from overparametrization: the example of the negative perceptron
• What intraclass covariance structures can symmetric Bernoulli random variables have?

This page was built for publication: A theory of capacity and sparse neural encoding

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6079092)