Non-deterministic learning dynamics in large neural networks due to structural data bias

DOI10.1088/0305-4470/33/48/309MaRDI QIDQ2703484zbMATH OpenOpenAlexFDO

Authors H. C. Rae, J. A. F. Heimel, A. C. C. Coolen

Publication date 5 March 2001

Published in Journal of Physics A: Mathematical and General (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/cond-mat/0007232

Mathematics Subject Classification ID

Neural nets applied to problems in time-dependent statistical mechanics (82C32) Neural networks for/in biological studies, artificial life and related topics (92B20)

Abstract: We study the dynamics of on-line learning in large perceptrons, for the case of training sets with a structural bias of the input vectors, by deriving exact and closed macroscopic dynamical laws using non-equilibrium statistical mechanical tools. In sharp contrast to the more conventional theories developed for homogeneously distributed or only weakly biased data, these laws are found to describe a non-trivial and persistently non-deterministic macroscopic evolution, and a generalisation error which retains both stochastic and sample-to-sample fluctuations, even for infinitely large networks. Furthermore, for the standard error-correcting microscopic algorithms (such as the perceptron learning rule) one obtains learning curves with distinct bias-induced phases. Our theoretical predictions find excellent confirmation in numerical simulations.

Recommendations

• scientific article; zbMATH DE number 1569113
• The connections of large perceptrons
• Transient dynamics of on-line learning in two-layered neural networks
• scientific article; zbMATH DE number 1569114
• Dynamics of online Hebbian learning with structurally unrealizable restricted training sets.

Cited in

(1)

• Data-dependence of plateau phenomenon in learning with neural network—statistical mechanical analysis ^*

This page was built for publication: Non-deterministic learning dynamics in large neural networks due to structural data bias

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2703484)