Weight-decay induced phase transitions in multilayer neural networks

DOI10.1088/0305-4470/32/27/301MaRDI QIDQ4947664zbMATH OpenOpenAlexWikidataFDO

Authors M. Ahr, E. Schlösser, Michael Biehl

Publication date 25 April 2000

Published in Journal of Physics A: Mathematical and General (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/cond-mat/9901179, https://pure.rug.nl/ws/files/3153518/1999JPhysAAhr1.pdf

zbMATH Keywords

layered neural networksphase transition

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Neural nets applied to problems in time-dependent statistical mechanics (82C32) Neural networks for/in biological studies, artificial life and related topics (92B20)

Abstract: We investigate layered neural networks with differentiable activation function and student vectors without normalization constraint by means of equilibrium statistical physics. We consider the learning of perfectly realizable rules and find that the length of student vectors becomes infinite, unless a proper weight decay term is added to the energy. Then, the system undergoes a first order phase transition between states with very long student vectors and states where the lengths are comparable to those of the teacher vectors. Additionally in both configurations there is a phase transition between a specialized and an unspecialized phase. An anti-specialized phase with long student vectors exists in networks with a small number of hidden units.

Recommendations

• Phase transitions in the generalization behaviour of multilayer neural networks
• Structural phase transitions in neural networks
• Phase transitions and learning in neural networks
• Boundary conditions and phase transitions in neural networks. Theoretical results
• Phase transitions in the generalization behaviour of multilayer perceptrons: II. The influence of noise
• Boundary conditions and phase transitions in neural networks. Simulation results
• Observation of a hexatic vortex glass in flux lattices of the high-<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:msub><mml:mrow><mml:mi mathvariant="italic">T</mml:mi></mml:mrow><mml:mrow><mml:mi mathvariant="ita
• Phase diagram for two-layer ReLU neural networks at infinite-width limit
• Threshold-induced phase transitions in perceptrons

Cited in

(3)

• Boundary conditions and phase transitions in neural networks. Simulation results
• On Breiman's dilemma in neural networks: phase transitions of margin dynamics
• Generalization performance of narrow shallow neural networks in the teacher-student setting

This page was built for publication: Weight-decay induced phase transitions in multilayer neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4947664)