Functional optimization of online algorithms in multilayer neural networks

DOI10.1088/0305-4470/30/17/002MaRDI QIDQ4235302zbMATH OpenOpenAlexFDO

Authors Renato Vicente, Nestor Caticha

Publication date 24 May 2001

Published in Journal of Physics A: Mathematical and General (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/cond-mat/9706015

zbMATH Keywords

learning algorithmonline dynamics of learningsoft committee machinessymmetric phase plateaux

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Neural nets applied to problems in time-dependent statistical mechanics (82C32) Neural networks for/in biological studies, artificial life and related topics (92B20)

Abstract: We study the online dynamics of learning in fully connected soft committee machines in the student-teacher scenario. The locally optimal modulation function, which determines the learning algorithm, is obtained from a variational argument in such a manner as to maximise the average generalisation error decay per example. Simulations results for the resulting algorithm are presented for a few cases. The symmetric phase plateaux are found to be vastly reduced in comparison to those found when online backpropagation algorithms are used. A discussion of the implementation of these ideas as practical algorithms is given.

Recommendations

• On-line learning in the committee machine
• scientific article; zbMATH DE number 1569107
• Globally optimal on-line learning rules for multi-layer neural networks
• Transient dynamics of on-line learning in two-layered neural networks

Cited in

(9)

• Hidden unit specialization in layered neural networks: ReLU vs. sigmoidal activation
• Globally optimal on-line learning rules for multi-layer neural networks
• Transient dynamics of on-line learning in two-layered neural networks
• scientific article; zbMATH DE number 1569107 (Why is no real title available?)
• scientific article; zbMATH DE number 1569108 (Why is no real title available?)
• On-line learning in the committee machine
• Matrix-update for accelerated on-line learning in multilayer neural networks
• Training multilayer perceptrons by principal component analysis
• The statistical physics of learning in under- and over-parameterized layered neural networks

This page was built for publication: Functional optimization of online algorithms in multilayer neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4235302)