10.1162/jmlr.2003.4.7-8.1393

DOI10.1162/JMLR.2003.4.7-8.1393zbMATH Open1061.62003arXivcond-mat/0309554OpenAlexW2097944542MaRDI QIDQ4669459FDOQ4669459

Authors: Gleb Basalyga, Magnus Rattray 

Publication date: 15 April 2005

Published in: CrossRef Listing of Deleted DOIs (Search for Journal in Brave)

Abstract: The learning dynamics of on-line independent component analysis is analysed in the limit of large data dimension. We study a simple Hebbian learning algorithm that can be used to separate out a small number of non-Gaussian components from a high-dimensional data set. The de-mixing matrix parameters are confined to a Stiefel manifold of tall, orthogonal matrices and we introduce a natural gradient variant of the algorithm which is appropriate to learning on this manifold. For large input dimension the parameter trajectory of both algorithms passes through a sequence of unstable fixed points, each described by a diffusion process in a polynomial potential. Choosing the learning rate too large increases the escape time from each of these fixed points, effectively trapping the learning in a sub-optimal state. In order to avoid these trapping states a very low learning rate must be chosen during the learning transient, resulting in learning time-scales of $O(N^2)$ or $O(N^3)$ iterations where $N$ is the data dimension. Escape from each sub-optimal state results in a sequence of symmetry breaking events as the algorithm learns each source in turn. This is in marked contrast to the learning dynamics displayed by related on-line learning algorithms for multilayer neural networks and principal component analysis. Although the natural gradient variant of the algorithm has nice asymptotic convergence properties, it has an equivalent transient dynamics to the standard Hebbian algorithm.

Full work available at URL: https://arxiv.org/abs/cond-mat/0309554

Recommendations

• scientific article; zbMATH DE number 1928762
• Stochastic trapping in a solvable model of on-line independent component analysis
• The dynamics of on-line principal component analysis
• The scaling limit of high-dimensional online independent component analysis
• From neural learning to independent components

zbMATH Keywords

statistical mechanicsHebbian learningnatural gradient

Mathematics Subject Classification ID

Statistical aspects of information-theoretic topics (62B10) Multivariate analysis (62H99) Learning and adaptive systems in artificial intelligence (68T05) Neural nets and related approaches to inference from stochastic processes (62M45)

Cited In (6)

• Multiplicative nonholonomic/Newton-like algorithm
• Title not available (Why is that?)
• A stochastic algorithm for probabilistic independent component analysis
• The scaling limit of high-dimensional online independent component analysis
• The dynamics of on-line principal component analysis
• Stochastic trapping in a solvable model of on-line independent component analysis

This page was built for publication: 10.1162/jmlr.2003.4.7-8.1393

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4669459)