Spectral learning on matrices and tensors

DOI10.1561/2200000057MaRDI QIDQ5213205zbMATH OpenOpenAlexWikidataFDO

Authors Majid Janzamin, Rong Ge, Jean Kossaifi, Anima Anandkumar

Publication date 31 January 2020

Published in Foundations and Trends® in Machine Learning (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2004.07984, https://authors.library.caltech.edu/103456/

zbMATH Keywords

dimensionality reductionindependent component analysismatrix decompositionspectral methodssurveytensor decomposition

Mathematics Subject Classification ID

Eigenvalues, singular values, and eigenvectors (15A18) Factorization of matrices (15A23) Multilinear algebra, tensor calculus (15A69) Factor analysis and principal components; correspondence analysis (62H25) Numerical linear algebra (65F99) Research exposition (monographs, survey articles) pertaining to computer science (68-02) Learning and adaptive systems in artificial intelligence (68T05) Artificial neural networks and deep learning (68T07)

Abstract: Spectral methods have been the mainstay in several domains such as machine learning and scientific computing. They involve finding a certain kind of spectral decomposition to obtain basis functions that can capture important structures for the problem at hand. The most common spectral method is the principal component analysis (PCA). It utilizes the top eigenvectors of the data covariance matrix, e.g. to carry out dimensionality reduction. This data pre-processing step is often effective in separating signal from noise. PCA and other spectral techniques applied to matrices have several limitations. By limiting to only pairwise moments, they are effectively making a Gaussian approximation on the underlying data and fail on data with hidden variables which lead to non-Gaussianity. However, in most data sets, there are latent effects that cannot be directly observed, e.g., topics in a document corpus, or underlying causes of a disease. By extending the spectral decomposition methods to higher order moments, we demonstrate the ability to learn a wide range of latent variable models efficiently. Higher-order moments can be represented by tensors, and intuitively, they can encode more information than just pairwise moment matrices. More crucially, tensor decomposition can pick up latent effects that are missed by matrix methods, e.g. uniquely identify non-orthogonal components. Exploiting these aspects turns out to be fruitful for provable unsupervised learning of a wide range of latent variable models. We also outline the computational techniques to design efficient tensor decomposition methods. We introduce Tensorly, which has a simple python interface for expressing tensor operations. It has a flexible back-end system supporting NumPy, PyTorch, TensorFlow and MXNet amongst others, allowing multi-GPU and CPU operations and seamless integration with deep-learning functionalities.

Recommendations

• scientific article; zbMATH DE number 1820149
• Tensor Decompositions and Applications
• Bayesian Tensor Decomposition for Signal Processing and Machine Learning
• Matrix analysis and applications
• Tensor Decompositions for Learning Latent Variable Models (A Survey for ALT)

Cited in

(11)

• Tensor decomposition for learning Gaussian mixtures from moments
• Spectral Algorithms for Supervised Learning
• Spectral Norm in Learning Theory: Some Selected Topics
• Efficient Tensor Decompositions
• Overcomplete order-3 tensor decomposition, blind deconvolution, and Gaussian mixture models
• Generalized canonical polyadic tensor decomposition
• Low-rank tensor methods for partial differential equations
• A dive into spectral inference networks: improved algorithms for self-supervised learning of continuous spectral representations
• Optimality conditions for Tucker low-rank tensor optimization
• On the multiway principal component analysis
• Algorithms for learning a mixture of linear classifiers

Describes a project that uses

Uses Software

• PyTorch
• MXNet
• TensorFlow
• NumPy

This page was built for publication: Spectral learning on matrices and tensors

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5213205)