Universal approximations of invariant maps by neural networks

DOI10.1007/S00365-021-09546-1zbMATH Open1505.41019arXiv1804.10306OpenAlexW3157282745WikidataQ110701955 ScholiaQ110701955MaRDI QIDQ2117338FDOQ2117338

Authors: Dmitry Yarotsky 

Publication date: 21 March 2022

Published in: Constructive Approximation (Search for Journal in Brave)

Abstract: We describe generalizations of the universal approximation theorem for neural networks to maps invariant or equivariant with respect to linear representations of groups. Our goal is to establish network-like computational models that are both invariant/equivariant and provably complete in the sense of their ability to approximate any continuous invariant/equivariant map. Our contribution is three-fold. First, in the general case of compact groups we propose a construction of a complete invariant/equivariant network using an intermediate polynomial layer. We invoke classical theorems of Hilbert and Weyl to justify and simplify this construction; in particular, we describe an explicit complete ansatz for approximation of permutation-invariant maps. Second, we consider groups of translations and prove several versions of the universal approximation theorem for convolutional networks in the limit of continuous signals on euclidean spaces. Finally, we consider 2D signal transformations equivariant with respect to the group SE(2) of rigid euclidean motions. In this case we introduce the "charge--conserving convnet" -- a convnet-like computational model based on the decomposition of the feature space into isotypic representations of SO(2). We prove this model to be a universal approximator for continuous SE(2)--equivariant signal transformations.

Full work available at URL: https://arxiv.org/abs/1804.10306

Recommendations

• Probabilistic symmetries and invariant neural networks
• What is... an Equivariant Neural Network?
• A unifying framework for invariant pattern recognition
• Group invariance, stability to deformations, and complexity of deep convolutional representations
• Universality of deep convolutional neural networks

zbMATH Keywords

invarianceapproximationneural networkpolarizationpolynomialequivariancelinear representationconvnet

Mathematics Subject Classification ID

General nonlinear regression (62J02) Learning and adaptive systems in artificial intelligence (68T05) Applications of group representations to physics and other areas of science (20C35) Neural nets and related approaches to inference from stochastic processes (62M45) Multidimensional problems (41A63) Actions of groups on commutative rings; invariant theory (13A50) Approximation by other special function classes (41A30)

Cites Work

• Deep learning
• Title not available (Why is that?)
• Title not available (Why is that?)
• Endomorphisms and automorphisms of the shift dynamical system
• Title not available (Why is that?)
• Multilayer feedforward networks are universal approximators
• Approximation by superpositions of a sigmoidal function
• Title not available (Why is that?)
• Group invariant scattering
• On invariance and selectivity in representation learning
• Approximation by superposition of sigmoidal and radial basis functions
• Rotation covariant image processing for biomedical applications
• Title not available (Why is that?)
• Zeros of equivariant vector fields: Algorithms for an invariant approach
• TDI-subspaces of \(C(\mathbb{R}^ d)\) and some density problems from neural networks

Cited In (21)

• Equivariant deep learning via morphological and linear scale space PDEs on the space of positions and orientations
• Ehresmann connections and feedforward neural networks
• \(\mathrm{SU}(1,1)\) equivariant neural networks and application to robust Toeplitz Hermitian positive definite matrix classification
• Iterative SE(3)-transformers
• A Proof that Artificial Neural Networks Overcome the Curse of Dimensionality in the Numerical Approximation of Black–Scholes Partial Differential Equations
• Title not available (Why is that?)
• A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networks
• Title not available (Why is that?)
• Probabilistic symmetries and invariant neural networks
• Approximation capabilities of measure-preserving neural networks
• Neural network approximation of continuous functions in high dimensions with applications to inverse problems
• The universal approximation theorem for complex-valued neural networks
• Statistical theory for image classification using deep convolutional neural network with cross-entropy loss under the hierarchical max-pooling model
• What is... an Equivariant Neural Network?
• Homogeneous vector bundles and \(G\)-equivariant convolutional neural networks
• Scale equivariant neural networks with morphological scale-spaces
• TransNet: shift invariant transformer network for side channel analysis
• Piecewise integrable neural network: an interpretable chaos identification framework
• Universal approximation of symmetric and anti-symmetric functions
• On the finite representation of linear group equivariant operators via permutant measures
• Full error analysis for the training of deep neural networks

Uses Software

• ScatNet
• PointNet
• RotEqNet

This page was built for publication: Universal approximations of invariant maps by neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2117338)