Neural network with unbounded activation functions is universal approximator
From MaRDI portal
Abstract: This paper presents an investigation of the approximation property of neural networks with unbounded activation functions, such as the rectified linear unit (ReLU), which is the new de-facto standard of deep learning. The ReLU network can be analyzed by the ridgelet transform with respect to Lizorkin distributions. By showing three reconstruction formulas by using the Fourier slice theorem, the Radon transform, and Parseval's relation, it is shown that a neural network with unbounded activation functions still satisfies the universal approximation property. As an additional consequence, the ridgelet transform, or the backprojection filter in the Radon domain, is what the network learns after backpropagation. Subject to a constructive admissibility condition, the trained network can be obtained by simply discretizing the ridgelet transform, without backpropagation. Numerical examples not only support the consistency of the admissibility condition but also imply that some non-admissible cases result in low-pass filtering.
Recommendations
- Approximation capabilities of neural networks on unbounded domains
- ReLU networks are universal approximators via piecewise linear or constant functions
- Nonlinear approximation and (deep) ReLU networks
- The universal approximation property. Characterization, construction, representation, and existence
- Error bounds for approximations with deep ReLU networks
Cites work
- scientific article; zbMATH DE number 741219 (Why is no real title available?)
- scientific article; zbMATH DE number 1022519 (Why is no real title available?)
- scientific article; zbMATH DE number 1405266 (Why is no real title available?)
- scientific article; zbMATH DE number 3204910 (Why is no real title available?)
- scientific article; zbMATH DE number 3240665 (Why is no real title available?)
- scientific article; zbMATH DE number 3272562 (Why is no real title available?)
- scientific article; zbMATH DE number 3329342 (Why is no real title available?)
- scientific article; zbMATH DE number 3187905 (Why is no real title available?)
- A Sobolev-type upper bound for rates of approximation by linear combinations of Heaviside plane waves
- A birth and death model of neuron firing
- A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training
- Approximation by superposition of sigmoidal and radial basis functions
- Classical Fourier Analysis
- Complexity estimates based on integral transforms induced by computational units
- Continuity of the Radon transform and its inverse on Euclidean space
- Convolution-backprojection method for the k-plane transform, and Calderón's identity for ridgelet transforms
- Functional analysis, Sobolev spaces and partial differential equations
- Harmonic analysis of neural networks
- Integral geometry and Radon transforms
- Morrey and Campanato meet Besov, Lizorkin and Triebel
- Ridge functions and orthonormal ridgelets
- Sparse image and signal processing. Wavelets, curvelets, morphological diversity
- The Calderón reproducing formula, windowed \(X\)-ray transforms, and Radon transforms in \(L^p\)-spaces
- The ridgelet transform and quasiasymptotic behavior of distributions
- The ridgelet transform of distributions
- Tight frames of k -plane ridgelets and the problem of representing objects that are smooth away from d -dimensional singularities in R n </sup
- Universal approximation bounds for superpositions of a sigmoidal function
Cited in
(47)- A mean-field optimal control formulation of deep learning
- Regression methods in waveform modeling: a comparative study
- Understanding neural networks with reproducing kernel Banach spaces
- Machine learning from a continuous viewpoint. I
- Neural dynamic sliding mode control of nonlinear systems with both matched and mismatched uncertainties
- Distributional extension and invertibility of the \(k\)-plane transform and its dual
- Nonconvex regularization for sparse neural networks
- A global universality of two-layer neural networks with ReLU activations
- A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networks
- Misspecified diffusion models with high-frequency observations and an application to neural networks
- An Interpretive Constrained Linear Model for ResNet and MgNet
- Geometric deep learning for computational mechanics. I: Anisotropic hyperelasticity
- Explicit representations for Banach subspaces of Lizorkin distributions
- Deep reinforcement learning for adaptive mesh refinement
- Deep learning as optimal control problems: models and numerical methods
- Approximation capabilities of neural networks on unbounded domains
- The shearlet transform and asymptotic behavior of Lizorkin distributions
- ReLU networks are universal approximators via piecewise linear or constant functions
- Piecewise linear functions representable with infinite width shallow ReLU neural networks
- Topological properties of the set of functions generated by neural networks of fixed size
- Theory of deep convolutional neural networks. III: Approximating radial functions
- scientific article; zbMATH DE number 7387622 (Why is no real title available?)
- The universal approximation property. Characterization, construction, representation, and existence
- Universal approximation properties for an ODENet and a ResNet: mathematical analysis and numerical experiments
- On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces
- scientific article; zbMATH DE number 7415100 (Why is no real title available?)
- Hilbert C∗-Module for Analyzing Structured Data
- A unified and constructive framework for the universality of neural networks
- Fast generalization error bound of deep learning without scale invariance of activation functions
- Transport analysis of infinitely deep neural network
- Learned query optimizers
- Estimation of agent-based models using Bayesian deep learning approach of BayesFlow
- A survey on modern trainable activation functions
- Theory of deep convolutional neural networks. II: Spherical analysis
- The role of nonpolynomiality in uniform approximation by RBF networks of Hankel translates
- Continuity properties of the shearlet transform and the shearlet synthesis operator on the Lizorkin type spaces
- Towards global neural network abstractions with locally-exact reconstruction
- Banach space representer theorems for neural networks and ridge splines
- Rapid estimation of permeability from digital rock using 3D convolutional neural network
- From kernel methods to neural networks: a unifying variational formulation
- Center manifold analysis of plateau phenomena caused by degeneration of three-layer perceptron
- scientific article; zbMATH DE number 7387620 (Why is no real title available?)
- Symmetry \& critical points for a model shallow neural network
- Heaviside function as an activation function
- Beating a Benchmark: Dynamic Programming May Not Be the Right Numerical Approach
- A deep learning semiparametric regression for adjusting complex confounding structures
- On the double windowed ridgelet transform and its inverse
This page was built for publication: Neural network with unbounded activation functions is universal approximator
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2399647)
