Neural network with unbounded activation functions is universal approximator
From MaRDI portal
Abstract: This paper presents an investigation of the approximation property of neural networks with unbounded activation functions, such as the rectified linear unit (ReLU), which is the new de-facto standard of deep learning. The ReLU network can be analyzed by the ridgelet transform with respect to Lizorkin distributions. By showing three reconstruction formulas by using the Fourier slice theorem, the Radon transform, and Parseval's relation, it is shown that a neural network with unbounded activation functions still satisfies the universal approximation property. As an additional consequence, the ridgelet transform, or the backprojection filter in the Radon domain, is what the network learns after backpropagation. Subject to a constructive admissibility condition, the trained network can be obtained by simply discretizing the ridgelet transform, without backpropagation. Numerical examples not only support the consistency of the admissibility condition but also imply that some non-admissible cases result in low-pass filtering.
Recommendations
- Approximation capabilities of neural networks on unbounded domains
- ReLU networks are universal approximators via piecewise linear or constant functions
- Nonlinear approximation and (deep) ReLU networks
- The universal approximation property. Characterization, construction, representation, and existence
- Error bounds for approximations with deep ReLU networks
Cites work
- A birth and death model of neuron firing
- A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training
- A Sobolev-type upper bound for rates of approximation by linear combinations of Heaviside plane waves
- Approximation by superposition of sigmoidal and radial basis functions
- Classical Fourier Analysis
- Complexity estimates based on integral transforms induced by computational units
- Continuity of the Radon transform and its inverse on Euclidean space
- Convolution-backprojection method for the k-plane transform, and Calderón's identity for ridgelet transforms
- Functional analysis, Sobolev spaces and partial differential equations
- Harmonic analysis of neural networks
- scientific article; zbMATH DE number 741219 (Why is no real title available?)
- scientific article; zbMATH DE number 1022519 (Why is no real title available?)
- scientific article; zbMATH DE number 1405266 (Why is no real title available?)
- scientific article; zbMATH DE number 3204910 (Why is no real title available?)
- scientific article; zbMATH DE number 3240665 (Why is no real title available?)
- scientific article; zbMATH DE number 3272562 (Why is no real title available?)
- scientific article; zbMATH DE number 3329342 (Why is no real title available?)
- scientific article; zbMATH DE number 3187905 (Why is no real title available?)
- Integral geometry and Radon transforms
- Morrey and Campanato meet Besov, Lizorkin and Triebel
- Ridge functions and orthonormal ridgelets
- Sparse image and signal processing. Wavelets, curvelets, morphological diversity
- The Calderón reproducing formula, windowed \(X\)-ray transforms, and Radon transforms in \(L^p\)-spaces
- The ridgelet transform and quasiasymptotic behavior of distributions
- The ridgelet transform of distributions
- Tight frames of k -plane ridgelets and the problem of representing objects that are smooth away from d -dimensional singularities in R n </sup
- Universal approximation bounds for superpositions of a sigmoidal function
Cited in
(58)- Geometric deep learning for computational mechanics. I: Anisotropic hyperelasticity
- Topological properties of the set of functions generated by neural networks of fixed size
- The universal approximation property. Characterization, construction, representation, and existence
- Fast generalization error bound of deep learning without scale invariance of activation functions
- Theory of deep convolutional neural networks. II: Spherical analysis
- Symmetry \& critical points for a model shallow neural network
- Understanding neural networks with reproducing kernel Banach spaces
- Nonconvex regularization for sparse neural networks
- On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces
- Rapid estimation of permeability from digital rock using 3D convolutional neural network
- A global universality of two-layer neural networks with ReLU activations
- Misspecified diffusion models with high-frequency observations and an application to neural networks
- Estimation of agent-based models using Bayesian deep learning approach of BayesFlow
- A deep learning semiparametric regression for adjusting complex confounding structures
- Deep learning as optimal control problems: models and numerical methods
- A mean-field optimal control formulation of deep learning
- Neural dynamic sliding mode control of nonlinear systems with both matched and mismatched uncertainties
- The role of nonpolynomiality in uniform approximation by RBF networks of Hankel translates
- ReLU networks are universal approximators via piecewise linear or constant functions
- Transport analysis of infinitely deep neural network
- Banach space representer theorems for neural networks and ridge splines
- NEU: a meta-algorithm for universal UAP-invariant feature representation
- scientific article; zbMATH DE number 7387620 (Why is no real title available?)
- scientific article; zbMATH DE number 7387622 (Why is no real title available?)
- Center manifold analysis of plateau phenomena caused by degeneration of three-layer perceptron
- The ridgelet prior: a covariance function approach to prior specification for Bayesian neural networks
- Regression methods in waveform modeling: a comparative study
- On the double windowed ridgelet transform and its inverse
- Piecewise linear functions representable with infinite width shallow ReLU neural networks
- Theory of deep convolutional neural networks. III: Approximating radial functions
- Approximation capabilities of neural networks on unbounded domains
- Universal approximation properties for an ODENet and a ResNet: mathematical analysis and numerical experiments
- A survey on modern trainable activation functions
- Continuity properties of the shearlet transform and the shearlet synthesis operator on the Lizorkin type spaces
- Deep reinforcement learning for adaptive mesh refinement
- Explicit representations for Banach subspaces of Lizorkin distributions
- Heaviside function as an activation function
- Beating a Benchmark: Dynamic Programming May Not Be the Right Numerical Approach
- Hilbert C∗-Module for Analyzing Structured Data
- An Interpretive Constrained Linear Model for ResNet and MgNet
- Towards global neural network abstractions with locally-exact reconstruction
- Distributional extension and invertibility of the k-plane transform and its dual
- A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networks
- The shearlet transform and asymptotic behavior of Lizorkin distributions
- A unified and constructive framework for the universality of neural networks
- Learned query optimizers
- From kernel methods to neural networks: a unifying variational formulation
- Universal approximation with complex-valued deep narrow neural networks
- Fredholm neural networks
- A Global-in-Time Neural Network Approach to Dynamic Portfolio Optimization
- A deep neural network two-part model and feature importance test for semicontinuous data
- Approximation of functions of several variables by deep operators activated by sigmoidal functions and rectified power units
- Neural fractional differential equations
- Approximation of neural network operators based on the Joukowski transformation
- Universal approximation results for neural networks with non-polynomial activation function over non-compact domains
- Approximate oblique dual frames in separable Q-Hilbert spaces
- The statistical physics of learning in under- and over-parameterized layered neural networks
- Machine learning from a continuous viewpoint. I
This page was built for publication: Neural network with unbounded activation functions is universal approximator
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2399647)
