Approximating smooth functions by deep neural networks with sigmoid activation function

DOI10.1016/J.JMVA.2020.104696MaRDI QIDQ2222228zbMATH OpenOpenAlexFDO

Authors Sophie Langer

Publication date 26 January 2021

Published in Journal of Multivariate Analysis (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2010.04596

zbMATH Keywords

deep learningfull connectivityneural networksuniform approximation

Mathematics Subject Classification ID

Rate of convergence, degree of approximation (41A25) Neural nets applied to problems in time-dependent statistical mechanics (82C32)

Abstract: We study the power of deep neural networks (DNNs) with sigmoid activation function. Recently, it was shown that DNNs approximate any $d$-dimensional, smooth function on a compact set with a rate of order $W^{-p/d}$, where $W$ is the number of nonzero weights in the network and $p$ is the smoothness of the function. Unfortunately, these rates only hold for a special class of sparsely connected DNNs. We ask ourselves if we can show the same approximation rate for a simpler and more general class, i.e., DNNs which are only defined by its width and depth. In this article we show that DNNs with fixed depth and a width of order $M^d$ achieve an approximation rate of $M^{-2p}$. As a conclusion we quantitatively characterize the approximation power of DNNs in terms of the overall weights $W_0$ in the network and show an approximation rate of $W_0^{-p/d}$. This more general result finally helps us to understand which network topology guarantees a special target accuracy.

Recommendations

• Deep Network Approximation for Smooth Functions
• Approximation of smooth functionals using deep ReLU networks
• Approximation by neural networks with sigmoidal functions
• Learning sparse and smooth functions by deep sigmoid nets
• Simultaneous neural network approximation for smooth functions
• Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations
• Function approximation by deep networks
• Approximation by neural networks with a generalized sigmoidal function
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Approximating smooth and sparse functions by deep neural networks: optimal approximation rates and saturation

Cites work

• Approximation by superpositions of a sigmoidal function
• Approximation of functions and their derivatives: A neural network implementation with applications
• Multilayer feedforward networks are universal approximators
• Nonparametric regression using deep neural networks with ReLU activation function
• On deep learning as a remedy for the curse of dimensionality in nonparametric regression
• Optimal global rates of convergence for noiseless regression estimation problems with adaptively chosen design
• Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units

Cited in

(20)

• Neural network with smooth activation functions and without bottlenecks is almost surely a Morse function
• Analysis of the rate of convergence of fully connected deep neural network regression estimates with smooth activation function
• Estimation of a regression function on a manifold by fully connected deep neural networks
• The gap between theory and practice in function approximation with deep neural networks
• A note on the applications of one primary function in deep neural networks
• Expressivity of Deep Neural Networks
• Simultaneous neural network approximation for smooth functions
• Neural Networks for Clustered and Longitudinal Data Using Mixed Effects Models
• Approximation error for neural network operators by an averaged modulus of smoothness
• Learning sparse and smooth functions by deep sigmoid nets
• Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations
• Research on cloud data encryption algorithm based on bidirectional activation neural network
• Neural network expression rates and applications of the deep parametric PDE method in counterparty credit risk
• Learning and approximating piecewise smooth functions by deep sigmoid neural networks
• Deep learning for Markov chains: Lyapunov functions, Poisson's equation, and stationary distributions
• Analysis of structured deep kernel networks
• On the rate of convergence of an over-parametrized deep neural network regression estimate with ReLU activation function learned by gradient descent
• Fitted value iteration methods for bicausal optimal transport
• Approximation of functions of several variables by deep operators activated by sigmoidal functions and rectified power units
• Frequency-adaptive multi-scale deep neural networks

This page was built for publication: Approximating smooth functions by deep neural networks with sigmoid activation function

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2222228)