Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class

DOI10.1137/21M144431XMaRDI QIDQ6137593zbMATH OpenOpenAlexFDO

Authors Yu Ling Jiao, Y. M. Lai, Xiliang Lu, Fengru Wang, Jerry Zhijian Yang, Yuanyuan Yang

Publication date 4 September 2023

Published in SIAM Journal on Mathematical Analysis (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2103.00542

zbMATH Keywords

curse of dimensionalityapproximationdeep neural networkHölder continuous function

Mathematics Subject Classification ID

Multidimensional problems (41A63) Approximation by other special function classes (41A30) Algorithms for approximation of functions (65D15)

Abstract: In this paper, we construct neural networks with ReLU, sine and $2^x$ as activation functions. For general continuous $f$ defined on $[0,1{]}^d$ with continuity modulus $omeg{a}_f(cdot)$, we construct ReLU-sine-$2^x$ networks that enjoy an approximation rate $mathcalO(omeg{a}_f(sqrtd)cdot{2}^{-M}+omeg{a}_{f}left(fracsqrtdNight))$, where $M,Ninmathbb{N}^{+}$ denote the hyperparameters related to widths of the networks. As a consequence, we can construct ReLU-sine-$2^x$ network with the depth $5$ and width $maxleftleftlceil2d^{3/2}left(frac3muepsilonight{)}^{1/alpha}ightceil,2leftlceillo{g}_{2}frac3mu{d}^{alpha/2}2epsilonightceil+2ight$ that approximates $finmathcal{H}_{mu}^{alpha}([0,1{]}^d)$ within a given tolerance $epsilon>0$ measured in $L^p$ norm $pin[1,infty)$, where $mathcal{H}_{mu}^{alpha}([0,1{]}^d)$ denotes the H"older continuous function class defined on $[0,1{]}^d$ with order $alphain(0,1]$ and constant $mu>0$. Therefore, the ReLU-sine-$2^x$ networks overcome the curse of dimensionality on $mathcal{H}_{mu}^{alpha}([0,1{]}^d)$. In addition to its supper expressive power, functions implemented by ReLU-sine-$2^x$ networks are (generalized) differentiable, enabling us to apply SGD to train.

Recommendations

• Deep ReLU neural networks in high-dimensional approximation
• Deep network with approximation error being reciprocal of width to power of square root of depth
• Neural network approximation: three hidden layers are enough
• Deep ReLU networks and high-order finite element methods
• Optimal approximation rate of ReLU networks in terms of width and depth

Cites work

• scientific article; zbMATH DE number 46303 (Why is no real title available?)
• scientific article; zbMATH DE number 1405266 (Why is no real title available?)
• scientific article; zbMATH DE number 7306873 (Why is no real title available?)
• A Stochastic Approximation Method
• Approximation by superpositions of a sigmoidal function
• Approximation rates for neural networks with general activation functions
• Approximation spaces of deep neural networks
• Deep Network Approximation for Smooth Functions
• Deep ReLU Networks Overcome the Curse of Dimensionality for Generalized Bandlimited Functions
• Deep network approximation characterized by number of neurons
• Deep network with approximation error being reciprocal of width to power of square root of depth
• Error bounds for approximations with deep ReLU networks
• Error bounds for approximations with deep ReLU neural networks in \(W^{s , p}\) norms
• Error bounds for deep ReLU networks using the Kolmogorov-Arnold superposition theorem
• Exponential ReLU DNN expression of holomorphic maps in high dimension
• Exponential convergence of the deep neural network approximation for analytic functions
• Multilayer feedforward networks are universal approximators
• Neural Network Learning
• Neural network approximation: three hidden layers are enough
• New error bounds for deep ReLU networks using sparse grids
• Nonparametric regression using deep neural networks with ReLU activation function
• On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Provable approximation properties for deep neural networks
• Universal approximation bounds for superpositions of a sigmoidal function

Cited in

(10)

• SignReLU neural network and its approximation ability
• Exponential ReLU DNN expression of holomorphic maps in high dimension
• Deep ReLU Networks Overcome the Curse of Dimensionality for Generalized Bandlimited Functions
• Analysis of the rate of convergence of two regression estimates defined by neural features which are easy to implement
• A semiparametric Gaussian mixture model with spatial dependence and its application to whole-slide image clustering analysis
• Finite expression method for solving high-dimensional partial differential equations
• DRM revisited: a complete error analysis
• Layer separation deep learning model with auxiliary variables for partial differential equations
• Operator theory of diffusion equations governed by self-similar measures
• Why shallow networks struggle to approximate and learn high frequencies

This page was built for publication: Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6137593)