Approximation of smoothness classes by deep rectifier networks

DOI10.1137/20M1360657MaRDI QIDQ5020751zbMATHFDO

Authors Mazen Ali, A. Nouy

Publication date 7 January 2022

Published in SIAM Journal on Numerical Analysis (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2007.15645

zbMATH Keywords

Besov spacesapproximation spacesdirect (Jackson) inequalitiesdirect embeddingsReLU neural networks

Mathematics Subject Classification ID

Numerical computation using splines (65D07) Artificial neural networks and deep learning (68T07) Spline approximation (41A15) Approximation by other special function classes (41A30)

Abstract: We consider approximation rates of sparsely connected deep rectified linear unit (ReLU) and rectified power unit (RePU) neural networks for functions in Besov spaces $B^{a}lph{a}_q(L^p)$ in arbitrary dimension $d$, on general domains. We show that alert{deep rectifier} networks with a fixed activation function attain optimal or near to optimal approximation rates for functions in the Besov space $B^{a}lph{a}_{au}(L^{a}u)$ on the critical embedding line $1/au=alpha/d+1/p$ for emph{arbitrary} smoothness order $alpha>0$. Using interpolation theory, this implies that the entire range of smoothness classes at or above the critical line is (near to) optimally approximated by deep ReLU/RePU networks.

Recommendations

• Better approximations of high dimensional smooth functions by deep neural networks with rectified power units
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Approximation spaces of deep neural networks
• Provable approximation properties for deep neural networks
• Error bounds for approximations with deep ReLU networks

Cites work

• scientific article; zbMATH DE number 1215245 (Why is no real title available?)
• scientific article; zbMATH DE number 2001584 (Why is no real title available?)
• scientific article; zbMATH DE number 278904 (Why is no real title available?)
• scientific article; zbMATH DE number 2208228 (Why is no real title available?)
• scientific article; zbMATH DE number 3329342 (Why is no real title available?)
• Besov Spaces on Domains in ℝ d
• Biorthogonal bases of compactly supported wavelets
• Deep ReLU networks and high-order finite element methods
• Error bounds for approximations with deep ReLU networks
• Maximal functions measuring smoothness
• On nonlinear 𝑛-widths
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Optimal approximation with sparsely connected deep neural networks
• Optimal nonlinear approximation
• Orthogonal Polynomials and the Construction of Piecewise Polynomial Smooth Wavelets
• Quasiconformal mappings and extendability of functions in Sobolev spaces
• Wavelet coefficients measuring smoothness in \(H_ p(\mathbb{R}^ d)\)
• \(H^p\) spaces of several variables

Cited in

(14)

• Weighted variation spaces and approximation by shallow ReLU networks
• Collocation approximation by deep neural ReLU networks for parametric and stochastic PDEs with lognormal inputs
• Optimal approximation with sparsely connected deep neural networks
• Neural network approximation
• Deep ReLU neural network approximation in Bochner spaces and applications to parametric PDEs
• High-order approximation rates for shallow neural networks with cosine and \(\mathrm{ReLU}^k\) activation functions
• PowerNet: efficient representations of polynomials and smooth functions by deep neural networks with rectified power units
• Better approximations of high dimensional smooth functions by deep neural networks with rectified power units
• Approximation theory of tree tensor networks: tensorized univariate functions
• Convergence rates of deep ReLU networks for multiclass classification
• Sobolev-type embeddings for neural network approximation spaces
• Integral representations of shallow neural network with Rectified Power Unit activation function
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Extracting a function encoded in amplitudes of a quantum state by tensor network and orthogonal function expansion

This page was built for publication: Approximation of smoothness classes by deep rectifier networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5020751)