Approximation of smoothness classes by deep rectifier networks

DOI10.1137/20M1360657zbMATH Open1494.41008arXiv2007.15645MaRDI QIDQ5020751FDOQ5020751

Mazen Ali, A. Nouy

Publication date: 7 January 2022

Published in: SIAM Journal on Numerical Analysis (Search for Journal in Brave)

Abstract: We consider approximation rates of sparsely connected deep rectified linear unit (ReLU) and rectified power unit (RePU) neural networks for functions in Besov spaces $B^{a}lph{a}_q(L^p)$ in arbitrary dimension $d$, on general domains. We show that alert{deep rectifier} networks with a fixed activation function attain optimal or near to optimal approximation rates for functions in the Besov space $B^{a}lph{a}_{au}(L^{a}u)$ on the critical embedding line $1/au=alpha/d+1/p$ for emph{arbitrary} smoothness order $alpha>0$. Using interpolation theory, this implies that the entire range of smoothness classes at or above the critical line is (near to) optimally approximated by deep ReLU/RePU networks.

Full work available at URL: https://arxiv.org/abs/2007.15645

Recommendations

• Better approximations of high dimensional smooth functions by deep neural networks with rectified power units
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Approximation spaces of deep neural networks
• Provable approximation properties for deep neural networks
• Error bounds for approximations with deep ReLU networks

zbMATH Keywords

Besov spacesapproximation spacesdirect (Jackson) inequalitiesdirect embeddingsReLU neural networks

Mathematics Subject Classification ID

Numerical computation using splines (65D07) Artificial neural networks and deep learning (68T07) Spline approximation (41A15) Approximation by other special function classes (41A30)

Cites Work

• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• \(H^p\) spaces of several variables
• Biorthogonal bases of compactly supported wavelets
• Quasiconformal mappings and extendability of functions in Sobolev spaces
• Besov Spaces on Domains in ℝ d
• Title not available (Why is that?)
• Title not available (Why is that?)
• Optimal nonlinear approximation
• Wavelet coefficients measuring smoothness in \(H_ p(\mathbb{R}^ d)\)
• Maximal functions measuring smoothness
• Orthogonal Polynomials and the Construction of Piecewise Polynomial Smooth Wavelets
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Error bounds for approximations with deep ReLU networks
• Optimal Approximation with Sparsely Connected Deep Neural Networks
• On nonlinear 𝑛-widths
• Deep ReLU networks and high-order finite element methods

Cited In (5)

• Collocation approximation by deep neural ReLU networks for parametric and stochastic PDEs with lognormal inputs
• Approximation theory of tree tensor networks: tensorized univariate functions
• Deep ReLU neural network approximation in Bochner spaces and applications to parametric PDEs
• Neural network approximation
• Extracting a function encoded in amplitudes of a quantum state by tensor network and orthogonal function expansion

This page was built for publication: Approximation of smoothness classes by deep rectifier networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5020751)