Approximation of BV functions by neural networks: A regularity theory approach

arXiv2012.08291MaRDI QIDQ6356161FDOQ6356161

Publication date: 15 December 2020

Abstract: In this paper we are concerned with the approximation of functions by single hidden layer neural networks with ReLU activation functions on the unit circle. In particular, we are interested in the case when the number of data-points exceeds the number of nodes. We first study the convergence to equilibrium of the stochastic gradient flow associated with the cost function with a quadratic penalization. Specifically, we prove a Poincar'e inequality for a penalized version of the cost function with explicit constants that are independent of the data and of the number of nodes. As our penalization biases the weights to be bounded, this leads us to study how well a network with bounded weights can approximate a given function of bounded variation (BV). Our main contribution concerning approximation of BV functions, is a result which we call the localization theorem. Specifically, it states that the expected error of the constrained problem, where the length of the weights are less than

R

, is of order

R^{- 1 / 9}

with respect to the unconstrained problem (the global optimum). The proof is novel in this topic and is inspired by techniques from regularity theory of elliptic partial differential equations. Finally we quantify the expected value of the global optimum by proving a quantitative version of the universal approximation theorem.

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Stochastic ordinary differential equations (aspects of stochastic analysis) (60H10) Estimates of eigenvalues in context of PDEs (35P15) Fokker-Planck equations (35Q84) Parabolic equations and parabolic systems (35K99)

This page was built for publication: Approximation of BV functions by neural networks: A regularity theory approach

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6356161)