Neural network approximation of continuous functions in high dimensions with applications to inverse problems (Q6056231)

The authors use neural networks to approximate Hölder continuous functions in high dimensions. It is shown that the size of the network only needs to grow exponentially with respect to the intrinsic complexity of the input set measured using either its Gaussian width or its covering numbers. Therefore, it is often more optimistic than previous estimates that require the size of the network to grow exponentially with respect to the extrinsic input dimension. The approach is based on the observation that the existence of a linear Johnson-Lindenstrauss (J-L) embedding \(\mathbf{A}\in \mathbb{R}^{d\times D}\) of a given high-dimensional set \(\mathcal{S}\subset \mathbb{R}^{D}\) into a low dimensional cube \([-M,M]^d\) implies that for any Hölder (or uniformly) continuous function \(f:\mathcal{S}\to \mathbb{R}^p\), there exists a Hölder (or uniformly) continuous function \(g:[-M,M]^d\to\mathbb{R}^p\) such that \(g(\mathbf{A}\mathbf{x})=f(\mathbf{x})\) for all \(\mathbf{x}\in \mathcal{S}\). Hence, if one has a neural network which approximates \(g:[-M,M]^d\to\mathbb{R}^p\), then a layer can be added that implements the J-L embedding \(\mathbf{A}\) to obtain a neural network that approximates \(f:\mathcal{S}\to \mathbb{R}^p\). By pairing J-L embedding results along with results on approximation of Hölder (or uniformly) continuous functions by neural networks, one then obtains results which bound the complexity required for a neural network to approximate Hölder (or uniformly) continuous functions on high dimensional sets. Finally, the authors focused on inverse problems and demonstrated how the main theorems can be used to provide a reasonable estimate of the size of the neural networks needed to solve some classical inverse problems in signal processing.