scientific article; zbMATH DE number 269846
From the mathematical viewpoint, a neural network (or, to be precise, a particular type of neural network that an author considers) is a function that can be represented as a following finite sum: \(f(x_ 1,\dots,x_ d)=\sum_ i W_ i g(\sum_ j w_{ij} x_ j-w_{0j})-W_ 0\), where \(w_{ij}\) and \(W_ i\) are real numbers (called weights), and \(g\) is a fixed function (called activation function). Neural networks are used to learn from examples. To be more precise, we start with finitely many cases in which we know both the input \(x_ 1^{(j)},\dots,x_ d^{(j)}\) and the desired output \(y^{(j)}\), and we try to find the values of the weights for which for \(j\)th input, the neural network generates the result that either coincides with \(j\)th output, or if it is impossible, is at least close to \(y^{(j)}\). In other words, neural network is used as a tool for extrapolation and interpolation. The reason why this particular tool was designed is because we humans are often good in extrapolating real-life events, and the above formula is an attempt to describe how cells (called neurons) that form the human brain work (hence the name of this tool). The number of terms in the sum corresponds (in biological terms) to the number of neurons that this particular neural network simulates. Sometimes, this tool is very efficient; sometimes, it is not. One of the possible reasons for inefficiency is that the dependency between \(x_ i\) and \(y\) is extremely complicated, and for a chosen number of terms in the sum, we cannot get a good approximation of the desired dependency \(F\) (for which \(y^{(j)}+F(x_ 1^{(j)},\dots,x_ d^{(j)}))\). In this case, a natural solution is to add more terms to the sum (in biological terms, to add more neurons). Thus, we will get more parameters to vary, and hopefully, by varying them, we will be able to approximate a given (unknown) function \(F\) better. The general feeling in numerical mathematics (supported by many theorems) is that if we get enough parameters, then we will be able to approximate an arbitrary function on a compact set with a given accuracy. This is true for polynomials, for trigonometric polynomials, etc. It has also been proved to be true for neural networks. As an example of this extrapolation process, we can take what happens when we see a movie: we take as input a discrete set of images (25 to 30 per second), and extrapolate it to a smooth trajectory. We are not only able to describe pretty accurately how a body moved, but we can also describe with a reasonable accuracy with what speed it was moving. In other words, our extrapolation not only approximates the original function, but it also approximates its time derivative. Is this a universal phenomenon? I.e., is it possible, given a smooth function \(F\) on a compact \(K\), and a real number \(\varepsilon>0\), to approximate \(F\) with a neural network \(f\) in such a way that not only \(f\) is \(\varepsilon\)-close to \(F\), but also the derivatives of \(f\) (up to a given order \(k\)) are \(\varepsilon\)-close to the derivatives of \(F\)? The positive answer to this question was first proved by \textit{K. Hornik}, \textit{M. Stinchcombe} and \textit{H. White} [Universal approximation of a unknown mapping and its derivative using multilayer feedforward networks, Neural Network 3(5), 551-560 (1990)]. Their proof is however, a pure existence proof and is thus not constructive. The author gives a more constructive proof in the sense that he explains how, if we know \(F\) and \(\varepsilon\), we can actually determine the weights of the desired approximating network.
- Simultaneous approximations of multivariate functions and their derivatives by neural networks with one hidden layer
- Constructive approximate interpolation by neural networks
- Neural networks for optimal approximation of continuous functions in \(\mathbb R^d\)
- Simultaneous Approximations of Polynomials and Derivatives and Their Applications to Neural Networks
- Pointwise approximation for neural networks
- Neural networks for optimal approximation of continuous functions on the unit sphere
- Differentiable approximation by means of the Radon transformation and its applications to neural networks
- Nonlinearity creates linear independence
- On the antiderivatives of \(x^p/(1 - x)\) with an application to optimize loss functions for classification with neural networks
- Random projection neural networks of best approximation: convergence theory and practical applications
This page was built for publication:
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5288697)