Critical points for least-squares problems involving certain analytic functions, with applications to sigmoidal nets (Q1923891)

This paper deals with nonlinear least squares problems involving the fitting to data of parametrized analytic functions. For generic regression data, a general result establishes the countability, and under stronger assumptions finiteness, of the set of functions giving rise to critical points of the quadratic loss functions. In the special case of usually called ``single hidden layer neural networks'', which are built upon the standard sigmoidal activation \(\text{tang}(x)\) (or equivalently \((1+e^{-x})^{-1}\)), a rough upper bound for this cardinality is established as well. One of the main results given in this paper is that the set of critical points is finite, and in particular less than \(2^{8(NK)^2}\) (assuming that there are enough samples, specially that \(N\geq 2K(m+2)+3\)). The finiteness results can be generalized to more general networks and in fact many of the intermediate results apply equally well to completely general least squares problems involving analytic functions.