Best k-layer neural network approximations

DOI10.1007/S00365-021-09545-2MaRDI QIDQ2117342zbMATH OpenOpenAlexWikidataFDO

Authors Lek-Heng Lim, Mateusz Michalek, Yang Qi

Publication date 21 March 2022

Published in Constructive Approximation (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1907.01507

zbMATH Keywords

best approximationneural networksecant locijoin loci

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Neural networks for/in biological studies, artificial life and related topics (92B20) Approximation by other special function classes (41A30) Best approximation, Chebyshev systems (41A50)

Abstract: We show that the empirical risk minimization (ERM) problem for neural networks has no solution in general. Given a training set $s_1,dots,s_{n}inmathbb{R}^p$ with corresponding responses $t_1,dots,t_{n}inmathbb{R}^q$, fitting a $k$-layer neural network $u_{h}eta:mathbb{R}^{p}omathbb{R}^q$ involves estimation of the weights $hetainmathbb{R}^m$ via an ERM: [ inf_{ heta in mathbb{R}^m} ; sum_{i=1}^n lVert t_i - u_ heta(s_i) Vert_2^2. ] We show that even for $k=2$, this infimum is not attainable in general for common activations like ReLU, hyperbolic tangent, and sigmoid functions. A high-level explanation is like that for the nonexistence of best rank-$r$ approximations of higher-order tensors --- the set of parameters is not a closed set --- but the geometry involved for best $k$-layer neural networks approximations is more subtle. In addition, we show that for smooth activations $$ and $sigma(x)=anh(x)$, such failure to attain an infimum can happen on a positive-measured subset of responses. For the ReLU activation $sigma(x)=max(0,x)$, we completely classifying cases where the ERM for a best two-layer neural network approximation attains its infimum. As an aside, we obtain a precise description of the geometry of the space of two-layer neural networks with $d$ neurons in the hidden layer: it is the join locus of a line and the $d$-secant locus of a cone.

Recommendations

• Critical points for least-squares problems involving certain analytic functions, with applications to sigmoidal nets
• Hardness results for neural network approximation problems
• Over-parametrized deep neural networks minimizing the empirical risk do not generalize well
• Error bounds for approximations with deep ReLU networks
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks

Cites work

• scientific article; zbMATH DE number 5968745 (Why is no real title available?)
• scientific article; zbMATH DE number 436950 (Why is no real title available?)
• Approximation by superpositions of a sigmoidal function
• Complex best \(r\)-term approximations almost always exist in finite dimensions
• Condition. The geometry of numerical algorithms
• Machine learning: from theory to applications. Cooperative research at Siemens and MIT
• Multilayer feedforward networks are universal approximators
• Networks and the best approximation property
• Tensor Rank and the Ill-Posedness of the Best Low-Rank Approximation Problem
• Topological properties of the set of functions generated by neural networks of fixed size
• Training neural networks with noisy data as an ill-posed problem

Cited in

(3)

• Neural networks with linear threshold activations: structure and algorithms
• On the existence of minimizers in shallow residual ReLU neural network optimization landscapes
• Analysis of a two-layer neural network via displacement convexity

Describes a project that uses

Uses Software

• ImageNet
• AlexNet

This page was built for publication: Best \(k\)-layer neural network approximations

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2117342)