Piecewise convexity of artificial neural networks
From MaRDI portal
Publication:2292218
Abstract: Although artificial neural networks have shown great promise in applications including computer vision and speech recognition, there remains considerable practical and theoretical difficulty in optimizing their parameters. The seemingly unreasonable success of gradient descent methods in minimizing these non-convex functions remains poorly understood. In this work we offer some theoretical guarantees for networks with piecewise affine activation functions, which have in recent years become the norm. We prove three main results. Firstly, that the network is piecewise convex as a function of the input data. Secondly, that the network, considered as a function of the parameters in a single layer, all others held constant, is again piecewise convex. Finally, that the network as a function of all its parameters is piecewise multi-convex, a generalization of biconvexity. From here we characterize the local minima and stationary points of the training objective, showing that they minimize certain subsets of the parameter space. We then analyze the performance of two optimization algorithms on multi-convex problems: gradient descent, and a method which repeatedly solves a number of convex sub-problems. We prove necessary convergence conditions for the first algorithm and both necessary and sufficient conditions for the second, after introducing regularization to the objective. Finally, we remark on the remaining difficulty of the global optimization problem. Under the squared error objective, we show that by varying the training data, a single rectifier neuron admits local minima arbitrarily far apart, both in objective value and parameter space.
Recommendations
- Optimization Landscape of Neural Networks
- Gradient descent optimizes over-parameterized deep ReLU networks
- The global optimization geometry of shallow linear neural networks
- Analysis of a two-layer neural network via displacement convexity
- Breaking the curse of dimensionality with convex neural networks
Cites work
- An Integral Upper Bound for Neural Network Approximation
- Bi-convexity and bi-martingales
- Biconvex sets and optimization with biconvex functions: a survey and extensions
- Complex-valued autoencoders
- Estimates of the approximation error using Rademacher complexity: Learning vector-valued functions
- scientific article; zbMATH DE number 3704885 (Why is no real title available?)
- scientific article; zbMATH DE number 3310599 (Why is no real title available?)
- scientific article; zbMATH DE number 3365044 (Why is no real title available?)
- Introductory lectures on convex optimization. A basic course.
- Max-min representation of piecewise linear functions
- Minimization of a Non-Separable Objective Function Subject to Disjoint Constraints
- On the convergence properties of the projected gradient method for convex optimization
- Piecewise-convex maximization problems.
- Uniform approximation by neural networks
- Universal approximation bounds for superpositions of a sigmoidal function
Cited in
(8)- Estimates for the neural network operators of the max-product type with continuous and \(p\)-integrable functions
- Approximation results in Orlicz spaces for sequences of Kantorovich MAX-product neural network operators
- Exploiting layerwise convexity of rectifier networks with sign constrained weights
- Convex and concave envelopes of artificial neural network activation functions for deterministic global optimization
- scientific article; zbMATH DE number 7626727 (Why is no real title available?)
- Breaking the curse of dimensionality with convex neural networks
- Stabilization of highly nonlinear hybrid neutral stochastic neural networks with time-varying delays by variable-delay feedback control
- Data-driven robust optimization using deep neural networks
This page was built for publication: Piecewise convexity of artificial neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2292218)
