Block layer decomposition schemes for training deep neural networks
From MaRDI portal
Publication:2173515
Abstract: Deep Feedforward Neural Networks' (DFNNs) weights estimation relies on the solution of a very large nonconvex optimization problem that may have many local (no global) minimizers, saddle points and large plateaus. As a consequence, optimization algorithms can be attracted toward local minimizers which can lead to bad solutions or can slow down the optimization process. Furthermore, the time needed to find good solutions to the training problem depends on both the number of samples and the number of variables. In this work, we show how Block Coordinate Descent (BCD) methods can be applied to improve performance of state-of-the-art algorithms by avoiding bad stationary points and flat regions. We first describe a batch BCD method ables to effectively tackle the network's depth and then we further extend the algorithm proposing a extit{minibatch} BCD framework able to scale with respect to both the number of variables and the number of samples by embedding a BCD approach into a minibatch framework. By extensive numerical results on standard datasets for several architecture networks, we show how the application of BCD methods to the training phase of DFNNs permits to outperform standard batch and minibatch algorithms leading to an improvement on both the training phase and the generalization performance of the networks.
Recommendations
- Block-cyclic stochastic coordinate descent for deep neural networks
- Distributed Newton Methods for Deep Neural Networks
- Exploring strategies for training deep neural networks
- Learning a deep convolutional neural network via tensor decomposition
- Multicomposite nonconvex optimization for training deep neural networks
Cites work
- A Stochastic Approximation Method
- Adaptive subgradient methods for online learning and stochastic optimization
- An incremental decomposition method for unconstrained optimization
- Convergent decomposition techniques for training RBF neural networks
- Coordinate descent algorithms
- Deep learning
- Efficiency of coordinate descent methods on huge-scale optimization problems
- Efficient block-coordinate descent algorithms for the group Lasso
- Global optimization issues in deep network regression: an overview
- Globally convergent block-coordinate techniques for unconstrained optimization
- Gradient Convergence in Gradient methods with Errors
- scientific article; zbMATH DE number 5060482 (Why is no real title available?)
- Incremental Least Squares Methods and the Extended Kalman Filter
- Large-scale machine learning with stochastic gradient descent
- On the convergence of block coordinate descent type methods
- Optimization methods for large-scale machine learning
Cited in
(4)
This page was built for publication: Block layer decomposition schemes for training deep neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2173515)
