Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training
From MaRDI portal
complexitydeep learningmultilevel methodsneural networksnonlinear optimizationobjective-function-free optimization (OFFO)
Numerical mathematical programming methods (65K05) Multigrid methods; domain decomposition for initial value and initial-boundary value problems involving PDEs (65M55) Artificial neural networks and deep learning (68T07) Large-scale problems in mathematical programming (90C06) Nonconvex programming, global optimization (90C26) Nonlinear programming (90C30)
Abstract: A class of multi-level algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multi-level feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behaviour in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.
Recommendations
- Convergence Properties of an Objective-Function-Free Optimization Regularization Algorithm, Including an \(\boldsymbol{\mathcal{O}(\epsilon^{-3/2})}\) Complexity Bound
- Novel algorithms for noisy minimization problems with applications to neural networks training
- scientific article; zbMATH DE number 6514307
- Stochastic multilevel composition optimization algorithms with level-independent convergence rates
- Data-driven nonsmooth optimization
Cites work
- A globalization strategy for the multigrid solution of elliptic optimal control problems
- A multigrid approach to discretized optimization problems
- A Multigrid Tutorial, Second Edition
- A note about sparsity exploiting quasi-Newton updates
- A proposal on machine learning via dynamical systems
- A Stochastic Approximation Method
- Adaptive subgradient methods for online learning and stochastic optimization
- An overview of first-order model management for engineering optimization
- Convergence Properties of an Objective-Function-Free Optimization Regularization Algorithm, Including an \(\boldsymbol{\mathcal{O}(\epsilon^{-3/2})}\) Complexity Bound
- Distributed block coordinate descent for minimizing partially separable functions
- Evaluation complexity of algorithms for nonconvex optimization. Theory, computation and perspectives
- Globally Convergent Multilevel Training of Deep Residual Networks
- scientific article; zbMATH DE number 3898623 (Why is no real title available?)
- scientific article; zbMATH DE number 3725604 (Why is no real title available?)
- scientific article; zbMATH DE number 107545 (Why is no real title available?)
- scientific article; zbMATH DE number 2068050 (Why is no real title available?)
- scientific article; zbMATH DE number 1561761 (Why is no real title available?)
- Layer-Parallel Training of Deep Residual Neural Networks
- Model Problems for the Multigrid Optimization of Systems Governed by Differential Equations
- Multilevel minimization for deep residual networks
- Numerical experience with a recursive trust-region method for multilevel nonlinear bound-constrained optimization
- On the Lambert \(w\) function
- Optimization problems involving group sparsity terms
- Recent advances in trust region algorithms
- Recursive Trust-Region Methods for Multiscale Nonlinear Optimization
- Stable architectures for deep neural networks
- Trust Region Methods
Cited in
(9)- Learning optimization of neural networks used for MIMO applications based on multivariate functions decomposition
- scientific article; zbMATH DE number 988747 (Why is no real title available?)
- Approximating the objective function's gradient using perceptrons for constrained minimization with application in drag reduction
- Multilevel Fine-Tuning: Closing Generalization Gaps in Approximation of Solution Maps under a Limited Budget for Training Data
- Multilevel‐in‐width training for deep neural network regression
- Enhancing training of physics-informed neural networks using domain decomposition-based preconditioning strategies
- Data-parallel neural network training via nonlinearly preconditioned trust-region method
- prunAdag: an adaptive pruning-aware gradient method
- Objective-function free multi-objective optimization: rate of convergence and performance of an Adagrad-like algorithm
This page was built for publication: Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6076869)
