Use of static surrogates in hyperparameter optimization
From MaRDI portal
Abstract: Optimizing the hyperparameters and architecture of a neural network is a long yet necessary phase in the development of any new application. This consuming process can benefit from the elaboration of strategies designed to quickly discard low quality configurations and focus on more promising candidates. This work aims at enhancing HyperNOMAD, a library that adapts a direct search derivative-free optimization algorithm to tune both the architecture and the training of a neural network simultaneously, by targeting two keys steps of its execution and exploiting cheap approximations in the form of static surrogates to trigger the early stopping of the evaluation of a configuration and the ranking of pools of candidates. These additions to HyperNOMAD are shown to improve on its resources consumption without harming the quality of the proposed solutions.
Recommendations
- An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization
- Hyperparameter optimization in learning systems
- Hyperband: a novel bandit-based approach to hyperparameter optimization
- Fast Bayesian hyperparameter optimization on large datasets
- Dynamic structure-based neural networks determination for simulation optimization
Cites work
- scientific article; zbMATH DE number 6276119 (Why is no real title available?)
- A Simplex Method for Function Minimization
- Algorithm 856
- Algorithm 909: NOMAD: nonlinear optimization with the MADS algorithm
- Derivative-free and blackbox optimization
- Fast Bayesian hyperparameter optimization on large datasets
- Generalized pattern searches with derivative information
- HyperNOMAD
- Hyperband: a novel bandit-based approach to hyperparameter optimization
- Introduction to Derivative-Free Optimization
- Mesh Adaptive Direct Search Algorithms for Constrained Optimization
- Neural architecture search: a survey
Cited in
(5)- Hyperparameter optimization in learning systems
- A tuning method for bi-directional long short-term memory neural network model based on robust design
- An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization
- Tuning parameters of deep neural network training algorithms pays off: a computational study
- ADVANCED ALGORITHM OF EVOLUTION STRATEGIES OF COVARIATION MATRIX ADAPTATION
Describes a project that uses
Uses Software
This page was built for publication: Use of static surrogates in hyperparameter optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2120124)
