Hyperband: a novel bandit-based approach to hyperparameter optimization
From MaRDI portal
Learning and adaptive systems in artificial intelligence (68T05) Artificial neural networks and deep learning (68T07) Problem solving in the context of artificial intelligence (heuristics, search strategies, etc.) (68T20) Online algorithms; streaming algorithms (68W27) Approximation methods and heuristics in mathematical programming (90C59)
Abstract: Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.
Recommendations
- Fast Bayesian hyperparameter optimization on large datasets
- scientific article; zbMATH DE number 6276119
- An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization
- No-regret Bayesian optimization with unknown hyperparameters
- Hyperparameter tuning methods in automated machine learning
Cites work
- X-armed bandits
- Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems
- Bayesian optimization in a billion dimensions via random embeddings
- Efficient multi-start strategies for local search algorithms
- Fast Bayesian hyperparameter optimization on large datasets
- Fast cross-validation via sequential testing
- scientific article; zbMATH DE number 6276119 (Why is no real title available?)
- In defense of one-vs-all classification
- Information rates of nonparametric Gaussian process methods
- Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
- Nonparametric guidance of autoencoder representations using label information
- On the complexity of best-arm identification in multi-armed bandit models
- Pure exploration in multi-armed bandits problems
Cited in
(68)- ML-plan: automated machine learning via hierarchical planning
- Fast Bayesian hyperparameter optimization on large datasets
- A machine learning approach for efficient uncertainty quantification using multiscale methods
- Scalable Gaussian process-based transfer surrogates for hyperparameter optimization
- Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification
- Bayesian optimization of pump operations in water distribution systems
- Hyperparameter optimization in learning systems
- Data-driven algorithm selection and tuning in optimization and signal processing
- A taxonomy of weight learning methods for statistical relational learning
- Automatic model training under restrictive time constraints
- Use of static surrogates in hyperparameter optimization
- Automated porosity estimation using CT-scans of extracted core data
- Estimating shape parameters of piecewise linear-quadratic problems
- Combining Bayesian optimization and Lipschitz optimization
- Joint detection of malicious domains and infected clients
- Research on spatio-temporal network prediction model of parallel-series traffic flow based on transformer and GCAT
- Local surrogate responses in the Schwarz alternating method for elastic problems on random voided domains
- Neural architecture search: a survey
- scientific article; zbMATH DE number 7255176 (Why is no real title available?)
- Risk bounds for unsupervised cross-domain mapping with IPMs
- Learning and meta-learning of stochastic advection–diffusion–reaction systems from sparse measurements
- SMAC3: a versatile Bayesian optimization package for hyperparameter optimization
- Hyperparameter tuning methods in automated machine learning
- Multiobjective Tree-Structured Parzen Estimator
- Escaping local minima with local derivative-free methods: a numerical investigation
- Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
- HEBO: An Empirical Study of Assumptions in Bayesian Optimisation
- Hyperparameter optimization via sequential uniform designs
- Are we forgetting about compositional optimisers in Bayesian optimisation?
- scientific article; zbMATH DE number 7415127 (Why is no real title available?)
- Best arm identification for contaminated bandits
- Benchmark and survey of automated machine learning frameworks
- An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization
- Computationally efficient integrated design and predictive control of flexible energy systems using multi‐fidelity simulation‐based Bayesian optimization
- Supervised Machine Learning Techniques: An Overview with Applications to Banking
- Investigation of the Lombard effect based on a machine learning approach
- Recurrent and convolutional neural networks in structural dynamics: a modified attention steered encoder-decoder architecture versus LSTM versus GRU versus TCN topologies to predict the response of shock wave-loaded plates
- Accelerated Componentwise Gradient Boosting Using Efficient Data Representation and Momentum-Based Optimization
- Naive automated machine learning
- AutonoML: Towards an Integrated Framework for Autonomous Machine Learning
- Automated Deep Learning: Neural Architecture Search Is Not the End
- Business processes resource management using rewriting logic and deep-learning-based predictive monitoring
- A data-driven explainable case-based reasoning approach for financial risk detection
- Goal-oriented sensitivity analysis of hyperparameters in deep learning
- Improving the efficiency of reinforcement learning for a spacecraft powered descent with Q-learning
- Stochastic adversarial noise in the ``black box optimization problem
- Automatic MILP Solver configuration by learning problem similarities
- Progressively strengthening and tuning MIP solvers for reoptimization
- OpenBox: a Python toolkit for generalized black-box optimization
- Profiling side-channel attacks on Dilithium. A small bit-fiddling leak breaks it all
- The ``black-box optimization problem: zero-order accelerated stochastic method via kernel approximation
- Detecting antimicrobial resistance through MALDI-TOF mass spectrometry with statistical guarantees using conformal prediction
- Explaining random forest and XGBoost with shallow decision trees by co-clustering feature importance
- Hyperparameter optimization in machine learning
- Data-rich economic forecasting for actuarial applications
- Data farming the parameters of simulation-optimization solvers
- A gradient-based bilevel optimization approach for tuning regularization hyperparameters
- Multistart algorithm for identifying all optima of nonconvex stochastic functions
- Train wisely: multifidelity Bayesian optimization hyperparameter tuning in deep learning-based side-channel analysis
- Surrogate-based autotuning for randomized sketching algorithms in regression problems
- Pricing and calibration in the 4-factor path-dependent volatility model
- Multi-objective model for architecture optimization and training of radial basis function neural networks
- Code generation for solving and differentiating through convex optimization problems
- mlr3hyperband
- Lexicographic Lipschitz bandits: new algorithms and a lower bound
- Bayesian optimization with Pareto-principled training for economical hyperparameter optimization
- Bayesian optimization by kernel regression and density-based exploration
- Imbalanced regression and extreme value prediction
This page was built for publication: Hyperband: a novel bandit-based approach to hyperparameter optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q72746)
