Hyperband: a novel bandit-based approach to hyperparameter optimization
From MaRDI portal
Learning and adaptive systems in artificial intelligence (68T05) Artificial neural networks and deep learning (68T07) Problem solving in the context of artificial intelligence (heuristics, search strategies, etc.) (68T20) Online algorithms; streaming algorithms (68W27) Approximation methods and heuristics in mathematical programming (90C59)
Abstract: Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.
Recommendations
- Fast Bayesian hyperparameter optimization on large datasets
- scientific article; zbMATH DE number 6276119
- An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization
- No-regret Bayesian optimization with unknown hyperparameters
- Hyperparameter tuning methods in automated machine learning
Cites work
- scientific article; zbMATH DE number 6276119 (Why is no real title available?)
- Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems
- Bayesian optimization in a billion dimensions via random embeddings
- Efficient multi-start strategies for local search algorithms
- Fast Bayesian hyperparameter optimization on large datasets
- Fast cross-validation via sequential testing
- In defense of one-vs-all classification
- Information rates of nonparametric Gaussian process methods
- Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
- Nonparametric guidance of autoencoder representations using label information
- On the complexity of best-arm identification in multi-armed bandit models
- Pure exploration in multi-armed bandits problems
- \(X\)-armed bandits
Cited in
(53)- The ``black-box optimization problem: zero-order accelerated stochastic method via kernel approximation
- OpenBox: a Python toolkit for generalized black-box optimization
- Profiling side-channel attacks on Dilithium. A small bit-fiddling leak breaks it all
- Automatic MILP Solver configuration by learning problem similarities
- Stochastic adversarial noise in the ``black box optimization problem
- Progressively strengthening and tuning MIP solvers for reoptimization
- scientific article; zbMATH DE number 7415103 (Why is no real title available?)
- scientific article; zbMATH DE number 7415127 (Why is no real title available?)
- Goal-oriented sensitivity analysis of hyperparameters in deep learning
- Business processes resource management using rewriting logic and deep-learning-based predictive monitoring
- Data-driven algorithm selection and tuning in optimization and signal processing
- ML-plan: automated machine learning via hierarchical planning
- Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification
- Supervised Machine Learning Techniques: An Overview with Applications to Banking
- Investigation of the Lombard effect based on a machine learning approach
- Use of static surrogates in hyperparameter optimization
- Accelerated Componentwise Gradient Boosting Using Efficient Data Representation and Momentum-Based Optimization
- scientific article; zbMATH DE number 7415092 (Why is no real title available?)
- HEBO: An Empirical Study of Assumptions in Bayesian Optimisation
- Multiobjective Tree-Structured Parzen Estimator
- Learning and meta-learning of stochastic advection–diffusion–reaction systems from sparse measurements
- Hyperparameter optimization in learning systems
- A data-driven explainable case-based reasoning approach for financial risk detection
- Imbalanced regression and extreme value prediction
- Improving the efficiency of reinforcement learning for a spacecraft powered descent with Q-learning
- Recurrent and convolutional neural networks in structural dynamics: a modified attention steered encoder-decoder architecture versus LSTM versus GRU versus TCN topologies to predict the response of shock wave-loaded plates
- Automatic model training under restrictive time constraints
- Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
- Computationally efficient integrated design and predictive control of flexible energy systems using multi‐fidelity simulation‐based Bayesian optimization
- Naive automated machine learning
- Combining Bayesian optimization and Lipschitz optimization
- Fast Bayesian hyperparameter optimization on large datasets
- Research on spatio-temporal network prediction model of parallel-series traffic flow based on transformer and GCAT
- Local surrogate responses in the Schwarz alternating method for elastic problems on random voided domains
- Scalable Gaussian process-based transfer surrogates for hyperparameter optimization
- mlr3hyperband
- Hyperparameter tuning methods in automated machine learning
- scientific article; zbMATH DE number 7625207 (Why is no real title available?)
- A taxonomy of weight learning methods for statistical relational learning
- scientific article; zbMATH DE number 7370607 (Why is no real title available?)
- Benchmark and survey of automated machine learning frameworks
- AutonoML: Towards an Integrated Framework for Autonomous Machine Learning
- Automated Deep Learning: Neural Architecture Search Is Not the End
- Neural architecture search: a survey
- A machine learning approach for efficient uncertainty quantification using multiscale methods
- Joint detection of malicious domains and infected clients
- Automated porosity estimation using CT-scans of extracted core data
- Bayesian optimization of pump operations in water distribution systems
- An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization
- Estimating shape parameters of piecewise linear-quadratic problems
- scientific article; zbMATH DE number 7255176 (Why is no real title available?)
- Escaping local minima with local derivative-free methods: a numerical investigation
- Best arm identification for contaminated bandits
This page was built for publication: Hyperband: a novel bandit-based approach to hyperparameter optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q72746)
