Hyperband: a novel bandit-based approach to hyperparameter optimization
From MaRDI portal
(Redirected from Publication:72746)
Learning and adaptive systems in artificial intelligence (68T05) Artificial neural networks and deep learning (68T07) Problem solving in the context of artificial intelligence (heuristics, search strategies, etc.) (68T20) Online algorithms; streaming algorithms (68W27) Approximation methods and heuristics in mathematical programming (90C59)
Abstract: Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.
Recommendations
- Fast Bayesian hyperparameter optimization on large datasets
- scientific article; zbMATH DE number 6276119
- An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization
- No-regret Bayesian optimization with unknown hyperparameters
- Hyperparameter tuning methods in automated machine learning
Cites work
- scientific article; zbMATH DE number 6276119 (Why is no real title available?)
- Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems
- Bayesian optimization in a billion dimensions via random embeddings
- Efficient multi-start strategies for local search algorithms
- Fast Bayesian hyperparameter optimization on large datasets
- Fast cross-validation via sequential testing
- In defense of one-vs-all classification
- Information rates of nonparametric Gaussian process methods
- Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
- Nonparametric guidance of autoencoder representations using label information
- On the complexity of best-arm identification in multi-armed bandit models
- Pure exploration in multi-armed bandits problems
- \(X\)-armed bandits
Cited in
(53)- The ``black-box optimization problem: zero-order accelerated stochastic method via kernel approximation
- OpenBox: a Python toolkit for generalized black-box optimization
- Profiling side-channel attacks on Dilithium. A small bit-fiddling leak breaks it all
- Automatic MILP Solver configuration by learning problem similarities
- Stochastic adversarial noise in the ``black box optimization problem
- Progressively strengthening and tuning MIP solvers for reoptimization
- scientific article; zbMATH DE number 7415103 (Why is no real title available?)
- scientific article; zbMATH DE number 7415127 (Why is no real title available?)
- Goal-oriented sensitivity analysis of hyperparameters in deep learning
- Business processes resource management using rewriting logic and deep-learning-based predictive monitoring
- Data-driven algorithm selection and tuning in optimization and signal processing
- ML-plan: automated machine learning via hierarchical planning
- Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification
- Supervised Machine Learning Techniques: An Overview with Applications to Banking
- Investigation of the Lombard effect based on a machine learning approach
- Use of static surrogates in hyperparameter optimization
- Accelerated Componentwise Gradient Boosting Using Efficient Data Representation and Momentum-Based Optimization
- scientific article; zbMATH DE number 7415092 (Why is no real title available?)
- HEBO: An Empirical Study of Assumptions in Bayesian Optimisation
- Multiobjective Tree-Structured Parzen Estimator
- Learning and meta-learning of stochastic advection–diffusion–reaction systems from sparse measurements
- Hyperparameter optimization in learning systems
- A data-driven explainable case-based reasoning approach for financial risk detection
- Imbalanced regression and extreme value prediction
- Improving the efficiency of reinforcement learning for a spacecraft powered descent with Q-learning
- Recurrent and convolutional neural networks in structural dynamics: a modified attention steered encoder-decoder architecture versus LSTM versus GRU versus TCN topologies to predict the response of shock wave-loaded plates
- Automatic model training under restrictive time constraints
- Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
- Computationally efficient integrated design and predictive control of flexible energy systems using multi‐fidelity simulation‐based Bayesian optimization
- Naive automated machine learning
- Combining Bayesian optimization and Lipschitz optimization
- Fast Bayesian hyperparameter optimization on large datasets
- Research on spatio-temporal network prediction model of parallel-series traffic flow based on transformer and GCAT
- Local surrogate responses in the Schwarz alternating method for elastic problems on random voided domains
- Scalable Gaussian process-based transfer surrogates for hyperparameter optimization
- mlr3hyperband
- Hyperparameter tuning methods in automated machine learning
- scientific article; zbMATH DE number 7625207 (Why is no real title available?)
- A taxonomy of weight learning methods for statistical relational learning
- scientific article; zbMATH DE number 7370607 (Why is no real title available?)
- Benchmark and survey of automated machine learning frameworks
- AutonoML: Towards an Integrated Framework for Autonomous Machine Learning
- Automated Deep Learning: Neural Architecture Search Is Not the End
- Neural architecture search: a survey
- A machine learning approach for efficient uncertainty quantification using multiscale methods
- Joint detection of malicious domains and infected clients
- Automated porosity estimation using CT-scans of extracted core data
- Bayesian optimization of pump operations in water distribution systems
- An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization
- Estimating shape parameters of piecewise linear-quadratic problems
- scientific article; zbMATH DE number 7255176 (Why is no real title available?)
- Escaping local minima with local derivative-free methods: a numerical investigation
- Best arm identification for contaminated bandits
This page was built for publication: Hyperband: a novel bandit-based approach to hyperparameter optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q72746)
