Hyperband: a novel bandit-based approach to hyperparameter optimization
From MaRDI portal
Learning and adaptive systems in artificial intelligence (68T05) Artificial neural networks and deep learning (68T07) Problem solving in the context of artificial intelligence (heuristics, search strategies, etc.) (68T20) Online algorithms; streaming algorithms (68W27) Approximation methods and heuristics in mathematical programming (90C59)
Abstract: Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.
Recommendations
- Fast Bayesian hyperparameter optimization on large datasets
- scientific article; zbMATH DE number 6276119
- An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization
- No-regret Bayesian optimization with unknown hyperparameters
- Hyperparameter tuning methods in automated machine learning
Cites work
- scientific article; zbMATH DE number 6276119 (Why is no real title available?)
- Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems
- Bayesian optimization in a billion dimensions via random embeddings
- Efficient multi-start strategies for local search algorithms
- Fast Bayesian hyperparameter optimization on large datasets
- Fast cross-validation via sequential testing
- In defense of one-vs-all classification
- Information rates of nonparametric Gaussian process methods
- Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
- Nonparametric guidance of autoencoder representations using label information
- On the complexity of best-arm identification in multi-armed bandit models
- Pure exploration in multi-armed bandits problems
- X-armed bandits
Cited in
(65)- SMAC3: a versatile Bayesian optimization package for hyperparameter optimization
- Neural architecture search: a survey
- Hyperparameter optimization in learning systems
- AutonoML: Towards an Integrated Framework for Autonomous Machine Learning
- Automated Deep Learning: Neural Architecture Search Is Not the End
- A taxonomy of weight learning methods for statistical relational learning
- Escaping local minima with local derivative-free methods: a numerical investigation
- A gradient-based bilevel optimization approach for tuning regularization hyperparameters
- Multistart algorithm for identifying all optima of nonconvex stochastic functions
- Automatic model training under restrictive time constraints
- Train wisely: multifidelity Bayesian optimization hyperparameter tuning in deep learning-based side-channel analysis
- Automatic MILP Solver configuration by learning problem similarities
- Use of static surrogates in hyperparameter optimization
- A machine learning approach for efficient uncertainty quantification using multiscale methods
- Business processes resource management using rewriting logic and deep-learning-based predictive monitoring
- Multiobjective Tree-Structured Parzen Estimator
- Surrogate-based autotuning for randomized sketching algorithms in regression problems
- A data-driven explainable case-based reasoning approach for financial risk detection
- Pricing and calibration in the 4-factor path-dependent volatility model
- Multi-objective model for architecture optimization and training of radial basis function neural networks
- Combining Bayesian optimization and Lipschitz optimization
- Stochastic adversarial noise in the ``black box optimization problem
- Progressively strengthening and tuning MIP solvers for reoptimization
- Bayesian optimization of pump operations in water distribution systems
- An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization
- OpenBox: a Python toolkit for generalized black-box optimization
- Computationally efficient integrated design and predictive control of flexible energy systems using multi‐fidelity simulation‐based Bayesian optimization
- scientific article; zbMATH DE number 7370607 (Why is no real title available?)
- Automated porosity estimation using CT-scans of extracted core data
- Benchmark and survey of automated machine learning frameworks
- scientific article; zbMATH DE number 7415092 (Why is no real title available?)
- Supervised Machine Learning Techniques: An Overview with Applications to Banking
- Investigation of the Lombard effect based on a machine learning approach
- Joint detection of malicious domains and infected clients
- Profiling side-channel attacks on Dilithium. A small bit-fiddling leak breaks it all
- mlr3hyperband
- HEBO: An Empirical Study of Assumptions in Bayesian Optimisation
- Data-driven algorithm selection and tuning in optimization and signal processing
- Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
- scientific article; zbMATH DE number 7255176 (Why is no real title available?)
- scientific article; zbMATH DE number 7415103 (Why is no real title available?)
- scientific article; zbMATH DE number 7415127 (Why is no real title available?)
- Improving the efficiency of reinforcement learning for a spacecraft powered descent with Q-learning
- Estimating shape parameters of piecewise linear-quadratic problems
- Recurrent and convolutional neural networks in structural dynamics: a modified attention steered encoder-decoder architecture versus LSTM versus GRU versus TCN topologies to predict the response of shock wave-loaded plates
- Best arm identification for contaminated bandits
- Learning and meta-learning of stochastic advection–diffusion–reaction systems from sparse measurements
- Hyperparameter tuning methods in automated machine learning
- Accelerated Componentwise Gradient Boosting Using Efficient Data Representation and Momentum-Based Optimization
- Imbalanced regression and extreme value prediction
- Research on spatio-temporal network prediction model of parallel-series traffic flow based on transformer and GCAT
- Local surrogate responses in the Schwarz alternating method for elastic problems on random voided domains
- Detecting antimicrobial resistance through MALDI-TOF mass spectrometry with statistical guarantees using conformal prediction
- Explaining random forest and XGBoost with shallow decision trees by co-clustering feature importance
- The ``black-box optimization problem: zero-order accelerated stochastic method via kernel approximation
- Code generation for solving and differentiating through convex optimization problems
- Hyperparameter optimization in machine learning
- ML-plan: automated machine learning via hierarchical planning
- Data-rich economic forecasting for actuarial applications
- Naive automated machine learning
- Goal-oriented sensitivity analysis of hyperparameters in deep learning
- Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification
- Scalable Gaussian process-based transfer surrogates for hyperparameter optimization
- Data farming the parameters of simulation-optimization solvers
- Fast Bayesian hyperparameter optimization on large datasets
This page was built for publication: Hyperband: a novel bandit-based approach to hyperparameter optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q72746)
