Hyperband: a novel bandit-based approach to hyperparameter optimization

DOI10.48550/ARXIV.1603.06560zbMATH Open1468.68204arXiv1603.06560MaRDI QIDQ72746FDOQ72746

Authors: Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar, Lisha Li, Kevin Jamieson, Giulia Desalvo, Afshin Rostamizadeh, Ameet Talwalkar 

Publication date: 21 March 2016

Abstract: Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.

Full work available at URL: https://arxiv.org/abs/1603.06560

Recommendations

• Fast Bayesian hyperparameter optimization on large datasets
• scientific article; zbMATH DE number 6276119
• An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization
• No-regret Bayesian optimization with unknown hyperparameters
• Hyperparameter tuning methods in automated machine learning

zbMATH Keywords

model selectiondeep learninghyperparameter optimizationinfinite-armed banditsonline optimization

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Artificial neural networks and deep learning (68T07) Problem solving in the context of artificial intelligence (heuristics, search strategies, etc.) (68T20) Online algorithms; streaming algorithms (68W27) Approximation methods and heuristics in mathematical programming (90C59)

Cites Work

• Fast Bayesian hyperparameter optimization on large datasets
• Bayesian optimization in a billion dimensions via random embeddings
• On the complexity of best-arm identification in multi-armed bandit models
• Efficient multi-start strategies for local search algorithms
• In defense of one-vs-all classification
• Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems
• Pure exploration in multi-armed bandits problems
• Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)

Cited In (49)

• The ``black-box optimization problem: zero-order accelerated stochastic method via kernel approximation
• Hyperparameter optimization in learning systems
• A taxonomy of weight learning methods for statistical relational learning
• Escaping local minima with local derivative-free methods: a numerical investigation
• Automatic MILP Solver configuration by learning problem similarities
• Automatic model training under restrictive time constraints
• Business processes resource management using rewriting logic and deep-learning-based predictive monitoring
• Use of static surrogates in hyperparameter optimization
• Multiobjective Tree-Structured Parzen Estimator
• Benchmark and Survey of Automated Machine Learning Frameworks
• A machine learning approach for efficient uncertainty quantification using multiscale methods
• A data-driven explainable case-based reasoning approach for financial risk detection
• Stochastic adversarial noise in the ``black box optimization problem
• Progressively strengthening and tuning MIP solvers for reoptimization
• Combining Bayesian optimization and Lipschitz optimization
• Best Arm Identification for Contaminated Bandits
• Bayesian optimization of pump operations in water distribution systems
• An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization
• OpenBox: a Python toolkit for generalized black-box optimization
• Computationally efficient integrated design and predictive control of flexible energy systems using multi‐fidelity simulation‐based Bayesian optimization
• Title not available (Why is that?)
• Automated porosity estimation using CT-scans of extracted core data
• Supervised Machine Learning Techniques: An Overview with Applications to Banking
• Investigation of the Lombard effect based on a machine learning approach
• Title not available (Why is that?)
• Profiling side-channel attacks on Dilithium. A small bit-fiddling leak breaks it all
• Joint detection of malicious domains and infected clients
• HEBO: An Empirical Study of Assumptions in Bayesian Optimisation
• Title not available (Why is that?)
• Data-driven algorithm selection and tuning in optimization and signal processing
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Improving the efficiency of reinforcement learning for a spacecraft powered descent with Q-learning
• Estimating shape parameters of piecewise linear-quadratic problems
• Title not available (Why is that?)
• Recurrent and convolutional neural networks in structural dynamics: a modified attention steered encoder-decoder architecture versus LSTM versus GRU versus TCN topologies to predict the response of shock wave-loaded plates
• Learning and meta-learning of stochastic advection–diffusion–reaction systems from sparse measurements
• mlr3hyperband
• Accelerated Componentwise Gradient Boosting Using Efficient Data Representation and Momentum-Based Optimization
• Imbalanced regression and extreme value prediction
• Research on spatio-temporal network prediction model of parallel-series traffic flow based on transformer and GCAT
• Local surrogate responses in the Schwarz alternating method for elastic problems on random voided domains
• Naive automated machine learning
• Goal-oriented sensitivity analysis of hyperparameters in deep learning
• Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification
• AutonoML: Towards an Integrated Framework for Autonomous Machine Learning
• Automated Deep Learning: Neural Architecture Search Is Not the End

Uses Software

• Spearmint
• CIFAR
• Hyperopt
• SMAC
• auto-sklearn
• Hyperband

This page was built for publication: Hyperband: a novel bandit-based approach to hyperparameter optimization

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q72746)