On \ell_p-hyperparameter Learning via Bilevel Nonsmooth Optimization

MaRDI QIDQ6302551arXivFDO

Authors Takayuki Okuno, Akiko Takeda, Akihiro Kawana, Motokazu Watanabe

Publication date 5 June 2018

Published in arXiv

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05)

arXiv classification

math.OC

Abstract: We propose a bilevel optimization strategy for selecting the best hyperparameter value for the nonsmooth $el{l}_p$ regularizer with $0<ple1$. The concerned bilevel optimization problem has a nonsmooth, possibly nonconvex, $el{l}_p$-regularized problem as the lower-level problem. Despite the recent popularity of nonconvex $el{l}_p$-regularizer and the usefulness of bilevel optimization for selecting hyperparameters, algorithms for such bilevel problems have not been studied because of the difficulty of $el{l}_p$-regularizer. Our contribution is the proposal of the first algorithm equipped with a theoretical guarantee for finding the best hyperparameter of $el{l}_p$-regularized supervised learning problems. Specifically, we propose a smoothing-type algorithm for the above mentioned bilevel optimization problems and provide a theoretical convergence guarantee for the algorithm. Indeed, since optimality conditions are not known for such bilevel optimization problems so far, new necessary optimality conditions, which are called the SB-KKT conditions, are derived and it is shown that a sequence generated by the proposed algorithm actually accumulates at a point satisfying the SB-KKT conditions under some mild assumptions. The proposed algorithm is simple and scalable as our numerical comparison to Bayesian optimization and grid search indicates.

This page was built for publication: On $\ell_p$-hyperparameter Learning via Bilevel Nonsmooth Optimization

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6302551)