An accelerated proximal algorithm for regularized nonconvex and nonsmooth bi-level optimization
From MaRDI portal
Publication:6161202
DOI10.1007/S10994-023-06329-6arXiv2203.16615OpenAlexW4362697005MaRDI QIDQ6161202FDOQ6161202
Authors: Ziyi Chen, Bhavya Kailkhura, Yi Zhou
Publication date: 27 June 2023
Published in: Machine Learning (Search for Journal in Brave)
Abstract: Many important machine learning applications involve regularized nonconvex bi-level optimization. However, the existing gradient-based bi-level optimization algorithms cannot handle nonconvex or nonsmooth regularizers, and they suffer from a high computation complexity in nonconvex bi-level optimization. In this work, we study a proximal gradient-type algorithm that adopts the approximate implicit differentiation (AID) scheme for nonconvex bi-level optimization with possibly nonconvex and nonsmooth regularizers. In particular, the algorithm applies the Nesterov's momentum to accelerate the computation of the implicit gradient involved in AID. We provide a comprehensive analysis of the global convergence properties of this algorithm through identifying its intrinsic potential function. In particular, we formally establish the convergence of the model parameters to a critical point of the bi-level problem, and obtain an improved computation complexity over the state-of-the-art result. Moreover, we analyze the asymptotic convergence rates of this algorithm under a class of local nonconvex geometries characterized by a {L}ojasiewicz-type gradient inequality. Experiment on hyper-parameter optimization demonstrates the effectiveness of our algorithm.
Full work available at URL: https://arxiv.org/abs/2203.16615
Recommendations
- Convex Bi-level Optimization Problems with Nonsmooth Outer Objective Function
- Methodology and first-order algorithms for solving nonsmooth and non-strongly convex bilevel optimization problems
- Accelerated fixed point algorithm for convex bi-level optimization problems in Hilbert spaces with applications
- A gentle introduction to algorithms for bilevel optimization from machine learning
Cites Work
- Splitting Algorithms for the Sum of Two Nonlinear Operators
- Introductory lectures on convex optimization. A basic course.
- Proximal alternating linearized minimization for nonconvex and nonsmooth problems
- New Branch-and-Bound Rules for Linear Bilevel Programming
- Title not available (Why is that?)
- Mathematical Programs with Optimization Problems in the Constraints
- An extended Kuhn-Tucker approach for linear bilevel programming
- The Łojasiewicz Inequality for Nonsmooth Subanalytic Functions with Applications to Subgradient Dynamical Systems
- Splitting methods with variable metric for Kurdyka-Łojasiewicz functions and general convergence rates
- On the convergence of the proximal algorithm for nonsmooth functions involving analytic features
- OnActor-Critic Algorithms
- Distributed Proximal Gradient Algorithm for Partially Asynchronous Computer Clusters
- A Two-Timescale Stochastic Algorithm Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Critic
Cited In (5)
- Alternating proximal algorithm for the problem of bi-level convex minimization
- Accelerated fixed point algorithm for convex bi-level optimization problems in Hilbert spaces with applications
- Analyzing inexact hypergradients for bilevel learning
- A gentle introduction to algorithms for bilevel optimization from machine learning
- Convex Bi-level Optimization Problems with Nonsmooth Outer Objective Function
This page was built for publication: An accelerated proximal algorithm for regularized nonconvex and nonsmooth bi-level optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6161202)