Data-Driven Mirror Descent with Input-Convex Neural Networks

DOI10.1137/22M1508613MaRDI QIDQ6171691zbMATH OpenOpenAlexFDO

Authors Subhadip Mukherjee, J. Tang, Carola-Bibiane Schönlieb

Publication date 14 August 2023

Published in SIAM Journal on Mathematics of Data Science (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2206.06733

zbMATH Keywords

data-driven convex optimization solversinput-convex neural networksinverse problemsmirror descent

Mathematics Subject Classification ID

Applications of functional analysis in optimization, convex analysis, mathematical programming, economics (46N10) Roundoff error (65G50) Numerical optimization and variational techniques (65K10) Learning and adaptive systems in artificial intelligence (68T05)

Abstract: Learning-to-optimize is an emerging framework that seeks to speed up the solution of certain optimization problems by leveraging training data. Learned optimization solvers have been shown to outperform classical optimization algorithms in terms of convergence speed, especially for convex problems. Many existing data-driven optimization methods are based on parameterizing the update step and learning the optimal parameters (typically scalars) from the available data. We propose a novel functional parameterization approach for learned convex optimization solvers based on the classical mirror descent (MD) algorithm. Specifically, we seek to learn the optimal Bregman distance in MD by modeling the underlying convex function using an input-convex neural network (ICNN). The parameters of the ICNN are learned by minimizing the target objective function evaluated at the MD iterate after a predetermined number of iterations. The inverse of the mirror map is modeled approximately using another neural network, as the exact inverse is intractable to compute. We derive convergence rate bounds for the proposed learned mirror descent (LMD) approach with an approximate inverse mirror map and perform extensive numerical evaluation on various convex problems such as image inpainting, denoising, learning a two-class support vector machine (SVM) classifier and a multi-class linear classifier on fixed features.

Recommendations

• Data-driven nonsmooth optimization
• Deep inverse optimization
• Mirror descent and nonlinear projected subgradient methods for convex optimization.
• Accelerated Forward-Backward Optimization using Deep Learning
• An accelerated stochastic mirror descent method

Cites work

• A first-order primal-dual algorithm for convex problems with applications to imaging
• A generalized online mirror descent with applications to classification and regression
• Accelerated Bregman proximal gradient methods for relatively smooth convex optimization
• An optimal method for stochastic composite optimization
• An optimal randomized incremental gradient method
• Convex optimization: algorithms and complexity
• Data-driven nonsmooth optimization
• Gradient methods for minimizing composite functions
• scientific article; zbMATH DE number 3790208 (Why is no real title available?)
• Mirror descent and nonlinear projected subgradient methods for convex optimization.
• Pattern recognition and machine learning.
• Relatively smooth convex optimization by first-order methods, and applications
• The ordered subsets mirror descent optimization method with applications to tomography
• Variational Analysis

Cited in

(4)

• Hessian informed mirror descent
• Data-driven performance guarantees for classical and learned optimizers
• A guide to stochastic optimisation for large-scale inverse problems
• Piecewise deterministic sampling for constrained distributions

This page was built for publication: Data-Driven Mirror Descent with Input-Convex Neural Networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6171691)