Bias of homotopic gradient descent for the hinge loss

DOI10.1007/S00245-020-09656-5MaRDI QIDQ2045131zbMATH OpenOpenAlexWikidataFDO

Authors Denali Molitor, D. Needell, Rachel Ward

Publication date 11 August 2021

Published in Applied Mathematics and Optimization (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1907.11746

zbMATH Keywords

convex optimizationgradient descenthinge lossnon-smooth optimizationsupport vector machines

Mathematics Subject Classification ID

Convex programming (90C25)

Abstract: Gradient descent is a simple and widely used optimization method for machine learning. For homogeneous linear classifiers applied to separable data, gradient descent has been shown to converge to the maximal margin (or equivalently, the minimal norm) solution for various smooth loss functions. The previous theory does not, however, apply to non-smooth functions such as the hinge loss which is widely used in practice. Here, we study the convergence of a homotopic variant of gradient descent applied to the hinge loss and provide explicit convergence rates to the max-margin solution for linearly separable data.

Recommendations

• The implicit bias of gradient descent on separable data
• Are Loss Functions All the Same?
• Gradient learning in a classification setting by gradient descent
• Learning rates of gradient descent algorithm for classification
• scientific article; zbMATH DE number 5578317

Cites work

• Convex optimization: algorithms and complexity
• Entropy-SGD: biasing gradient descent into wide valleys
• Estimation of dependences based on empirical data. Transl. from the Russian by Samuel Kotz
• scientific article; zbMATH DE number 5957245 (Why is no real title available?)
• scientific article; zbMATH DE number 3446442 (Why is no real title available?)
• scientific article; zbMATH DE number 1391397 (Why is no real title available?)
• Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalization
• Optimization methods for large-scale machine learning
• Support-vector networks
• The implicit bias of gradient descent on separable data
• Towards a deeper geometric, analytic and algorithmic understanding of margins
• Training a Support Vector Machine in the Primal

Cited in

(5)

• scientific article; zbMATH DE number 5578317 (Why is no real title available?)
• scientific article; zbMATH DE number 2040718 (Why is no real title available?)
• The implicit bias of gradient descent on separable data
• Gradient methods never overfit on separable data
• Iterative regularization in classification via hinge loss diagonal descent

Describes a project that uses

Uses Software

• Entropy-SGD

This page was built for publication: Bias of homotopic gradient descent for the hinge loss

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2045131)