Dropout training for SVMs with data augmentation

DOI10.1007/S11704-018-7314-7MaRDI QIDQ1713848zbMATHOpenAlexFDO

Authors Ning Chen, J. Zhu, Jianfei Chen, Ting Chen

Publication date 30 January 2019

Published in Frontiers of Computer Science (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1508.02268

zbMATH Keywords

data augmentationlogistic regressiondropoutiteratively reweighted least squaresvms

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Generalized linear models (logistic models) (62J12)

Abstract: Dropout and other feature noising schemes have shown promising results in controlling over-fitting by artificially corrupting the training data. Though extensive theoretical and empirical studies have been performed for generalized linear models, little work has been done for support vector machines (SVMs), one of the most successful approaches for supervised learning. This paper presents dropout training for both linear SVMs and the nonlinear extension with latent representation learning. For linear SVMs, to deal with the intractable expectation of the non-smooth hinge loss under corrupting distributions, we develop an iteratively re-weighted least square (IRLS) algorithm by exploring data augmentation techniques. Our algorithm iteratively minimizes the expectation of a re-weighted least square problem, where the re-weights are analytically updated. For nonlinear latent SVMs, we consider learning one layer of latent representations in SVMs and extend the data augmentation technique in conjunction with first-order Taylor-expansion to deal with the intractable expected non-smooth hinge loss and the nonlinearity of latent representations. Finally, we apply the similar data augmentation ideas to develop a new IRLS algorithm for the expected logistic loss under corrupting distributions, and we further develop a non-linear extension of logistic regression by incorporating one layer of latent representations. Our algorithms offer insights on the connection and difference between the hinge loss and logistic loss in dropout training. Empirical results on several real datasets demonstrate the effectiveness of dropout training on significantly boosting the classification accuracy of both linear and nonlinear SVMs. In addition, the nonlinear SVMs further improve the prediction performance on several image datasets.

Recommendations

• Data augmentation for support vector machines
• The dropout learning algorithm
• scientific article; zbMATH DE number 6378127
• Robust Truncated Hinge Loss Support Vector Machines
• Stream-suitable optimization algorithms for some soft-margin support vector machine variants

Cites work

• scientific article; zbMATH DE number 6378127 (Why is no real title available?)
• scientific article; zbMATH DE number 3954145 (Why is no real title available?)
• scientific article; zbMATH DE number 1149420 (Why is no real title available?)
• scientific article; zbMATH DE number 823069 (Why is no real title available?)
• Are Loss Functions All the Same?
• Bayesian Inference for Logistic Models Using Pólya–Gamma Latent Variables
• Data augmentation for support vector machines
• Gibbs max-margin topic models with data augmentation
• In defense of one-vs-all classification
• Learning to classify with missing and corrupted features
• On the inductive bias of dropout
• On the limited memory BFGS method for large scale optimization
• The dropout learning algorithm

Cited in

(3)

• Data augmentation via Lévy processes
• Data augmentation for support vector machines
• On the inductive bias of dropout

Describes a project that uses

Uses Software

• L-BFGS
• BayesLogit
• ElemStatLearn
• CIFAR

This page was built for publication: Dropout training for SVMs with data augmentation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1713848)