Efficient learning with robust gradient descent (Q2320583)

Gradient descent is a very popular optimization method with wide applications to many machine learning problems. However, gradient descent may not generalize well if the training data is noisy or heavy-tailed. To address this defect, this paper proposes an efficient robust gradient descent method. The idea is to use a robust estimate of the risk gradient rather than the risk itself, which is achieved by a soft-truncation of errant values. The proposed algorithm enjoys the computation ease of gradient descent, and is backed up by both theoretical and experimental analyses.